:W&Bá¾( VM ickk4hl3ª; 9 /0.28.1+bc2cf24b1bd9fa3a1b6cbeb920087b832afc526f0.70.0¹å,‚‚" jk6qih7c1gew ickk4hl3ŠÞ ickk4hl3escheuller-uc-san-diego spatop-sweep"ž _wandb‚{}  batch_size‚2048  dropout‚0.1398185687940936  hidden_dim‚256 ' learning_rate‚0.0006918484518852536  num_embedding_layers‚10Bdpfx_vanilla_ickk4hl3bnqupkqclj%dp-spatop-sweepagent-vanilla-v2-h9cgsŠ —´èÒ¨³Èuš3 (H¨¸hp"3.12.10*0.28.1j linux-x86_64%{º…; ickk4hl3*  _wandb‚ {"runtime":0}  _runtime‚0ˆaÒÛ -Linux-6.8.0-134-generic-x86_64-with-glibc2.36CPython 3.12.10 —´èÒ过/*-of*>/data/spatop/options_files/sweep_fixed_dp/config_ickk4hl3.json*-l* /scratch/logs*-n*dpfx_vanilla_ickk4hl32-m spanet.trainRdanielprimosch@gmail.comZ /scratch/logsb%dp-spatop-sweepagent-vanilla-v2-h9cgsr/opt/conda/bin/pythonº fcngd2c48oc18bqh3s3y0qturzo6nbqiNÿ$ÖÉÒÀ€ˆ€’NVIDIA A100-SXM4-80GB˜¢ /€€€¥È.€àÚÁœª€àçÓ½ÂS NVIDIA A100-SXM4-80GB€€€€À€6"Ampere*(GPU-0c583aac-6f68-6140-7078-f37971b4b9bbÊ13.2º fcngd2c48oc18bqh3s3y0qturzo6nbqiµª½Ç2  requirements.txtˆe˜QW ickk4hl3ZF (H¨¸ (H¨¸ (hpð"3.12.10*0.28.1j linux-x86_640“³Y ickk4hl3ZH (H¨¸ (H¨¸ (8hpð"3.12.10*0.28.1j linux-x86_64O‰âÏ*  ickk4hl3b trainer/global_step2™®3\  ickk4hl3ZK (H¨¸ (H¨¸(8hpð"3.12.10*0.28.1j linux-x86_64˜0/  ickk4hl3b*"trainer/global_step2ËÞÇ‹K  ickk4hl3j: —´èÒàÛÖ *LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] ŸŸó’  ickk4hl3j€  ˜´èÒ¸¥‰3î â”â”â”â”┳â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”┳â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”┳â”â”â”â”â”â”â”â”┳â”â”â”â”â”â”â”┳â”â”â”â”â”â”â”┓ ┃ ┃ Name ┃ Type ┃ Params ┃ Mode ┃ FLOPs ┃ ┡â”â”â”╇â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”╇â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”╇â”â”â”â”â”â”â”â”╇â”â”â”â”â”â”â”╇â”â”â”â”â”â”â”┩ │ 0 │ embedding │ MultiInputVectorEmbed… │ 3.4 M │ train │ 0 │ │ 1 │ encoder │ JetEncoder │ 5.3 M │ train │ 0 │ │ 2 │ branch_decoders │ ModuleList │ 75.8 M │ train │ 0 │ │ 3 │ regression_decoder │ RegressionDecoder │ 0 │ train │ 0 │ │ 4 │ classification_decoder │ ClassificationDecoder │ 0 │ train │ 0 │ │ │ other params │ n/a │ 12 │ n/a │ n/a │ └───┴────────────────────────┴────────────────────────┴────────┴───────┴───────┘ F?[ ickk4hl3jÉ ˜´èÒàâ‰4·Trainable params: 84.5 M Non-trainable params: 50 Total params: 84.5 M Total estimated model params size (MB): 337.922 Modules in train mode: 2299 Modules in eval mode: 0 Total FLOPs: 0 ŠZñ=p ickk4hl3j_ ˜´èÒ°•¨4Pwandb: WARNING Config item 'hidden_dim' was locked by 'sweep' (ignored update). Vâ®z ickk4hl3ji ˜´èÒèü·4Zwandb: WARNING Config item 'num_embedding_layers' was locked by 'sweep' (ignored update). ðbp ickk4hl3j_ ˜´èÒèä¿4Pwandb: WARNING Config item 'batch_size' was locked by 'sweep' (ignored update). ªUñÍs ickk4hl3jb ˜´èÒ°µÇ4Swandb: WARNING Config item 'learning_rate' was locked by 'sweep' (ignored update). [Mm ickk4hl3j\ ˜´èÒˆÚÍ4Mwandb: WARNING Config item 'dropout' was locked by 'sweep' (ignored update). ¥öÿ¦ð ickk4hl3*Þ  transformer_dim‚256  transformer_dim_scale‚2.0  initial_embedding_dim‚256  position_embedding_dim‚256  num_encoder_layers‚4 ! num_branch_embedding_layers‚3  num_branch_encoder_layers‚3  num_jet_embedding_layers‚0  num_jet_encoder_layers‚2  num_detector_layers‚2  num_regression_layers‚3  num_classification_layers‚3  split_symmetric_attention‚1  num_attention_heads‚8 ! transformer_activation‚"gelu"  skip_connections‚1 ( "initial_embedding_skip_connections‚1  linear_block_type‚"GRU"  transformer_type‚"Gated"  linear_activation‚"gelu"  normalization‚ "LayerNorm"  masking‚ "Filling" linear_prelu_activation‚true # use_pairwise_interactions‚false  num_pairwise_features‚4  pairwise_embedding_dim‚8  pairwise_input_source‚""  use_moe‚false  num_experts‚8  num_experts_per_tok‚2  moe_loss_scale‚0.01 K event_info_file‚7"/data/spatop/event_files/v11/tt_hadronic_v7_full.yaml" F training_file‚4"/data/spatop/tpm70_wpm30_FB350_fixed/all_merged.h5"  validation_file‚""  testing_file‚""  normalize_features‚1  limit_to_num_jets‚0  balance_particles‚0  balance_jets‚0 ! balance_classifications‚false  partial_events‚1  dataset_limit‚1.0  dataset_randomization‚0  train_validation_split‚0.95  limit_val_batches‚1.0  num_dataloader_workers‚0  mask_sequence_vectors‚1  combine_pair_loss‚"min"  optimizer‚"AdamW"  focal_gamma‚0.0  combinatorial_scale‚0.0 # learning_rate_warmup_epochs‚1.0  learning_rate_cycles‚1  assignment_loss_scale‚1.0  detection_loss_scale‚1.0  kl_loss_scale‚0.0  regression_loss_scale‚0.0 ! classification_loss_scale‚0.0 ! use_mass_classification‚false  mass_classes‚[]  balance_losses‚true  l2_penalty‚0.0002  gradient_clip‚10.0 epochs‚50 num_gpu‚1  verbose_output‚false  usable_gpus‚""  trial_time‚"" $ trial_output_dir‚"./test_output"Dªßíê ickk4hl3jØ ˜´èÒØÁø8È/opt/conda/lib/python3.12/site-packages/pytorch_lightning/utilities/_pytree.py:21: `isinstance(treespec, LeafSpec)` is deprecated, use `isinstance(treespec, TreeSpec) and treespec.is_leaf()` instead. õšR¨M ickk4hl3j» ˜´èÒ€Þ†9«/opt/conda/lib/python3.12/site-packages/pytorch_lightning/trainer/connectors/data_connector.py:434: The 'val_dataloader' does not have many workers which may be a bottleneck. Consider increasing the value of the `num_workers` argument` to `num_workers=3` in the `DataLoader` to improve performance. K”¾yë ickk4hl3jÙ ›´èÒè÷•¬È/opt/conda/lib/python3.12/site-packages/pytorch_lightning/utilities/_pytree.py:21: `isinstance(treespec, LeafSpec)` is deprecated, use `isinstance(treespec, TreeSpec) and treespec.is_leaf()` instead. ƒðï™P ickk4hl3j¾ ›´èÒ€º§¬­/opt/conda/lib/python3.12/site-packages/pytorch_lightning/trainer/connectors/data_connector.py:434: The 'train_dataloader' does not have many workers which may be a bottleneck. Consider increasing the value of the `num_workers` argument` to `num_workers=3` in the `DataLoader` to improve performance. aI‚(:“ ¦´èÒýÑ–ù gpu.0.gpu‚99.0 gpu.0.memory‚29+ gpu.0.memoryAllocated‚57.74421691894531* gpu.0.memoryAllocatedBytes‚ 49601904640 gpu.0.temp‚39.0 gpu.0.powerWatts‚283.559' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚56.711800000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‹Z¯¾±‚(:§ ¦´èÒ©¡Ë disk.rbd1.in‚0 disk.sda3.out‚ 0.40234375 disk.nvme0n1.in‚0$ memory_percent‚97.00582027435303 disk.sda1.in‚0 disk.dm-1.in‚0" proc.memory.rssMB‚ 2762.5390625 proc.cpu.threads‚17 disk.drbd2165.out‚0 disk.rbd0.out‚0 disk.dm-0.out‚ 10.2099609375 disk.sda1.out‚0( proc.memory.availableMB‚ 490.56640625( proc.memory.percent‚16.8612003326416 disk.dm-0.in‚0 disk.rbd1.out‚0.0625 disk.sda.in‚0 disk.drbd2165.in‚0 network.sent‚32995 network.recv‚41855 disk.rbd0.in‚0$ disk.nvme0c0n1.out‚ 10.2099609375" disk.nvme0n1.out‚ 10.2099609375) disk./.usagePercent‚36.98709453731533 disk.rbd3.out‚0 disk.nvme0c0n1.in‚0$ disk./.usageGB‚551.1344375610352 disk.rbd3.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda.out‚ 0.40234375 disk.sda3.in‚0 disk.dm-1.out‚ 0.40234375 cpu‚26.98743346016208-žF§“‚(:‰ µ´èÒ¾êÅø gpu.0.gpu‚92.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.749099731445305* gpu.0.memoryAllocatedBytes‚ 49606098944 gpu.0.temp‚42.0 gpu.0.powerWatts‚322.073' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚64.4146 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0=W±›´‚(:ª µ´èÒŸ‡Òú disk.dm-1.in‚0 disk.drbd2165.out‚0$ memory_percent‚97.00436592102051# proc.memory.rssMB‚ 2762.57421875 cpu‚26.538209440879875 disk.rbd0.in‚0 disk.dm-0.out‚ 24.9912109375 disk.sda2.in‚0 disk.rbd1.out‚0.0625 disk.rbd1.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 network.recv‚45957 disk.dm-1.out‚ 0.68359375 disk.dm-0.in‚0 disk.drbd2165.in‚0" disk.nvme0n1.out‚ 24.9912109375 disk.sda3.out‚ 0.68359375 proc.cpu.threads‚17 disk.nvme0c0n1.in‚0 disk.nvme0n1.in‚0 disk.sda3.in‚0$ disk.nvme0c0n1.out‚ 24.9912109375 disk.sda2.out‚0 network.sent‚41615) disk./.usagePercent‚36.98748648485025$ disk./.usageGB‚551.1402778625488 disk.sda.out‚ 0.68359375' proc.memory.availableMB‚ 490.8046875* proc.memory.percent‚16.861414909362793 disk.sda.in‚0 disk.rbd0.out‚0 disk.rbd3.in‚0 disk.rbd3.out‚0Ó’Š’‚(:ˆ Ä´èÒõïÙø gpu.0.gpu‚99.0 gpu.0.memory‚29+ gpu.0.memoryAllocated‚57.75154113769532* gpu.0.memoryAllocatedBytes‚ 49608196096 gpu.0.temp‚42.0 gpu.0.powerWatts‚291.044' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚58.2088 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ðü*³¯‚(:¥ Ä´èÒÑëöú* disk./.usagePercent‚36.987929377884484 disk.sda3.out‚ 0.8359375" disk.nvme0n1.out‚ 32.7041015625 disk.sda2.in‚0 network.sent‚46987 disk.sda3.in‚0 disk.rbd1.in‚0 disk.sda.in‚0 disk.rbd0.in‚0 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.rbd3.out‚0 disk.nvme0n1.in‚0% proc.memory.availableMB‚ 490.71875* proc.memory.percent‚16.861605644226074 disk.rbd3.in‚0 disk.nvme0c0n1.in‚0 disk.rbd1.out‚0.125$ memory_percent‚97.00489044189453 disk.sda1.out‚0 disk.sda2.out‚0 disk.rbd0.out‚0 cpu‚26.284138911626645 proc.cpu.threads‚17 disk.dm-0.in‚0$ disk./.usageGB‚551.1468772888184$ disk.nvme0c0n1.out‚ 32.7041015625 disk.dm-1.out‚ 0.8359375 network.recv‚49490 disk.sda1.in‚0 disk.sda.out‚ 0.8359375 disk.dm-1.in‚0# proc.memory.rssMB‚ 2762.60546875 disk.dm-0.out‚ 32.7041015625¼g¡“‚(:‰ Ó´èÒæ©ðø gpu.0.gpu‚100.0 gpu.0.memory‚33+ gpu.0.memoryAllocated‚57.75154113769532* gpu.0.memoryAllocatedBytes‚ 49608196096 gpu.0.temp‚44.0 gpu.0.powerWatts‚333.378' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚66.6756 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0.š)É ‚(:¿ Ó´èÒ¯¢û disk.rbd3.in‚0# disk.nvme0n1.out‚46.04248046875 disk.rbd0.out‚0$ disk./.usageGB‚551.1520576477051 disk.sda.in‚0 disk.dm-1.out‚ 1.0234375 disk.drbd2165.out‚0% disk.nvme0c0n1.out‚46.04248046875 disk.nvme0n1.in‚0.015625* proc.memory.percent‚16.861701011657715 cpu‚26.124607097773286 disk.sda1.out‚0 disk.sda.out‚ 1.0234375 disk.rbd3.out‚0 disk.sda2.out‚0 disk.rbd1.in‚0 proc.cpu.threads‚17 disk.drbd2165.in‚0 disk.sda2.in‚0 disk.sda3.in‚0 disk.dm-0.out‚46.04248046875 disk.sda1.in‚0 network.sent‚51525) disk./.usagePercent‚36.98827703611599 disk.dm-1.in‚0 disk.dm-0.in‚0.015625 disk.sda3.out‚ 1.0234375# proc.memory.rssMB‚ 2762.62109375 network.recv‚52665 disk.rbd0.in‚0 disk.nvme0c0n1.in‚0.015625 disk.rbd1.out‚0.125$ memory_percent‚97.00376987457275( proc.memory.availableMB‚ 490.90234375Þ+ÓP’!‚(:ˆ â´èÒÍá¨ù gpu.0.gpu‚99.0 gpu.0.memory‚29+ gpu.0.memoryAllocated‚57.75154113769532* gpu.0.memoryAllocatedBytes‚ 49608196096 gpu.0.temp‚42.0 gpu.0.powerWatts‚287.951' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚57.5902 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Í—;x¾"‚(:´ â´èÒªÞÂû network.sent‚57075 disk.rbd0.in‚0 disk.dm-0.out‚ 50.341796875! disk.nvme0n1.out‚ 50.341796875 disk.rbd1.out‚0.1875 disk.dm-1.out‚1.234375* proc.memory.percent‚16.861796379089355 cpu‚26.00619484907589 disk.dm-0.in‚0.078125 disk.drbd2165.in‚0& proc.memory.availableMB‚ 490.890625 proc.cpu.threads‚17 network.recv‚56695 disk.rbd0.out‚0# disk.nvme0c0n1.out‚ 50.341796875 disk.sda1.out‚0 disk.nvme0c0n1.in‚0.078125 disk.sda.in‚0 disk.nvme0n1.in‚0.078125$ memory_percent‚97.00384140014648 disk.drbd2165.out‚0 disk.rbd3.in‚0 disk.sda2.out‚0 disk.dm-1.in‚0 disk.sda3.out‚1.234375 disk.rbd3.out‚0 disk.sda2.in‚0 disk.rbd1.in‚0# proc.memory.rssMB‚ 2762.63671875) disk./.usagePercent‚36.98826756383722$ disk./.usageGB‚551.1519165039062 disk.sda1.in‚0 disk.sda3.in‚0 disk.sda.out‚1.234375f"õå3#b/ lr-AdamW/pg1"trainer/global_step2:JPi¢3$b/ lr-AdamW/pg2"trainer/global_step2:JP~N~v-%) _wandbruntime‚81  _runtime‚81lX¬iÀ&» _step‚0 & lr-AdamW/pg1‚8.475143535594357e-05 & lr-AdamW/pg2‚8.475143535594357e-05 trainer/global_step‚49 % _timestamp‚1.7842898975851145e+09 _runtime‚ 81.802337349oSòÂ'½ trainer/global_step‚49 % _timestamp‚1.7842898975851145e+09 _runtime‚ 81.802337349 _step‚0 & lr-AdamW/pg1‚8.475143535594357e-05 & lr-AdamW/pg2‚8.475143535594357e-05æF&j`(b\ 9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries"trainer/global_step2:JPÜ|˜ål)bh EDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated"trainer/global_step2:JPK±i*be BDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current"trainer/global_step2:JP~|ûüm+bi FDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak"trainer/global_step2:JPð…¢p,bl IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated"trainer/global_step2:JP¿ä6s-bo LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak"trainer/global_step2:JPIäàc.b_ b\ 9DeviceStatsMonitor.on_train_batch_start/segment.all.freed"trainer/global_step2:JPz¸,k?bg DDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak"trainer/global_step2:JP kËjh@bd ADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed"trainer/global_step2:JPKHúÊsAbo LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated"trainer/global_step2:JPkËÓoBbk HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed"trainer/global_step2:JPÚÌu#gCbc @DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated"trainer/global_step2:JP… í}nDbj GDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated"trainer/global_step2:JP ¾5tEbp MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed"trainer/global_step2:JPªÑ”„jFbf CDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated"trainer/global_step2:JP^j3SnGbj GDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak"trainer/global_step2:JPÊž²¸iHbe BDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current"trainer/global_step2:JPåsjrIbn KDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated"trainer/global_step2:JP ”hÒpJbl IDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current"trainer/global_step2:JP£€þQnKbj GDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed"trainer/global_step2:JP ‰¤ tLbp MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed"trainer/global_step2:JPiûEëxMbt QDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated"trainer/global_step2:JPh7báeNba >DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak"trainer/global_step2:JPðÜÁNqObm JDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current"trainer/global_step2:JP™òDwmPbi FDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak"trainer/global_step2:JPoÂefQbb ?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak"trainer/global_step2:JP•*'TlRbh EDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed"trainer/global_step2:JP fënSbj GDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak"trainer/global_step2:JP÷l{rnTbj GDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak"trainer/global_step2:JPä]þnpUbl IDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current"trainer/global_step2:JP¥@a¿kVbg DDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current"trainer/global_step2:JP“_à,rWbn KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated"trainer/global_step2:JPêôÝ=nXbj GDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current"trainer/global_step2:JP8ÑBfYbb ?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed"trainer/global_step2:JPDÁgZbc @DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak"trainer/global_step2:JP}_|Œv[br ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current"trainer/global_step2:JPˆÁÈh\bd ADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed"trainer/global_step2:JP%ä͇q]bm JDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current"trainer/global_step2:JPg*b^b^ ;DeviceStatsMonitor.on_train_batch_start/segment.all.current"trainer/global_step2:JPœ_‚Wk_bg DDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated"trainer/global_step2:JP­ú¥e`ba >DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak"trainer/global_step2:JP~¤´ceaba >DeviceStatsMonitor.on_train_batch_start/allocation.all.current"trainer/global_step2:JPß13ibbe BDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak"trainer/global_step2:JPÅÆcYecba >DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak"trainer/global_step2:JPÇ•\Zodbk HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed"trainer/global_step2:JP@}×nebj GDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed"trainer/global_step2:JPü²Ygfbc @DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed"trainer/global_step2:JPJ‹³ˆngbj GDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed"trainer/global_step2:JP ñOnhbj GDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak"trainer/global_step2:JPzÖéScib_ DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed"trainer/global_step2:JPjlî;iŒbd ADeviceStatsMonitor.on_train_batch_start/active.large_pool.current"trainer/global_step2:JP½A=nbi FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed"trainer/global_step2:JPÌ”§oŽbj GDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed"trainer/global_step2:JP9è¥Vtbo LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated"trainer/global_step2:JP)ÿKpbk HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed"trainer/global_step2:JP{DX,s‘bn KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated"trainer/global_step2:JPí7oq’bl IDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current"trainer/global_step2:JP?© j“be BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated"trainer/global_step2:JPý¥Üg”bb ?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak"trainer/global_step2:JP1y˜'n•bi FDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated"trainer/global_step2:JPwYèŠi–bd ADeviceStatsMonitor.on_train_batch_start/oversize_segments.current"trainer/global_step2:JPUWÕ–g—bb ?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed"trainer/global_step2:JPÔÆ×j˜be BDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current"trainer/global_step2:JPMŸÎr™bm JDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current"trainer/global_step2:JPB÷âqšbl IDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current"trainer/global_step2:JPšI#Lk›bf CDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated"trainer/global_step2:JPÛH¬/gœbb ?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed"trainer/global_step2:JPO`°¹hbc @DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed"trainer/global_step2:JPSoñ0cžb^ ;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak"trainer/global_step2:JP6ÛjŸbe BDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak"trainer/global_step2:JPdbûm bh EDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current"trainer/global_step2:JP ÛïËh¡bc @DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed"trainer/global_step2:JPOn…g¢bb ?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak"trainer/global_step2:JPœ&–f.£) _wandbruntime‚81  _runtime‚81˜™ÛÒܤO VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1679936000 GDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚14032387673088 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 4126860197376 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚132403 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚13980602740736 XHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 45531893768 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1713373184 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚14033759056384 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚502 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversizŒdEÇe_segments.current‚0 J?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚414665 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚264 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 NCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚414671 J?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚422323 K@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚269494 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11801 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9622 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚14027479233298 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1713373184 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚13986802647040 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1104240640 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345209856 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚817 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚13986776473600 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679317390 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1713373184 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47544943616 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34716160 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚14026109206536 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344598794 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1992 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚817 # _timestamp‚1.78428989758631e+09 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11744 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9628 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11744 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚13980577312768 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0YR‰§'¥¡O YIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 46956409344 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1992 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9628 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49064968192 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚14032387673088 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 4126860197376 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚132403 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚13980602740736 XHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 45531893768 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1713373184 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚14033759056384 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚502 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 J?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚414665 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚264 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 NCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚414671 J?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚422323 K@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚269494 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11801 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9622 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚14027479233298 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1713373184 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚13986802647040 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1104240640 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345209856 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚817 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚13986776473600 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679317390 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1713373184 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47544943616 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34716160 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚14026109206536 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344598794 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1992 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚817 # _timestamp‚1.78428989758631e+09 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11744 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9628 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11744 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚13980577312768 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1679936000 GDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak"trainer/global_step2:JP¬ÊÈe©b` =DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed"trainer/global_step2:JPPCÏÓhªbc @DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated"trainer/global_step2:JPpêªw«br ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated"trainer/global_step2:JP#Q`a¬b\ 9DeviceStatsMonitor.on_train_batch_end/segment.all.current"trainer/global_step2:JPåóí2h­bc @DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current"trainer/global_step2:JPñž«n®bi FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed"trainer/global_step2:JP4W›¹m¯bh EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak"trainer/global_step2:JP};I¾e°b` =DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak"trainer/global_step2:JP#`‘i±bd ADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated"trainer/global_step2:JPaÖEq²bl IDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated"trainer/global_step2:JPú:œf³ba >DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed"trainer/global_step2:JP\>~Jf´ba >DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed"trainer/global_step2:JPá$¿sjµbe BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated"trainer/global_step2:JP~\>+g¶bb ?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed"trainer/global_step2:JPN$d·b_ DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed"trainer/global_step2:JP/Ìõs»bn KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed"trainer/global_step2:JP#Í]¼bX 5DeviceStatsMonitor.on_train_batch_end/num_device_free"trainer/global_step2:JP^,Ög½bb ?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed"trainer/global_step2:JPŽm¾bh EDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak"trainer/global_step2:JPŠs%òm¿bh EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed"trainer/global_step2:JP²ËXcÀb^ ;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated"trainer/global_step2:JP_õ\iÁbd ADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed"trainer/global_step2:JPôsÜ«kÂbf CDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current"trainer/global_step2:JP/"A>`Ãb[ 8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections"trainer/global_step2:JP,o¶"hÄbc @DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current"trainer/global_step2:JP¼©5£jÅbe BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated"trainer/global_step2:JP¢eÅuÆbp MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current"trainer/global_step2:JPpeÿã^ÇbY 6DeviceStatsMonitor.on_train_batch_end/segment.all.peak"trainer/global_step2:JP­&ÇdrÈbm JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated"trainer/global_step2:JPé èÊfÉba >DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated"trainer/global_step2:JP Z¾ËhÊbc @DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current"trainer/global_step2:JP»ý£iËbd ADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated"trainer/global_step2:JPA÷DoÌbj GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated"trainer/global_step2:JPŽ”|’rÍbm JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated"trainer/global_step2:JPˆÐ)ôdÎb_ DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak"trainer/global_step2:JPs{Ó\nÒbi FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed"trainer/global_step2:JP•R+ŠiÓbd ADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current"trainer/global_step2:JPÚmÔbh EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak"trainer/global_step2:JPîÐQˆlÕbg DDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak"trainer/global_step2:JPýúP_ÖbZ 7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries"trainer/global_step2:JPçQ­¿f×ba >DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed"trainer/global_step2:JPW[ÍegØbb ?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current"trainer/global_step2:JPœ—¬kÙbf CDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed"trainer/global_step2:JP…èÄnmÚbh EDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated"trainer/global_step2:JP©“ÃërÛbm JDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak"trainer/global_step2:JP»Ê¯pkÜbf CDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated"trainer/global_step2:JP$ÊnlÝbg DDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak"trainer/global_step2:JP<*ÞqÞbl IDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated"trainer/global_step2:JPÔî˜mßbh EDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current"trainer/global_step2:JPœÙ–ejàbe BDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak"trainer/global_step2:JPÜûv:kábf CDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed"trainer/global_step2:JPžôþ\âbW 4DeviceStatsMonitor.on_train_batch_end/max_split_size"trainer/global_step2:JP=õjZdãb_ DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current"trainer/global_step2:JPTêÎ)m†bh EDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed"trainer/global_step2:JP•ìg‡bb ?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak"trainer/global_step2:JPeà¤mˆbh EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed"trainer/global_step2:JPW!›±o‰bj GDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current"trainer/global_step2:JPl‡oeŠb` =DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak"trainer/global_step2:JPÙánül‹bg DDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed"trainer/global_step2:JPˆi h]ŒbX 5DeviceStatsMonitor.on_train_batch_end/active.all.peak"trainer/global_step2:JP!›geb` =DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed"trainer/global_step2:JP`}-àpŽbk HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated"trainer/global_step2:JPä&“pbk HDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current"trainer/global_step2:JP—Ðkbf CDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current"trainer/global_step2:JPåmK§l‘bg DDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak"trainer/global_step2:JPýæõÐr’bm JDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak"trainer/global_step2:JPáœéÙ^“bY 6DeviceStatsMonitor.on_train_batch_end/num_device_alloc"trainer/global_step2:JPÀL h”bc @DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed"trainer/global_step2:JPHPe•b` =DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak"trainer/global_step2:JPBo´m–bh EDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current"trainer/global_step2:JPŒ(>æe—b` =DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak"trainer/global_step2:JP«2¡ e˜b` =DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed"trainer/global_step2:JPLÑ#DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47544944640 H=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚422871 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚14312686926336 _ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 62759310336 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1992 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚14264779300864 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49064968192 LADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚440437 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚140123 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚817 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚14311315535872 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚274810 [KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 62724601344 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚14304914450958 TEDeviceStatsMonitor.on_train_batch_end/req MŸ {uested_bytes.small_pool.peak‚ 1679317398 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1992 LADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚422871 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9636 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49064968192 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1992 ZJDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 47881452032 I>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚863314 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚817 LADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚422877 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚14264805474304 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚14264805474304 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499469591 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚14258459090944 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371390464 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1679937024 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚475 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚14264779300864 PEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚422877 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1104240640 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚14306284477832 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1713373184 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 SCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 46536235008 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ODeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371390464 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚134943 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚817 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 4205548750336 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11803 H=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚430801 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 4205951550464 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345217024 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 237518336 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1992 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345217024 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚502 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1713373184 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚14312686926336 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚139867 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0[xf³&£­M ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11803 H=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚430801 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 4205951550464 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345217024 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 237518336 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1992 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345217024 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚502 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1713373184 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚14312686926336 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚139867 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ZJDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 47799958920 VFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 46455360014 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 4143192240128 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 _runtime‚ 83.192977397 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402800128 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47544944640 H=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚422871 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚14312686926336 _ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 62759310336 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1992 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚14264779300864 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49064968192 LADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚440437 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚140123 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚817 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚14311315535872 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚274810 [KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 62724601344 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚14304914450958 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679317398 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1992 LADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚422871 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9636 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49064968192 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1992 ZJDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 47881452032 I>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚863314 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚817 LADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚422877 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚14264805474304 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚14264805474304 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499469591 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚14258459090944 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371390464 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1679937024 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚475 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚14264779300864 PEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚422877 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1104240640 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚14306284477832 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1713373184 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 SCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 46536235008 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ODeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371390464 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚134943 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚817 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 4205548750336ähA¤b< loss/FRt1/assignment_loss"trainer/global_step2:JPkL2ÿB¥b= loss/SRqqt2/detection_loss"trainer/global_step2:JPÝ&’A¦b< loss/FRt2/assignment_loss"trainer/global_step2:JPÇK`A§b< loss/FBt1/assignment_loss"trainer/global_step2:JP´£>Ï@¨b; loss/FRt2/detection_loss"trainer/global_step2:JPç¿àÀ@©b; loss/FBt1/detection_loss"trainer/global_step2:JP´7_ï7ªb2 loss/total_loss"trainer/global_step2:JP™öçC«b> loss/SRqqt1/assignment_loss"trainer/global_step2:JPןٖC¬b> loss/SRqqt2/assignment_loss"trainer/global_step2:JP©¹ßiB­b= loss/SRqqt1/detection_loss"trainer/global_step2:JPÓd×÷-®b( epoch"trainer/global_step2:JPó’æ•A¯b< loss/FBt2/assignment_loss"trainer/global_step2:JPä6Îf@°b; loss/FRt1/detection_loss"trainer/global_step2:JP·*i@±b; loss/FBt2/detection_loss"trainer/global_step2:JP—ïni.²) _wandbruntime‚83  _runtime‚83ŽY•bú³ô /loss/FRt1/assignment_loss‚4.524271019366609 1loss/SRqqt2/detection_loss‚1.1373370066869555 0loss/FRt2/assignment_loss‚5.3134649962068705 0loss/FBt1/assignment_loss‚0.5850754536879368 /loss/FRt2/detection_loss‚1.5333985700770287 -loss/FBt1/detection_loss‚34.3384024388133 %loss/total_loss‚67.99853707413189 trainer/global_step‚49 _step‚3 2loss/SRqqt1/assignment_loss‚4.1809741701890415 1loss/SRqqt2/assignment_loss‚4.057877510506034 0loss/SRqqt1/detection_loss‚4.570782498856636 epoch‚0 _runtime‚ 83.277329135 0loss/FBt2/assignment_loss‚0.2823852994870917 /loss/FRt1/detection_loss‚0.3743278396466528 /loss/FBt2/detection_loss‚7.1002402706077365 % _timestamp‚1.7842898990603802e+09Š –þ´ø 0loss/FRt2/assignment_loss‚5.3134649962068705 0loss/FBt1/assignment_loss‚0.5850754536879368 /loss/FRt2/detection_loss‚1.5333985700770287 -loss/FBt1/detection_loss‚34.3384024388133 %loss/total_loss‚67.99853707413189 trainer/global_step‚49 _step‚3 2loss/SRqqt1/assignment_loss‚4.1809741701890415 1loss/SRqqt2/assignment_loss‚4.057877510506034 0loss/SRqqt1/detection_loss‚4.570782498856636 epoch‚0 _runtime‚ 83.277329135 0loss/FBt2/assignment_loss‚0.2823852994870917 /loss/FRt1/detection_loss‚0.3743278396466528 /loss/FBt2/detection_loss‚7.1002402706077365 % _timestamp‚1.7842898990603802e+09 /loss/FRt1/assignment_loss‚4.524271019366609 1loss/SRqqt2/detection_loss‚1.1373370066869555ëòΓµ‚(:ˆ ñ´èÒ¾ÍÃù gpu.0.gpu‚59.0 gpu.0.memory‚19+ gpu.0.memoryAllocated‚57.75398254394531* gpu.0.memoryAllocatedBytes‚ 49610293248 gpu.0.temp‚43.0 gpu.0.powerWatts‚323.137' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚64.6274 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ZM÷Î̶‚(:Á ñ´èÒ»Ôþû# disk.nvme0n1.out‚55.44189453125 disk.sda1.out‚0 disk.sda3.in‚0 disk.drbd2165.in‚0$ memory_percent‚97.01085090637207 proc.cpu.threads‚17 network.recv‚63224 disk.rbd1.in‚0 disk.rbd0.out‚0 disk.dm-0.in‚0.078125) disk./.usagePercent‚36.98827396402557 disk.sda.in‚0 cpu‚25.92788485949921 disk.nvme0n1.in‚0.078125 disk.sda3.out‚ 1.48828125' proc.memory.availableMB‚ 489.7421875$ disk./.usageGB‚551.1520118713379 disk.sda2.in‚0 disk.dm-1.out‚ 1.48828125 disk.sda2.out‚0 disk.rbd1.out‚0.1875 disk.dm-1.in‚0 disk.drbd2165.out‚0 disk.sda.out‚ 1.48828125 disk.rbd0.in‚0 disk.rbd3.in‚0 disk.rbd3.out‚0# proc.memory.rssMB‚ 2762.69921875* proc.memory.percent‚16.862177848815918 disk.nvme0c0n1.in‚0.078125% disk.nvme0c0n1.out‚55.44189453125 disk.dm-0.out‚55.44189453125 disk.sda1.in‚0 network.sent‚68753ÓJû·‚(:’ €µèÒ±’€ù gpu.0.gpu‚99.0 gpu.0.memory‚30+ gpu.0.memoryAllocated‚57.75398254394531* gpu.0.memoryAllocatedBytes‚ 49610293248 gpu.0.temp‚43.0 gpu.0.powerWatts‚279.55' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚55.910000000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0œz’äɸ‚(:¾ €µèÒÚó°û disk.sda3.in‚0 disk.sda2.out‚0 cpu‚25.86250042353046 proc.cpu.threads‚17 network.sent‚116288 network.recv‚68314( disk./.usagePercent‚36.9996837078116 disk.dm-1.out‚ 2.0859375 disk.sda1.out‚0 disk.rbd3.out‚0 disk.rbd1.in‚0$ disk./.usageGB‚551.3220252990723 disk.rbd0.in‚0$ disk.nvme0n1.out‚219.26220703125$ memory_percent‚97.01383113861084 disk.sda.in‚0 disk.sda3.out‚ 2.0859375 disk.drbd2165.in‚0 disk.nvme0c0n1.in‚0.078125 disk.rbd3.in‚0& disk.nvme0c0n1.out‚219.26220703125 disk.rbd1.out‚0.25) proc.memory.percent‚16.86224937438965 disk.dm-1.in‚0" proc.memory.rssMB‚ 2762.7109375 disk.rbd0.out‚0 disk.sda1.in‚0 disk.drbd2165.out‚0 disk.nvme0n1.in‚0.078125( proc.memory.availableMB‚ 489.25390625 disk.sda.out‚ 2.0859375 disk.dm-0.in‚0.078125! disk.dm-0.out‚219.26220703125 disk.sda2.in‚0%Šð`“¹‚(:ˆ µèÒÔŠ«ø gpu.0.gpu‚99.0 gpu.0.memory‚33+ gpu.0.memoryAllocated‚57.75642395019531* gpu.0.memoryAllocatedBytes‚ 49612390400 gpu.0.temp‚43.0 gpu.0.powerWatts‚340.866' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚68.1732 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ùÝS¢Ðº‚(:Å µèÒ­êºú disk.dm-1.in‚0 disk.nvme0c0n1.in‚0.078125 disk.rbd1.out‚0.25( proc.memory.availableMB‚ 488.91796875 disk.rbd0.out‚0& disk.nvme0c0n1.out‚543.45654296875$ disk.nvme0n1.out‚543.45654296875 disk.dm-0.in‚0.078125 disk.drbd2165.in‚0 disk.rbd3.out‚0 disk.sda2.out‚0$ memory_percent‚97.01588153839111# proc.memory.rssMB‚ 2762.71484375) proc.memory.percent‚16.86227321624756 network.sent‚121764 disk.rbd0.in‚0 disk.sda3.in‚0 disk.drbd2165.out‚0 proc.cpu.threads‚17$ disk./.usageGB‚551.8148307800293! disk.dm-0.out‚543.45654296875 disk.sda1.out‚0 disk.rbd3.in‚0 disk.sda.in‚0 disk.rbd1.in‚0 network.recv‚72564* disk./.usagePercent‚37.032756297129964 disk.dm-1.out‚ 2.40234375 disk.nvme0n1.in‚0.078125 disk.sda2.in‚0 cpu‚25.814418326326283 disk.sda.out‚ 2.40234375 disk.sda1.in‚0 disk.sda3.out‚ 2.40234375´ÓYž»‚(:“ žµèÒ¨…ýø gpu.0.gpu‚99.0 gpu.0.memory‚29+ gpu.0.memoryAllocated‚57.75642395019531* gpu.0.memoryAllocatedBytes‚ 49612390400 gpu.0.temp‚43.0 gpu.0.powerWatts‚283.559' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚56.711800000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0{ϼÁ¼‚(:¶ žµèÒÌôŒû) disk./.usagePercent‚37.06668164754488 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-0.in‚0.078125 disk.sda1.in‚0 proc.cpu.threads‚17 disk.sda1.out‚0$ memory_percent‚97.01592922210693 disk.sda3.out‚2.625& disk.nvme0c0n1.out‚901.66162109375 disk.rbd1.out‚0.3125) proc.memory.percent‚16.86232089996338 disk.drbd2165.out‚0 disk.sda.in‚0 disk.dm-1.out‚2.625! disk.dm-0.out‚901.66162109375 disk.nvme0n1.in‚0.078125$ disk.nvme0n1.out‚901.66162109375 disk.rbd3.in‚0 disk.rbd0.in‚0# proc.memory.rssMB‚ 2762.72265625 disk.drbd2165.in‚0 disk.rbd3.out‚0 disk.dm-1.in‚0 cpu‚25.77175295920255 network.recv‚76608$ disk./.usageGB‚552.3203430175781 disk.sda3.in‚0 disk.nvme0c0n1.in‚0.078125 disk.rbd0.out‚0 network.sent‚127292 disk.rbd1.in‚0 disk.sda.out‚2.625( proc.memory.availableMB‚ 488.91015625fƒóDž½‚(:“ ­µèÒå—£ù gpu.0.gpu‚90.0 gpu.0.memory‚28+ gpu.0.memoryAllocated‚57.75642395019531* gpu.0.memoryAllocatedBytes‚ 49612390400 gpu.0.temp‚43.0 gpu.0.powerWatts‚253.165' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚50.632999999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÃrËÙϾ‚(:Ä ­µèÒ¹‘¾û network.recv‚80878 disk.dm-0.in‚0.078125 disk.sda3.out‚ 2.94140625 disk.rbd1.in‚0$ memory_percent‚97.01635837554932 disk.nvme0c0n1.in‚0.078125 disk.rbd0.in‚0( proc.memory.availableMB‚ 488.83984375) disk./.usagePercent‚37.10027521219543$ disk./.usageGB‚552.8209114074707 disk.sda1.out‚0& disk.nvme0c0n1.out‚1557.5771484375 disk.sda2.out‚0 proc.cpu.threads‚17 disk.dm-1.in‚0 disk.rbd0.out‚0! disk.dm-0.out‚1557.5771484375 disk.drbd2165.in‚0 disk.rbd1.out‚0.3125 disk.sda.out‚ 2.94140625 cpu‚25.73944893073324 network.sent‚132876 disk.rbd3.in‚0" proc.memory.rssMB‚ 2762.7421875 disk.rbd3.out‚0) proc.memory.percent‚16.86244010925293 disk.sda1.in‚0 disk.sda3.in‚0 disk.drbd2165.out‚0 disk.sda2.in‚0 disk.dm-1.out‚ 2.94140625 disk.nvme0n1.in‚0.078125$ disk.nvme0n1.out‚1557.5771484375 disk.sda.in‚0ïª50¿+ _wandbruntime‚160  _runtime‚160 DW0ÄÀ¾ ' lr-AdamW/pg1‚0.00017123249184160027 ' lr-AdamW/pg2‚0.00017123249184160027 trainer/global_step‚99 % _timestamp‚1.7842899762312286e+09 _runtime‚ 160.448370509 _step‚4F ½ÈÁ ' lr-AdamW/pg1‚0.00017123249184160027 ' lr-AdamW/pg2‚0.00017123249184160027 trainer/global_step‚99 % _timestamp‚1.7842899762312286e+09 _runtime‚ 160.448370509 _step‚4ž§”80Â+ _wandbruntime‚160  _runtime‚160Íîë lÃ¥O ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1993 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚27874691649536 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1715470336 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9622 YIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 93272339968 HDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47544944640 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9628 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404904448 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679317398 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49067065344 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 NCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚857697 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345209856 K@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚536742 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 120861347840 XHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 91769006876 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1312448000 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 266878464 \LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 93113605670 trainer/global_step‚99 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 7939760885760 J?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚848075 RGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚857697 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 NCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed;V€¼?‚848075 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 8060622233600 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371383296 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚27978844966912 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 J?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚824965 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚259407 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚27980216350208 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499469591 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚27874666221568 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9628 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚277339 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚27967805255206 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚818 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344598794 UEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 91927130112 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49067065344 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1993 _runtime‚ 160.449582827 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚27966435228444 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1679937024 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47544944640 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11746 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 120898161152 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11746 ODDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚537032 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚818 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚27886917836800 XHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 91927130112 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1679937024 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11803 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚27978844966912 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚546 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36813312 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1993 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 RGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚824971 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 8061027138048 \LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 93272339968 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1104240640 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚818 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚04ª¶¯'Ä©O XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚27967805255206 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚818 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344598794 UEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 91927130112 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49067065344 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1993 _runtime‚ 160.449582827 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚27966435228444 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1679937024 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47544944640 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11746 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 120898161152 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11746 ODDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚537032 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚818 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚27886917836800 XHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 91927130112 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1679937024 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11803 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚27978844966912 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚546 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36813312 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1993 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 RGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚824971 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 8061027138048 \LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 93272339968 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1104240640 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚818 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1993 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚27874691649536 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1715470336 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9622 YIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 93272339968 HDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47544944640 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9628 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404904448 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679317398 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49067065344 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 NCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚857697 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345209856 K@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚536742 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 120861347840 XHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 91769006876 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1312448000 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 266878464 \LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 93113605670 trainer/global_step‚99 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 7939760885760 J?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚848075 RGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚857697 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 NCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚848075 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 8060622233600 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371383296 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚27978844966912 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 J?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚824965 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚259407 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚27980216350208 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499469591 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚27874666221568 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9628 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚277339*!o]0Å+ _wandbruntime‚161  _runtime‚161z²Ãµ&ƯM ZJDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 94037072028 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚1689724 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11803 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 8139287354368 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371390464 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 8017617481728 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 122074769920 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49067065344 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚818 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 LADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚833177 LADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚833171 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚28152573427712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚818 _runtime‚ 161.834880348 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345217024 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚281 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚280133 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1104240640 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499469591 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49067065344 $ _timestamp‚1.784289977617488e+09 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9636 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚28257772829696 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚28152547999744 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9642 ZJDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 94197382656 VFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 92852165632 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 122037963776 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47544944640 MBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚542357 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1715470336 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1993 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1679937024 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚28164920664064 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1715470336 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404897280 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49067065344 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1715470336 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9636 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47544944640 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚833171 PEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚866189 I>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚542076 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1993 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚1699366 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371390464 PEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚833177 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36806144 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚28246610499740 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚28245240472866 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344598906 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11746 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1314545152 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679317398 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1993 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0p X¬ dzM F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚1699366 LADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚866189 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 270481920 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 92692473122 WGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 94197382656 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 8139692251648 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚1699366 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371390464 PEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚833177 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36806144 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚28246610499740 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚28245240472866 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344598906 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11746 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1314545152 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679317398 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1993 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZJDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 94037072028 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚1689724 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11803 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 8139287354368 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371390464 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 8017617481728 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 122074769920 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49067065344 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚818 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 LADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚833177 LADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚833171 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚28152573427712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚818 _runtime‚ 161.834880348 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345217024 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚281 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚280133 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1104240640 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499469591 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49067065344 $ _timestamp‚1.784289977617488e+09 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9636 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚28257772829696 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚28152547999744 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9642 ZJDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 94197382656 VFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 92852165632 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 122037963776 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 K;DeviceStatsMonitor.on_train_batch_end/active_b› ‹Ã¯ytes.all.peak‚ 47544944640 MBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚542357 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1715470336 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1993 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1679937024 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚28164920664064 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1715470336 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404897280 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49067065344 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1715470336 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9636 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47544944640 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚833171 PEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚866189 I>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚542076 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1993 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚289 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9628 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚1235271 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11746 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚531 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 407001600 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370026762 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344598794 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9628 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚11994142096384 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345209856 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚41787085373440 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1104240640 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1717567488 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚41926673644032 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚41787085373440 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚2518720 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚1283449 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 284096512 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11746 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚416679 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚38910464 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚41925302260736 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345209856 K@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚802793 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1329225216 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚386403 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚11993735094784 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 179548967424 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1717567488 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 HDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚ 238.976426887 _step‚9 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 139588270592 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1679937024 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9622 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49069162496 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1994 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚41925302260736 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚1283449 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 139588270592 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1679937024 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679317398 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371383296 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ODDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚803086 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1717567488I¡ÞÃ'Ú½O XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1717567488 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371383296 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1994 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚41908131277114 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499469591 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚819 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚1235265 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 % _timestamp‚1.7842900547590559e+09 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚416390 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚41768755130368 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚819 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9622 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚41926673644032 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚41906761250352 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 trainer/global_step‚149 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚2509092 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11803 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 138243060736 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚293 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚558 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚11814554218496 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47544944640 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚11814186127360 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 139350718778 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚819 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1994 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚289 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9628 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚1235271 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11746 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚531 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 407001600 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370026762 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344598794 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9628 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚11994142096384 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345209856 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚41787085373440 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1104240640 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1717567488 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚41926673644032 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚41787085373440 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚2518720 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚1283449 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 284096512 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11746 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚416679 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚38910464 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚41925302260736 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345209856 K@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚802793 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1329225216 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚386403 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚11993735094784 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 179548967424 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1717567488 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 HDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚ 238.976426887 _step‚9 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 139588270592 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1679937024 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9622 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49069162496 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1994 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚41925302260736 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚1283449 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 139588270592 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1679937024 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679317398 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371383296 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ODDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚803086 uQµÛ‚(:’ ‡¶èÒæÅûø gpu.0.gpu‚99.0 gpu.0.memory‚29+ gpu.0.memoryAllocated‚57.75886535644531* gpu.0.memoryAllocatedBytes‚ 49614487552 gpu.0.temp‚43.0 gpu.0.powerWatts‚272.302' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚54.46040000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ߪøŽ¢Ü‚(:Ä ‡¶èÒñè’û disk.sda.out‚ 4.55859375 disk.dm-1.out‚ 4.55859375) proc.memory.percent‚16.86258316040039 disk.drbd2165.in‚0 disk.dm-1.in‚0$ memory_percent‚97.01969623565674 cpu‚25.61871680747456) disk./.usagePercent‚37.10552797478297$ disk./.usageGB‚552.8991813659668 disk.rbd3.in‚0 disk.nvme0c0n1.in‚ 0.2421875 disk.rbd1.out‚0.5625 network.sent‚208114 network.recv‚104968% disk.nvme0c0n1.out‚1898.955078125# disk.nvme0n1.out‚1898.955078125 disk.rbd0.in‚0 disk.dm-0.out‚1898.955078125 disk.sda1.out‚0 disk.rbd1.in‚0! proc.memory.rssMB‚ 2762.765625 disk.sda1.in‚0}S s- disk.sda3.in‚0 disk.sda3.out‚ 4.55859375 disk.rbd0.out‚0 disk.dm-0.in‚ 0.2421875 disk.drbd2165.out‚0 disk.rbd3.out‚0 disk.sda2.in‚0 disk.sda.in‚0( proc.memory.availableMB‚ 488.29296875 proc.cpu.threads‚17 disk.nvme0n1.in‚ 0.2421875 disk.sda2.out‚0ïG% 0Ý+ _wandbruntime‚240  _runtime‚240±_tÊ&ÞÄM UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1679937024 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 180726174208 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49069162496 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚2525776 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚42205601513984 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1994 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚42046662336512 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚819 MBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚808379 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚419432 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1717567488 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚1291941 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚42185566494774 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚42204230123520 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370026874 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49069162496 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚388943 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679317398 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9636 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1679937024 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49069162496 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345217024 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47544944640 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚2535418 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚1282305 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚808085 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚531 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚38903296 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371390464 ODeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499469591 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚42046636908544 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚819 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 140513313280 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 406994432 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1994 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 % _timestamp‚1.7842900561445913e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11803 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚2535418 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9636 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚12072400806400 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚11891674632192 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1717567488 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1994 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345217024 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 139168096256 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11803 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1994 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚2525776 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0ncäRÎ&ßÈM WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344598906 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1717567488 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚42065062027264 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚419142 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 180765077504 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚819 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚1243477 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚12072807800832 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚1243477 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1329225216 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚42186936521648 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9642 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚1243471 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚1243471 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11746 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚558 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚388947 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499469591 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚42046636908544 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚819 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 140513313280 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 406994432 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1994 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 % _timestamp‚1.7842900561445913e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11803 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚2535418 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9636 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚12072400806400 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚11891674632192 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1717567488 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1994 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345217024 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 139168096256 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11803 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1994 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚2525776 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1679937024 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 180726174208 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49069162496 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚2525776 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚42205601513984 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1994 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚42046662336512 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚819 MBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚808379 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚419432 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1717567488 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚1291941 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚42185566494774 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚42204230123520 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370026874 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49069162496 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚388943 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679317398 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9636 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1679937024 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49069162496 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345217024 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47544944640 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚2535418 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚1282305 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚808085 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚531 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚38903296 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371390464 ODeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚55847087272260 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚1709201 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚55687226736640 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚55687200563200 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚15688611368960 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 237901240320 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9628 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345209856 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚1645571 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚1645565 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚55662844039168 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 $ _timestamp‚1.784290133268384e+09 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47544944640 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344598794 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371383296 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚55871759554560 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚306 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚41007616 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚1645571 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚554447 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499469591 XEDeviceStaªåÔõ€tsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚55873130937856 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚1645565 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345209856 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚1699579 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _step‚13 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 184558991360 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1679937024 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚554749 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚55848457299022 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11803 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 HDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11746 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9628 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371383296 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚1068156 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚3354772 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚513407 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 185587831886 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚1709201 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚302 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚55873130937856 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11803 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 trainer/global_step‚199 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 185904201216 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9622 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 _runtime‚ 317.485692461 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚55871759554560 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1104240640 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 409098752 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚55662869467136 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995G¡ÂÅ'ò¿O trainer/global_step‚199 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 185904201216 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9622 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 _runtime‚ 317.485692461 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚55871759554560 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1104240640 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 409098752 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚55662869467136 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚3345144 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1679937024 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47544944640 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚513403 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚55847087272260 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚1709201 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚55687226736640 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚55687200563200 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚15688611368960 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 237901240320 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9628 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345209856 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚1645571 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚1645565 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚55662844039168 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 $ _timestamp‚1.784290133268384e+09 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47544944640 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344598794 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371383296 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚55871759554560 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚306 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚41007616 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚1645571 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚554447 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499469591 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚55873130937856 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚1645565 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345209856 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚1699579 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _step‚13 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 184558991360 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1679937024 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚554749 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚55848457299022 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11803 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 HDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11746 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9628 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371383296 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚1068156 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚3354772 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚513407 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 185587831886 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚1709201 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚302 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚55873130937856 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11803 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 4'¼|0ó+ _wandbruntime‚318  _runtime‚31811߉Ì&ôÆM P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1073116 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370026874 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚55940751245312 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚1073414 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345217024 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚1708057 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1104240640 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 239063956480 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚1653771 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 185484026880 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9636 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 239022956032 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚55940725817344 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚16005531921408 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371390464 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47544944640 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47544944640 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚15766467964928 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚15766099873792 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚3361828 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 _runtime‚ 318.870249898 _step‚14 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499469591 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚56152058807808 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 trainer/global_step‚199 ODeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚3371470 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚298 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 186829243904 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11803 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚1653771 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚515943 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345217024 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚56127262543556 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚557467 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚1653777 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚294 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679317398 % _timestamp‚1.7842901346529043e+09 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚55965229563904 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 185484026880 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚56150687417344 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚16005122829824 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚1717693 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371390464 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚56125892516682 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648{BœõÊM ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11803 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚55965203390464 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1679937024 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚3371470 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚298 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 186829243904 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11803 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚1653771 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚515943 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345217024 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚56127262543556 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚557467 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚1653777 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚294 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679317398 % _timestamp‚1.7842901346529043e+09 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚55965229563904 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 185484026880 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚56150687417344 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚16005122829824 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚1717693 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371390464 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚56125892516682 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1073116 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370026874 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚55940751245312 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚1073414 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345217024 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚1708057 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1104240640 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 239063956480 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚1653771 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 185484026880 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9636 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 239022956032 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚55940725817344 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚16005531921408 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371390464 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47544944640 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47544944640 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚15766467964928 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚15766099873792 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚3361828 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 _runtime‚ 318.870249898 _step‚14 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499469591 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚56152058807808 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 trainer/global_step‚199 ODeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47544944640 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 230874921984 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345209856 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚69587341926400 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9628 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 % _timestamp‚1.7842902121001031e+09 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚2134953 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚69587368099840 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚2125331 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11803 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚1333003 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚69556932947968 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344598794 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚270 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚640407 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚19859421707264 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚2055865 HDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚2055871 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚2055871 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11746 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚266 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚19859826611712 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36813312 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 231824944994 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚69587341926400 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345209856 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499469591 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚4181196 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9622 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚2125331 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 _runtime‚ 396.317473997 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 232220131840 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1679937024 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1104240640 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚19563404701696 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370026762 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 trainer/global_step‚249 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9628 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚69587368099840 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚69818216848384 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9622 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47544944640 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404904448 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11746 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚69819588231680 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371383296 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚19563036610560 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 230480346200 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820‡Ù(xw$ˆÀO OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚2055871 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚2055871 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11746 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚266 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚19859826611712 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36813312 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 231824944994 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚69587341926400 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345209856 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499469591 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚4181196 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9622 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚2125331 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 _runtime‚ 396.317473997 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 232220131840 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1679937024 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1104240640 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚19563404701696 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370026762 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 trainer/global_step‚249 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9628 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚69587368099840 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚69818216848384 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9622 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47544944640 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404904448 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11746 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚69819588231680 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371383296 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚19563036610560 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 230480346200 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371383296 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚2134953 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚69787413294168 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47544944640 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 230874921984 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345209856 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚69587341926400 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9628 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 % _timestamp‚1.7842902121001031e+09 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚2134953 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚69587368099840 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚2125331 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11803 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚1333003 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚69556932947968 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344598794 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚270 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚640407 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚19859421707264 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚2055865 HDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0IuwM0‰+ _wandbruntime‚397  _runtime‚397o¼ËÌ&ŠÆM K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47544944640 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 233145174528 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11803 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚695083 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚695347 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚18 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1338026 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚1338294 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚69834840154112 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚642947 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚4197880 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371390464 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47544944640 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1104240640 trainer/global_step‚249 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚69865370927104 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚70097144711168 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11746 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 231403812446 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 231799957504 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1679937024 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9636 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9636 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚2143445 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 231799957504 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚70066218538590 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344598906 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚2064071 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚264 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404897280 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499469591 _runtime‚ 397.702427799 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚69865370927104 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚70098516101632 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚4207522 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚2143445 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚69865344753664 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7842902134850523e+09 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚19938493096448 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚4197880 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7842902134850523e+09 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚19938493096448 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚4197880 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚18 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1338026 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚1338294 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚69834840154112 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚642947 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚4197880 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371390464 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47544944640 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1104240640 trainer/global_step‚249 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚69865370927104 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚70097144711168 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11746 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 231403812446 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 231799957504 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1679937024 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9636 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9636 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚2143445 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 231799957504 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚70066218538590 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344598906 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚2064071 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚264 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404897280 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499469591 _runtime‚ 397.702427799 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚69865370927104 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚70098516101632 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚4207522 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚2143445 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚69865344753664 EíS“‚(:ˆ ¬·èÒ…Æù gpu.0.gpu‚94.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚282.495' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚56.499 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0·¶d΂(:à ¬·èÒÓÑ×û disk.rbd0.in‚0 disk.rbd0.out‚0) proc.memory.percent‚16.86272621154785 disk.sda1.out‚0 disk.sda3.in‚0& disk.nvme0c0n1.out‚1981.1142578125 disk.sda.out‚ 6.98046875 disk.dm-1.out‚ 6.98046875* disk./.usagePercent‚37.107734759728125 disk.dm-0.in‚ 1.19921875! disk.dm-0.out‚1981.1142578125 disk.sda3.out‚ 6.98046875 disk.rbd3.in‚0 disk.rbd3.out‚0 proc.cpu.threads‚17 disk.drbd2165.in‚0 disk.sda2.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚1 disk.sda.in‚0$ proc.memory.availableMB‚488.9375 disk.nvme0c0n1.in‚ 1.19921875 disk.dm-1.in‚0$ memory_percent‚97.01576232910156" proc.memory.rssMB‚ 2762.7890625 cpu‚25.52912541460073 network.sent‚372536 disk.drbd2165.out‚0 disk.nvme0n1.in‚ 1.19921875 disk.sda2.out‚0$ disk./.usageGB‚552.9320640563965 network.recv‚150366 disk.sda1.in‚0$ disk.nvme0n1.out‚1981.1142578125É㸅ž‘‚(:“ »·èÒþ°ãø gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚236.479' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚47.29580000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ìHšÓ’‚(:È »·èÒ…èˆû* disk./.usagePercent‚37.108181236867836 disk.dm-1.out‚ 7.21484375 disk.rbd3.out‚0& disk.nvme0c0n1.out‚1993.7998046875 disk.sda2.out‚0 disk.rbd0.in‚0 disk.drbd2165.out‚0 network.recv‚155205 disk.rbd1.in‚0 disk.rbd0.out‚0 disk.sda1.out‚0 network.sent‚420084! disk.dm-0.out‚1993.7998046875 disk.sda1.in‚0 disk.nvme0c0n1.in‚ 1.23046875 disk.sda.out‚ 7.21484375 disk.drbd2165.in‚0 disk.nvme0n1.in‚ 1.23046875 disk.sda2.in‚0 disk.sda.in‚0 disk.dm-1.in‚0 disk.sda3.out‚ 7.21484375$ memory_percent‚97.01664447784424* proc.memory.percent‚16.862821578979492 proc.cpu.threads‚17$ disk./.usageGB‚552.9387168884277 disk.dm-0.in‚ 1.23046875 disk.sda3.in‚0( proc.memory.availableMB‚ 488.79296875 disk.rbd1.out‚1 disk.rbd3.in‚0$ disk.nvme0n1.out‚1993.7998046875" proc.memory.rssMB‚ 2762.8046875 cpu‚25.52539390790332aÖQL”“‚(:‰ Ê·èÒ‹Õìø gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚277.663' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.5326 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0àÀJæД‚(:Å Ê·èÒ¿‘ÿú disk.dm-1.in‚0 disk.dm-0.in‚ 1.52734375 disk.sda1.out‚0 disk.rbd1.out‚1.0625# memory_percent‚97.0181941986084 proc.cpu.threads‚17* disk./.usagePercent‚37.108562176078784 disk.sda1.in‚0 disk.drbd2165.in‚0" disk.nvme0n1.out‚ 2004.43359375 disk.sda2.out‚0 network.sent‚426601 network.recv‚157927 disk.rbd0.out‚0 disk.drbd2165.out‚0 disk.sda.in‚0 disk.sda.out‚ 7.39453125 disk.rbd0.in‚0 disk.dm-0.out‚ 2004.43359375* proc.memory.percent‚16.862821578979492 disk.sda3.in‚0 disk.rbd3.in‚0$ disk.nvme0c0n1.out‚ 2004.43359375" proc.memory.rssMB‚ 2762.8046875 cpu‚25.520206680069006 disk.dm-1.out‚ 7.39453125 disk.sda3.out‚ 7.39453125 disk.rbd3.out‚0 disk.nvme0c0n1.in‚ 1.52734375 disk.nvme0n1.in‚ 1.52734375# disk./.usageGB‚552.944393157959 disk.rbd1.in‚0 disk.sda2.in‚0' proc.memory.availableMB‚ 488.5390625k3“•‚(:ˆ Ù·èÒæ‘Úø gpu.0.gpu‚62.0 gpu.0.memory‚19, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚289.215' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚57.843 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¸« DÚ–‚(:Ï Ù·èÒî—ýú disk.rbd3.out‚0% disk.nvme0n1.out‚2013.78564453125( proc.memory.availableMB‚ 488.28515625$ disk./.usageGB‚552.9512214660645 disk.sda.in‚0 disk.sda1.in‚0" proc.memory.rssMB‚ 2762.8046875 network.sent‚429919 disk.sda1.out‚0 disk.sda3.out‚ 7.62890625 disk.drbd2165.in‚0' disk.nvme0c0n1.out‚2013.78564453125 disk.sda2.out‚0) disk./.usagePercent‚37.10902042956508 disk.dm-0.in‚ 1.55859375" disk.dm-0.out‚2013.78564453125 proc.cpu.threads‚17 disk.nvme0c0n1.in‚ 1.55859375 cpu‚25.51587100421619 network.recv‚161704 disk.dm-1.out‚ 7.62890625 disk.nvme0n1.in‚ 1.55859375 disk.sda2.in‚0 disk.rbd1.in‚0$ memory_percent‚97.01974391937256 disk.drbd2165.out‚0* proc.memory.percent‚16.862821578979492 disk.sda.out‚ 7.62890625 disk.rbd0.out‚0 disk.rbd1.out‚1.0625 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.sda3.in‚0 disk.rbd3.in‚0†ÉžŸ—‚(:” è·èÒ‘Ýø gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚271.258' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚54.251599999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0£ìVƒÞ˜‚(:Ó è·èÒ¯ïèú disk.rbd0.out‚0( proc.memory.availableMB‚ 488.47265625 disk.sda1.in‚0 disk.sda2.in‚0% disk.nvme0n1.out‚2020.14892578125 disk.sda2.out‚0 network.sent‚437351 network.recv‚165464 disk.dm-1.in‚0" disk.dm-0.out‚2020.14892578125 disk.nvme0c0n1.in‚ 1.85546875 disk.nvme0n1.in‚ 1.85546875 disk.dm-0.in‚ 1.85546875 disk.sda3.out‚ 7.88671875 disk.rbd3.out‚0' disk.nvme0c0n1.out‚2020.14892578125 cpu‚25.511303662096118 disk.sda.out‚ 7.88671875 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.drbd2165.out‚0" proc.memory.rssMB‚ 2762.8046875* proc.memory.percent‚16.862821578979492 proc.cpu.threads‚17 disk.rbd1.out‚ 1.19140625 disk.dm-1.out‚ 7.88671875 disk.rbd0.in‚0$ memory_percent‚97.01859951019287) disk./.usagePercent‚37.10947100282534 disk.rbd3.in‚0# disk./.usageGB‚552.957935333252 disk.rbd1.in‚0 disk.sda.in‚0 disk.sda3.in‚0{ýÞ½0™+ _wandbruntime‚474  _runtime‚4743œ/8Äš¾ & lr-AdamW/pg1‚0.0005171567177842271 & lr-AdamW/pg2‚0.0005171567177842271 trainer/global_step‚299 % _timestamp‚1.7842902907460766e+09 _runtime‚ 474.964583245 _step‚20¢ƒùț _runtime‚ 474.964583245 _step‚20 & lr-AdamW/pg1‚0.0005171567177842271 & lr-AdamW/pg2‚0.0005171567177842271 trainer/global_step‚299 % _timestamp‚1.7842902907460766e+09ËÝ2ª0œ+ _wandbruntime‚474  _runtime‚474ë@˜ »O ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 278536062464 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚767407 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚83451047284736 HDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11746 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345209856 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11746 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345209856 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 355785999360 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytesñúÖ¬.all.peak‚ 49071259648 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚83766045525504 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1679937024 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚1598048 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 trainer/global_step‚299 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚5026876 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚23793650658816 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47544944640 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚5017248 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9628 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499469591 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚83764674142208 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚83487509463040 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚2560705 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9622 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371383296 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚83766045525504 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 % _timestamp‚1.7842902907468448e+09 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1104240640 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 _step‚21 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9622 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚83487509463040 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371383296 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34716160 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 276717459308 HDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚83764674142208 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚83487483289600 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚83727739316076 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679317398 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 _runtime‚ 474.96471852 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚2551083 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 277190852608 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚2466165 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚1598309 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚830645 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚83729109342838 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚2560705 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47544944640 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚261 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚767403 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚2466171 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚23437829943296 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344598794Ðÿ$µÅ'ž¿O Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚83764674142208 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚83487483289600 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚83727739316076 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679317398 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 _runtime‚ 474.96471852 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚2551083 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 277190852608 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚2466165 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚1598309 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚830645 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚83729109342838 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚2560705 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47544944640 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚261 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚767403 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚2466171 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚23437829943296 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344598794 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 278536062464 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚767407 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚83451047284736 HDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11746 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345209856 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11746 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345209856 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 355785999360 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚83766045525504 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1679937024 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚1598048 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 trainer/global_step‚299 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚5026876 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚23793650658816 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47544944640 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚5017248 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9628 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499469591 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚83764674142208 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚83487509463040 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚2560705 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9622 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371383296 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚83766045525504 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 % _timestamp‚1.7842902907468448e+09 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1104240640 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 _step‚21 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9622 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚83487509463040 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371383296 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34716160 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 276717459308 HDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371390464 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚84044973395456 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚84044973395456 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1603347 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚769947 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499469591 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚5033932 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 279461105152 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345217024 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚5033932 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚2569197 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370026874 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 278115888128 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 279461105152 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚251 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34708992 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47544944640 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371390464 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚83765512290304 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚2559561 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚769943 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 357007329792 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679317398 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47544944640 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚2569197 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚23872325777920 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚23515318448128 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344598906 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚5043574 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 278115888128 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402800128 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 356972620800 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚2474377 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1104240640 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚84006544560498 _runtime‚ 476.348439974 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9642 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 % _timestamp‚1.7842902921310933e+09 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1679937024 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚5043574 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11746 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚2474371 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚23871922977792 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47544944640 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371390464 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚83765512290304 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚2559561 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚769943 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 357007329792 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679317398 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47544944640 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚2569197 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚23872325777920 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚23515318448128 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344598906 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚5043574 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 278115888128 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402800128 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 356972620800 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚2474377 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1104240640 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚84006544560498 _runtime‚ 476.348439974 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9642 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 % _timestamp‚1.7842902921310933e+09 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1679937024 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚5043574 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11746 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚2474371 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚23871922977792 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371390464 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚84044973395456 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚84044973395456 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1603347 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚769947 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499469591 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚5033932 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 279461105152 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345217024 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚5033932 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚2569197 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370026874 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 278115888128 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 279461105152 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚251 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34708992œ+«0¢+ _wandbruntime‚476  _runtime‚476=£ú /loss/FRt1/assignment_loss‚4.219168359397823 &loss/total_loss‚29.673997505836738 trainer/global_step‚299 _step‚23 0loss/FRt2/assignment_loss‚3.5028211691996436 2loss/SRqqt1/assignment_loss‚1.8309974926184878 .loss/FBt1/detection_loss‚3.236075725170857 _runtime‚ 476.435583363 1loss/SRqqt2/assignment_loss‚1.467057258049647 /loss/FRt1/detection_loss‚0.4659548381676105 /loss/FRt2/detection_loss‚1.4040069939581665 1loss/SRqqt2/detection_loss‚0.8556899798630426 .loss/FBt2/detection_loss‚9.185853412305867 epoch‚0 % _timestamp‚1.7842902922186713e+09 1loss/FBt1/assignment_loss‚0.34363213279504945 1loss/FBt2/assignment_loss‚0.42999636611946496 0loss/SRqqt1/detection_loss‚2.732743778191079¤; I¤þ 1loss/FBt2/assignment_loss‚0.42999636611946496 0loss/SRqqt1/detection_loss‚2.732743778191079 /loss/FRt1/assignment_loss‚4.219168359397823 &loss/total_loss‚29.673997505836738 trainer/global_step‚299 _step‚23 0loss/FRt2/assignment_loss‚3.5028211691996436 2loss/SRqqt1/assignment_loss‚1.8309974926184878 .loss/FBt1/detection_loss‚3.236075725170857 _runtime‚ 476.435583363 1loss/SRqqt2/assignment_loss‚1.467057258049647 /loss/FRt1/detection_loss‚0.4659548381676105 /loss/FRt2/detection_loss‚1.4040069939581665 1loss/SRqqt2/detection_loss‚0.8556899798630426 .loss/FBt2/detection_loss‚9.185853412305867 epoch‚0 % _timestamp‚1.7842902922186713e+09 1loss/FBt1/assignment_loss‚0.34363213279504945Õ®¯”¥‚(:‰ ÷·èÒÄ•ù gpu.0.gpu‚94.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚332.266' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚66.4532 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÉÀ9Ò¦‚(:Ç ÷·èÒ¨ˆ¯û disk.nvme0c0n1.in‚ 1.85546875 disk.rbd0.in‚0 disk.sda3.out‚ 8.1953125 disk.nvme0n1.in‚ 1.85546875 disk.sda1.in‚0 disk.sda.out‚ 8.1953125% disk.nvme0c0n1.out‚2032.806640625 disk.rbd1.in‚0 proc.cpu.threads‚17 disk.sda3.in‚0# proc.memory.rssMB‚ 2762.80859375 disk.dm-0.in‚ 1.85546875 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.sda2.in‚0 disk.rbd1.out‚ 1.19140625 network.recv‚169642 disk.sda1.out‚0# disk.nvme0n1.out‚2032.806640625& proc.memory.availableMB‚ 488.421875* proc.memory.percent‚16.862845420837402) disk./.usagePercent‚37.10969091329726$ disk./.usageGB‚552.9612121582031 disk.sda.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 8.1953125 disk.rbd0.out‚0 disk.drbd2165.in‚0 disk.sda2.out‚0 network.sent‚453679 disk.dm-0.out‚2032.806640625 disk.drbd2165.out‚0# memory_percent‚97.0189094543457 cpu‚25.5084255312549¶z­A’§‚(:‡ †¸èÒãêÝø gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚271.19' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.238 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0LAݨ‚(:Ò †¸èÒ¹óéú disk.rbd3.in‚0 disk.rbd3.out‚0 disk.nvme0n1.in‚ 1.88671875 disk.sda2.out‚0* disk./.usagePercent‚37.109922856123276 disk.rbd0.out‚0 disk.sda.out‚ 8.31640625# proc.memory.rssMB‚ 2762.80859375* proc.memory.percent‚16.862845420837402! disk.dm-0.out‚2042.9794921875 disk.sda1.in‚0 disk.sda3.in‚0 disk.sda2.in‚0 disk.rbd1.out‚ 1.26953125 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.dm-0.in‚ 1.88671875 disk.dm-1.out‚ 8.31640625 cpu‚25.50389484658825 proc.cpu.threads‚17 disk.drbd2165.in‚0 disk.rbd1.in‚0 disk.sda3.out‚ 8.31640625$ disk.nvme0n1.out‚2042.9794921875( proc.memory.availableMB‚ 488.44921875$ disk./.usageGB‚552.9646682739258 disk.drbd2165.out‚0 disk.nvme0c0n1.in‚ 1.88671875 disk.sda.in‚0$ memory_percent‚97.01874256134033 network.sent‚488192 disk.sda1.out‚0& disk.nvme0c0n1.out‚2042.9794921875 network.recv‚174361ìîõï’©‚(:‡ •¸èÒÓ„Øø gpu.0.gpu‚3.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚337.393' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚67.4786 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ת‚(:Ì •¸èÒþâú disk.sda.out‚ 9.64453125$ disk.nvme0c0n1.out‚ 2067.81640625 disk.nvme0n1.in‚ 2.32421875 disk.dm-1.in‚0 proc.cpu.threads‚17" disk.nvme0n1.out‚ 2067.81640625$ memory_percent‚97.01879024505615# proc.memory.rssMB‚ 2762.80859375 disk.nvme0c0n1.in‚ 2.32421875( proc.memory.availableMB‚ 488.44140625 network.sent‚493643* disk./.usagePercent‚37.110671166145856 disk.sda3.in‚0 disk.sda3.out‚ 9.64453125 disk.drbd2165.in‚0 disk.rbd3.out‚0 disk.sda2.out‚0 disk.dm-0.out‚ 2067.81640625 disk.rbd1.out‚ 1.26953125 disk.dm-1.out‚ 9.64453125* proc.memory.percent‚16.862845420837402 network.recv‚178458 disk.drbd2165.out‚0 disk.rbd1.in‚0 disk.sda.in‚0$ disk./.usageGB‚552.9758186340332 disk.dm-0.in‚ 2.32421875 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 cpu‚25.50144685283035 disk.rbd3.in‚0Öf‡d”«‚(:‰ ¤¸èÒ¼†ù gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚270.126' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.0252 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÝÆÛÿì‚(:¸ ¤¸èÒ–ú¹ú disk.nvme0c0n1.in‚ 2.35546875 network.sent‚499049# disk./.usageGB‚552.979377746582$ disk.nvme0n1.out‚2083.0576171875 disk.rbd0.out‚0 disk.sda3.out‚13 disk.sda2.out‚0 proc.cpu.threads‚17 disk.sda.out‚13 disk.dm-1.out‚13 disk.dm-0.in‚ 2.35546875 disk.sda3.in‚0 disk.nvme0n1.in‚ 2.35546875$ memory_percent‚97.01879024505615 disk.rbd1.out‚ 1.39453125 disk.rbd0.in‚0& disk.nvme0c0n1.out‚2083.0576171875 disk.sda2.in‚0 disk.dm-1.in‚0! disk.dm-0.out‚2083.0576171875 disk.drbd2165.in‚0 disk.rbd1.in‚0( proc.memory.availableMB‚ 488.44140625( disk./.usagePercent‚37.1109100211753 network.recv‚182377 disk.sda1.in‚0# proc.memory.rssMB‚ 2762.80859375* proc.memory.percent‚16.862845420837402 cpu‚25.497334447258083 disk.sda.in‚0 disk.sda1.out‚0 disk.drbd2165.out‚0 disk.rbd3.in‚0 disk.rbd3.out‚0]½@”­‚(:‰ ³¸èÒì—ˆù gpu.0.gpu‚99.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚333.229' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚66.6458 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0sÙ®‚(:Î ³¸èҭĬû$ disk./.usageGB‚552.9857559204102 disk.sda2.out‚0 disk.sda.out‚ 14.00390625 disk.rbd0.in‚0$ memory_percent‚97.01881408691406 disk.drbd2165.out‚0 network.recv‚186251) disk./.usagePercent‚37.11133806577255 disk.rbd3.out‚0 disk.nvme0n1.in‚ 2.65234375 disk.dm-1.in‚0 disk.dm-0.in‚ 2.65234375 disk.sda1.in‚0 network.sent‚506465# disk.nvme0n1.out‚2099.888671875 disk.rbd1.in‚0 disk.rbd1.out‚ 1.39453125 disk.rbd0.out‚0 disk.drbd2165.in‚0* proc.memory.percent‚16.862845420837402 cpu‚25.495100870963096 disk.nvme0c0n1.in‚ 2.65234375 disk.dm-0.out‚2099.888671875 disk.sda1.out‚0 disk.sda3.in‚0 proc.cpu.threads‚17 disk.rbd3.in‚0% disk.nvme0c0n1.out‚2099.888671875 disk.dm-1.out‚ 14.00390625 disk.sda2.in‚0 disk.sda.in‚0# proc.memory.rssMB‚ 2762.80859375 disk.sda3.out‚ 14.00390625$ proc.memory.availableMB‚488.4375ØmÀ0¯+ _wandbruntime‚553  _runtime‚553VKöð½ trainer/global_step‚349 % _timestamp‚1.7842903694298828e+09 _runtime‚ 553.64702482 _step‚24 & lr-AdamW/pg1‚0.0006036377742698839 & lr-AdamW/pg2‚0.0006036377742698839«=ñDZÁ & lr-AdamW/pg2‚0.0006036377742698839 trainer/global_step‚349 % _timestamp‚1.7842903694298828e+09 _runtime‚ 553.64702482 _step‚24 & lr-AdamW/pg1‚0.0006036377742698839Ž 0²+ _wandbruntime‚553  _runtime‚553ðÞ%Á'³»O C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9628 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚1864262 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚97387650826240 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9622 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 400710144 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚32619008 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚5853300 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚97711131436032 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11746 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚894403 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚241 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 324299171210 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11803 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚97387650826240 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 324851993088 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11746 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371383296 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚969614 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚27312255184896 _step‚25 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345209856 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚97711131436032 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47544944640 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 415551204352 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344598794 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 324851993088 HDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9628 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚894407 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499469591 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11803 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371383296 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47544944640 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚1864017 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚ 553.648426468 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 HDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 400710144 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚32619008 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚5853300 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚97711131436032 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11746 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚894403 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚241 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 324299171210 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11803 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚97387650826240 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 324851993088 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11746 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371383296 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚969614 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚27312255184896 _step‚25 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345209856 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚97711131436032 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47544944640 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 415551204352 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344598794 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 324851993088 HDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9628 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚894407 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499469591 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11803 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371383296 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47544944640 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚1864017 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚ 553.648426468 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 H8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚2884671 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 _step‚26 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679317398 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚232 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47544944640 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371390464 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9642 ODeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499469591 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 324431818752 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11803 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚236 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚972633 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚27389375598592 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47544944640 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 416746173440 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1869344 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚97622992543744 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371390464 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚97623017971712 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 400702976 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11803 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚2985313 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚5879626 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚97946870582406 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚1869580 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1104240640 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370026874 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 323878038662 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 _runtime‚ 555.04161291 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚97990059298816 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 325777035776 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11746 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚27389743689728 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚97948240609280 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚97665627480064 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚27806121772032 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚972401 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 416778785280 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9636 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚97665653653504 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345217024 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 324431818752 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 325777035776 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚2985313 % _timestamp‚1.7842903708242583e+09 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚97991430689280 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚2994949 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499469591 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 324431818752 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11803 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚236 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚972633 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚27389375598592 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47544944640 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 416746173440 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1869344 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚97622992543744 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371390464 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚97623017971712 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 400702976 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11803 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚2985313 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚5879626 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚97946870582406 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚1869580 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1104240640 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370026874 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 323878038662 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 _runtime‚ 555.04161291 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚97990059298816 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 325777035776 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11746 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚27389743689728 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚97948240609280 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚97665627480064 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚27806121772032 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚972401 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 416778785280 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9636 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚97665653653504 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345217024 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 324431818752 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 325777035776 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚2985313 % _timestamp‚1.7842903708242583e+09 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚97991430689280 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚2994949 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚2884671 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 _step‚26 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679317398 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚232 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47544944640 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371390464 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9642 OÓG”¿‚(:‰ à¸èÒéè´ø gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚295.976' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚59.1952 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0)”ÄPÑÀ‚(:Æ à¸èÒÌɧú disk.drbd2165.out‚0 disk.rbd0.out‚0 disk.dm-0.in‚ 2.98046875" disk.dm-0.out‚2137.13427734375* proc.memory.percent‚16.862869262695312 disk.sda3.out‚17.9375 disk.nvme0c0n1.in‚ 2.98046875' disk.nvme0c0n1.out‚2137.13427734375 disk.rbd1.in‚0 disk.rbd1.out‚ 1.51953125 disk.sda1.in‚0( proc.memory.availableMB‚ 488.05078125 network.recv‚198734) disk./.usagePercent‚37.11292531248485 disk.sda.out‚17.9375 disk.dm-1.out‚17.9375 disk.sda1.out‚0$ memory_percent‚97.02117443084717 proc.memory.rssMB‚ 2762.8125 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.nvme0n1.in‚ 2.98046875 disk.sda3.in‚0 cpu‚25.48651233319177 network.sent‚583825 disk.sda2.in‚0 disk.rbd0.in‚0 proc.cpu.threads‚17% disk.nvme0n1.out‚2137.13427734375 disk.sda2.out‚0 disk.sda.in‚0 disk.drbd2165.in‚0# disk./.usageGB‚553.009407043457 disk.dm-1.in‚0"úÛ•”Á‚(:‰ ï¸èÒΣÏø gpu.0.gpu‚90.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚297.228' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚59.4456 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0蕨=Ò‚(:Ç ï¸èÒ§“Ñú disk.sda2.in‚0 disk.dm-1.out‚ 20.71484375 proc.memory.rssMB‚ 2762.8125* proc.memory.percent‚16.862869262695312 disk.sda2.out‚0 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.sda3.in‚0 cpu‚25.484000977864394 network.recv‚203953 disk.rbd1.out‚ 1.51953125 disk.sda.out‚ 20.71484375 disk.sda1.in‚0 disk.sda1.out‚0$ memory_percent‚97.02110290527344$ proc.memory.availableMB‚488.0625# disk./.usageGB‚553.010498046875 disk.rbd1.in‚0 disk.dm-0.out‚ 2164.46484375 network.sent‚589826* disk./.usagePercent‚37.112998530639636$ disk.nvme0c0n1.out‚ 2164.46484375 disk.nvme0n1.in‚ 3.01171875 disk.drbd2165.in‚0 disk.nvme0c0n1.in‚ 3.01171875 disk.sda.in‚0 disk.drbd2165.out‚0 disk.rbd3.in‚0" disk.nvme0n1.out‚ 2164.46484375 disk.dm-0.in‚ 3.01171875 disk.sda3.out‚ 20.71484375 disk.rbd3.out‚0 disk.rbd0.out‚0 proc.cpu.threads‚17Ñ$(”Â(:‰ þ¸èÒÙˆ·ø gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚270.729' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.1458 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0S¹UØÄ‚(:Í þ¸èÒ˜¸Öú# memory_percent‚97.0226526260376( proc.memory.availableMB‚ 487.80859375* proc.memory.percent‚16.862916946411133' disk.nvme0c0n1.out‚2176.02197265625 network.recv‚208125 disk.rbd1.out‚ 1.58203125 disk.sda1.out‚0 disk.sda3.in‚0 disk.drbd2165.in‚0 disk.rbd1.in‚0 disk.sda.in‚0 disk.dm-1.in‚0 disk.dm-0.in‚ 3.30859375 disk.drbd2165.out‚0 disk.nvme0c0n1.in‚ 3.30859375" proc.memory.rssMB‚ 2762.8203125 disk.rbd0.in‚0 disk./.usageGB‚ 553.015625 disk.dm-1.out‚ 21.8671875 disk.sda1.in‚0 disk.rbd3.out‚0 cpu‚25.482416202726025% disk.nvme0n1.out‚2176.02197265625 disk.sda.out‚ 21.8671875 disk.rbd0.out‚0" disk.dm-0.out‚2176.02197265625* disk./.usagePercent‚37.113342604765656 disk.sda2.out‚0 disk.nvme0n1.in‚ 3.30859375 proc.cpu.threads‚17 disk.sda3.out‚ 21.8671875 network.sent‚597398 disk.sda2.in‚0 disk.rbd3.in‚0NC×ϔł(:‰ ¹èÒÿ‹Ãø gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚258.062' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚51.6124 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0!€…[ãÆ‚(:Ø ¹èÒˆÒáú' disk.nvme0c0n1.out‚2191.58837890625 disk.sda2.out‚0 network.sent‚600428 disk.dm-1.out‚ 22.65234375 disk.drbd2165.in‚0# proc.memory.rssMB‚ 2762.82421875 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.rbd1.in‚0 disk.dm-0.in‚ 3.33984375 disk.rbd3.out‚0 disk.nvme0c0n1.in‚ 3.33984375$ memory_percent‚97.02260494232178( proc.memory.availableMB‚ 487.81640625 network.recv‚211964" disk.dm-0.out‚2191.58837890625 disk.sda1.in‚0 disk.rbd3.in‚0 disk.rbd1.out‚ 1.58203125 cpu‚25.479750861317413$ disk./.usageGB‚553.0224723815918 disk.sda.out‚ 22.65234375% disk.nvme0n1.out‚2191.58837890625 disk.sda2.in‚0) disk./.usagePercent‚37.11380213828962 disk.sda.in‚0 disk.sda1.out‚0 disk.drbd2165.out‚0 disk.nvme0n1.in‚ 3.33984375* proc.memory.percent‚16.862940788269043 disk.dm-1.in‚0 disk.sda3.in‚0 disk.sda3.out‚ 22.65234375 disk.rbd0.out‚0çÏ^ð0Ç+ _wandbruntime‚632  _runtime‚632%”¢PÄȾ & lr-AdamW/pg1‚0.0006901188307555405 & lr-AdamW/pg2‚0.0006901188307555405 trainer/global_step‚399 % _timestamp‚1.7842904480460415e+09 _runtime‚ 632.263916754 _step‚28z‹¬ÈÉ _runtime‚ 632.263916754 _step‚28 & lr-AdamW/pg1‚0.0006901188307555405 & lr-AdamW/pg2‚0.0006901188307555405 trainer/global_step‚399 % _timestamp‚1.7842904480460415e+09ý¸˜0Ê+ _wandbruntime‚632  _runtime‚632,ZÎO:ËÌO PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚2130741 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚31662262267392 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 475581840896 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679317398 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345209856 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370026762 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 473WÊé9˜$51595008 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 trainer/global_step‚399 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 371167923712 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚3412209 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚2130489 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344598794 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371383296 HDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9628 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚3286765 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚111608391359892 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 369191685524 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚6689352 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚111657588729856 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚111658960113152 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚111287766016000 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 HDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11746 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 371167923712 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 475549221888 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11746 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚32619008 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚111239225102336 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚111287792189440 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚6698980 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚111657588729856 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚111287766016000 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 369822713856 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11803 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚3286765 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1679937024 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 369822713856 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9622 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1104240640 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚111239199674368 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚252 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚111658960113152uJåúÖ'ÌÐO ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 371167923712 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚3412209 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚2130489 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344598794 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371383296 HDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9628 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚3286765 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚111608391359892 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 369191685524 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚6689352 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚111657588729856 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚111658960113152 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚111287766016000 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 HDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11746 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 371167923712 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 475549221888 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11746 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚4 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚32619008 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚111239225102336 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚111287792189440 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚6698980 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚111657588729856 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚111287766016000 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 369822713856 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11803 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚3286765 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1679937024 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 369822713856 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9622 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1104240640 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚111239199674368 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚252 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚111658960113152 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚2130741 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚31662262267392 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 475581840896 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679317398 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345209856 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370026762 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 trainer/global_step‚399I>c0Í+ _wandbruntime‚633  _runtime‚633@~µÛ&ÎÕM WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚111565768843264 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499469591 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚111517106880512 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚31740538887168 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚31264168931328 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 _runtime‚ 633.649059565 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9636 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1112111 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 371459750676 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679317398 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚3411065 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚6706036 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345217024 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1023943 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402800128 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚3294971 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚3411065 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370026874 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚111887196604314 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚6715678 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚3294971 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚111888566631188 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 370747749376 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚2135812 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344598906 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 370747749376 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1023947 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1111869 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 476772755968 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 370115151770 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371390464 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚3294977 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚30 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11803 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11746 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚3420701 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1679937024 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚111517081452544 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 372092966400 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 476738046976 trainer/global_step‚399 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚6715678 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 372092966400 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34708992 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚3294977 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47544944640 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚111565768843264 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚31740941687296 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47544944640 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371390464 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚31263800840192 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345217024 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚6715678 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚3294971 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚111888566631188 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 370747749376 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚2135812 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_trainch‹Ÿ{_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344598906 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚4 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 370747749376 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1023947 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1111869 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 476772755968 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 370115151770 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371390464 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚3294977 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚30 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11803 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11746 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚3420701 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1679937024 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚111517081452544 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 372092966400 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 476738046976 trainer/global_step‚399 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚6715678 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 372092966400 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34708992 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚3294977 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47544944640 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚111565768843264 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚31740941687296 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47544944640 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371390464 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚31263800840192 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345217024 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499469591 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚111517106880512 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚31740538887168 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚31264168931328 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 _runtime‚ 633.649059565 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9636 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1112111 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 371459750676 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679317398 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚3411065 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚6706036Fà€´0Ð+ _wandbruntime‚633  _runtime‚633<”þ,ÿÑù _runtime‚ 633.735371932 _step‚31 1loss/FBt2/assignment_loss‚0.03930525761305554 /loss/FRt1/detection_loss‚0.5528122658097774 .loss/FBt1/detection_loss‚4.105961565362317 /loss/FBt2/detection_loss‚2.4754350456808294 %loss/total_loss‚21.40196449034476 /loss/FRt1/assignment_loss‚3.928914210816537 2loss/SRqqt1/assignment_loss‚1.5963301321366028 2loss/SRqqt2/assignment_loss‚1.4668453738829237 % _timestamp‚1.7842904495184402e+09 epoch‚0 /loss/FRt2/assignment_loss‚2.873156870341605 1loss/FBt1/assignment_loss‚0.13177270231601804 1loss/SRqqt2/detection_loss‚0.7365277281942214 trainer/global_step‚399 .loss/FRt2/detection_loss‚1.433762815013857 0loss/SRqqt1/detection_loss‚2.061140523177017n¤iðÒý 1loss/SRqqt2/detection_loss‚0.7365277281942214 trainer/global_step‚399 .loss/FRt2/detection_loss‚1.433762815013857 0loss/SRqqt1/detection_loss‚2.061140523177017 _runtime‚ 633.735371932 _step‚31 1loss/FBt2/assignment_loss‚0.03930525761305554 /loss/FRt1/detection_loss‚0.5528122658097774 .loss/FBt1/detection_loss‚4.105961565362317 /loss/FBt2/detection_loss‚2.4754350456808294 %loss/total_loss‚21.40196449034476 /loss/FRt1/assignment_loss‚3.928914210816537 2loss/SRqqt1/assignment_loss‚1.5963301321366028 2loss/SRqqt2/assignment_loss‚1.4668453738829237 % _timestamp‚1.7842904495184402e+09 epoch‚0 /loss/FRt2/assignment_loss‚2.873156870341605 1loss/FBt1/assignment_loss‚0.13177270231601804áIžÃ•Ó‚(:Š œ¹èÒ†ñÈú gpu.0.gpu‚100.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚327.729' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚65.5458 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ò{0ßÔ‚(:Ô œ¹èÒãî°þ' disk.nvme0c0n1.out‚2195.68701171875 disk.nvme0n1.in‚ 3.33984375 disk.rbd3.out‚0$ memory_percent‚97.50452041625977$ disk./.usageGB‚553.0259780883789 disk.dm-1.in‚0& proc.memory.availableMB‚ 408.859375 cpu‚25.51994571258764 proc.cpu.threads‚17 disk.sda.in‚0 disk.dm-1.out‚ 22.94140625# proc.memory.rssMB‚ 2844.20703125) disk./.usagePercent‚37.11403740921359 disk.rbd1.in‚0 disk.rbd0.out‚0 disk.dm-0.in‚ 3.33984375" disk.dm-0.out‚2195.68701171875 network.sent‚605501 disk.sda3.in‚0 disk.sda3.out‚ 22.94140625% disk.nvme0n1.out‚2195.68701171875 disk.sda.out‚ 22.94140625) proc.memory.percent‚17.35966205596924 disk.rbd3.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd0.in‚0 network.recv‚215461 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.nvme0c0n1.in‚ 3.33984375 disk.rbd1.out‚ 1.64453125V§pí•Õ‚(:Š «¹èÒ™Ö¼ø gpu.0.gpu‚100.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚322.601' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚64.5202 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÊúUoÜÖ‚(:Ñ «¹èÒ«…·ü disk.nvme0n1.in‚ 3.63671875 disk.sda2.in‚0 disk.sda3.out‚ 23.10546875 cpu‚25.582228716249222) disk./.usagePercent‚37.11447108597659 disk.dm-1.in‚0 disk.rbd0.out‚0" disk.dm-0.out‚2209.07763671875 disk.sda1.out‚0 proc.memory.rssMB‚ 2841.0625* proc.memory.percent‚17.340469360351562 disk.dm-1.out‚ 23.10546875 disk.rbd0.in‚0$ disk./.usageGB‚553.0324401855469 disk.drbd2165.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 1.64453125 network.sent‚652608 disk.rbd3.out‚0 disk.nvme0c0n1.in‚ 3.63671875 disk.sda.out‚ 23.10546875 disk.dm-0.in‚ 3.63671875% proc.memory.availableMB‚ 409.46875 disk.rbd3.in‚0' disk.nvme0c0n1.out‚2209.07763671875 disk.sda3.in‚0$ memory_percent‚97.50080108642578 disk.sda2.out‚0 network.recv‚220115 disk.drbd2165.out‚0 disk.sda.in‚0 proc.cpu.threads‚17% disk.nvme0n1.out‚2209.07763671875 disk.sda1.in‚0//n<×b7 particle/specificity"trainer/global_step2:JP•ÔZ¥HØbC Purity/*FRt1SRqqt1FBt/FBt_purity"trainer/global_step2:JP<î¬ëNÙbI &Purity/1FRt*SRqqt*FBt/event_proportion"trainer/global_step2:JP½`£JÚbE "Purity/0FRt1SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPqaÚ@JÛbE "Purity/1FRt1SRqqt*FBt/event_purity"trainer/global_step2:JP‡V‚ÄHÜbC Purity/1FRt2SRqqt0FBt/FRt_purity"trainer/global_step2:JP‚Œ­ÁJÝbE "Purity/2FRt2SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP—ßêžNÞbI &Purity/2FRt*SRqqt2FBt/event_proportion"trainer/global_step2:JPP,ÆàJßbE "Purity/*FRt*SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPüM@FHàbC Purity/*FRt2SRqqt*FBt/FRt_purity"trainer/global_step2:JPL¶õJábE "Purity/0FRt*SRqqt2FBt/event_purity"trainer/global_step2:JPÔç‰ñHâbC Purity/1FRt0SRqqt2FBt/FBt_purity"trainer/global_step2:JP}x¾GJãbE "Purity/2FRt1SRqqt2FBt/event_purity"trainer/global_step2:JP-lNäbI &Purity/*FRt1SRqqt0FBt/event_proportion"trainer/global_step2:JP¤´Í[NåbI &Purity/*FRt2SRqqt2FBt/event_proportion"trainer/global_step2:JPÄoUÀJæbE "Purity/0FRt1SRqqt1FBt/event_purity"trainer/global_step2:JPâ‚?NçbI &Purity/2FRt2SRqqt2FBt/event_proportion"trainer/global_step2:JPÿí$JèbE "Purity/*FRt1SRqqt0FBt/event_purity"trainer/global_step2:JPXL¥:JébE "Purity/0FRt1SRqqt*FBt/event_purity"trainer/global_step2:JP8êÎJêbE "Purity/1FRt1SRqqt1FBt/SRqqt_purity"trainer/global_step2:JP‚ú}„HëbC Purity/*FRt0SRqqt*FBt/FBt_purity"trainer/global_step2:JPŒàäµHìbC Purity/*FRt1SRqqt2FBt/FBt_purity"trainer/global_step2:JP³yNíbI &Purity/0FRt2SRqqt1FBt/event_proportion"trainer/global_step2:JP¦pPHîbC Purity/1FRt1SRqqt*FBt/FBt_purity"trainer/global_step2:JPÀ]ݪJïbE "Purity/1FRt1SRqqt0FBt/event_purity"trainer/global_step2:JPØuq¶JðbE "Purity/0FRt1SRqqt2FBt/event_purity"trainer/global_step2:JP®·MHñbC Purity/2FRt1SRqqt1FBt/FRt_purity"trainer/global_step2:JP@…×]NòbI &Purity/*FRt1SRqqt1FBt/event_proportion"trainer/global_step2:JPbiNóbI &Purity/0FRt*SRqqt0FBt/event_proportion"trainer/global_step2:JP‹I=0NôbI &Purity/0FRt*SRqqt2FBt/event_proportion"trainer/global_step2:JPF SÚJõbE "Purity/0FRt2SRqqt*FBt/event_purity"trainer/global_step2:JP”ÓNöbI &Purity/0FRt2SRqqt0FBt/event_proportion"trainer/global_step2:JP§­GYN÷bI &Purity/1FRt1SRqqt2FBt/event_proportion"trainer/global_step2:JP½ÒNJøbE "Purity/2FRt*SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP}}NùbI &Purity/1FRt2SRqqt2FBt/event_proportion"trainer/global_step2:JPãxÏHúbC Purity/2FRt0SRqqt*FBt/FRt_purity"trainer/global_step2:JP4¯¶^HûbC Purity/*FRt*SRqqt*FBt/FBt_purity"trainer/global_step2:JPzôNübI &Purity/0FRt1SRqqt1FBt/event_proportion"trainer/global_step2:JP­dJýbE "Purity/0FRt2SRqqt0FBt/event_purity"trainer/global_step2:JP—déJþbE "Purity/1FRt*SRqqt0FBt/event_purity"trainer/global_step2:JPuCëHÿbC Purity/1FRt0SRqqt1FBt/FBt_purity"trainer/global_step2:JPžé J€bE "Purity/2FRt2SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPñ˜U_;b6 loss/val_total_loss"trainer/global_step2:JPPfcéJ‚bE "Purity/1FRt2SRqqt*FBt/event_purity"trainer/global_step2:JPÂ>QNƒbI &Purity/2FRt1SRqqt1FBt/event_proportion"trainer/global_step2:JP„ù/d;„b6 jet/accuracy_4_of_4"trainer/global_step2:JP”"8(J…bE "Purity/2FRt0SRqqt2FBt/event_purity"trainer/global_step2:JP4¶|H†bC Purity/2FRt1SRqqt2FBt/FRt_purity"trainer/global_step2:JPK›80;‡b6 jet/accuracy_1_of_5"trainer/global_step2:JP±—@ˆb; particle/accuracy_2_of_5"trainer/global_step2:JPÌØL£J‰bE "Purity/0FRt*SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP‰pHŠbC Purity/0FRt0SRqqt1FBt/FBt_purity"trainer/global_step2:JP ýuÝH‹bC Purity/0FRt2SRqqt1FBt/FBt_purity"trainer/global_step2:JP¼«p[JŒbE "Purity/2FRt0SRqqt1FBt/event_purity"trainer/global_step2:JPM³èÞNbI &Purity/2FRt0SRqqt*FBt/event_proportion"trainer/global_step2:JPHÍ,JŽbE "Purity/*FRt2SRqqt*FBt/event_purity"trainer/global_step2:JP;gTHbC Purity/1FRt2SRqqt*FBt/FBt_purity"trainer/global_step2:JPGmfÍJbE "Purity/*FRt*SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPÄ<\H‘bC Purity/0FRt*SRqqt*FBt/FBt_purity"trainer/global_step2:JPâ}YëH’bC Purity/0FRt0SRqqt2FBt/FBt_purity"trainer/global_step2:JP¨á¯'H“bC Purity/1FRt*SRqqt*FBt/FRt_purity"trainer/global_step2:JPðDÿH”bC Purity/*FRt*SRqqt0FBt/FRt_purity"trainer/global_step2:JPiA,øJ•bE "Purity/*FRt0SRqqt1FBt/event_purity"trainer/global_step2:JPµ˜FßJ–bE "Purity/0FRt*SRqqt0FBt/event_purity"trainer/global_step2:JPõ¨†«N—bI &Purity/*FRt0SRqqt*FBt/event_proportion"trainer/global_step2:JP·$4ÖN˜bI &Purity/0FRt0SRqqt1FBt/event_proportion"trainer/global_step2:JP@k?î;™b6 jet/accuracy_2_of_4"trainer/global_step2:JPf¦HšbC Purity/2FRt*SRqqt2FBt/FRt_purity"trainer/global_step2:JP~=çH›bC Purity/*FRt0SRqqt0FBt/FRt_purity"trainer/global_step2:JP„žNœbI &Purity/*FRt0SRqqt1FBt/event_proportion"trainer/global_step2:JPO‘+ÄHbC Purity/1FRt1SRqqt0FBt/FRt_purity"trainer/global_step2:JPU$q†HžbC Purity/2FRt*SRqqt1FBt/FRt_purity"trainer/global_step2:JPܱJëbE "Purity/*FRt1SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP?TŸ&JìbE "Purity/0FRt1SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPáÓáHíbC Purity/1FRt1SRqqt*FBt/FRt_purity"trainer/global_step2:JPÇèNîbI &Purity/1FRt2SRqqt0FBt/event_proportion"trainer/global_step2:JP#ëTéNïbI &Purity/*FRt0SRqqt2FBt/event_proportion"trainer/global_step2:JP½O]¨JðbE "Purity/*FRt1SRqqt1FBt/event_purity"trainer/global_step2:JPÊ ÿNñbI &Purity/0FRt0SRqqt*FBt/event_proportion"trainer/global_step2:JPÆNŒ+JòbE "Purity/1FRt*SRqqt1FBt/event_purity"trainer/global_step2:JPt±îJóbE "Purity/1FRt1SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP¼sDJôbE "Purity/2FRt1SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPŒŠ9õb4 particle/accuracy"trainer/global_step2:JP«Xð¾JöbE "Purity/*FRt*SRqqt*FBt/event_purity"trainer/global_step2:JPýà@J÷bE "Purity/0FRt*SRqqt1FBt/event_purity"trainer/global_step2:JPö;øb6 jet/accuracy_3_of_4"trainer/global_step2:JPŽJ–NùbI &Purity/*FRt*SRqqt2FBt/event_proportion"trainer/global_step2:JP\»ÆHúbC Purity/*FRt2SRqqt*FBt/FBt_purity"trainer/global_step2:JPK%ëÎNûbI &Purity/2FRt0SRqqt1FBt/event_proportion"trainer/global_step2:JPa^o5JübE "Purity/1FRt2SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPòa ÎJýbE "Purity/0FRt0SRqqt*FBt/event_purity"trainer/global_step2:JP†Í,JþbE "Purity/0FRt1SRqqt0FBt/event_purity"trainer/global_step2:JPJÌUSJÿbE "Purity/1FRt1SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPI„÷J€bE "Purity/2FRt2SRqqt*FBt/event_purity"trainer/global_step2:JPâÌ«µ@b; particle/accuracy_1_of_2"trainer/global_step2:JPzýH‚bC Purity/2FRt*SRqqt2FBt/FBt_purity"trainer/global_step2:JP{V‹JƒbE "Purity/*FRt*SRqqt2FBt/SRqqt_purity"trainer/global_step2:JP¤ì~3N„bI &Purity/0FRt1SRqqt0FBt/event_proportion"trainer/global_step2:JPU¢»©J…bE "Purity/*FRt0SRqqt*FBt/event_purity"trainer/global_step2:JPT†ÜJ†bE "Purity/1FRt1SRqqt2FBt/event_purity"trainer/global_step2:JP”g«J‡bE "Purity/2FRt1SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP·k׫NˆbI &Purity/0FRt2SRqqt*FBt/event_proportion"trainer/global_step2:JPÛ‡ç¿N‰bI &Purity/1FRt0SRqqt0FBt/event_proportion"trainer/global_step2:JP.HŠbC Purity/1FRt0SRqqt1FBt/FRt_purity"trainer/global_step2:JPž Quƒ@›b; particle/accuracy_4_of_4"trainer/global_step2:JPLv@œb; particle/accuracy_5_of_5"trainer/global_step2:JP±ZÁ±JbE "Purity/*FRt1SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPÉ yJžbE "Purity/*FRt2SRqqt1FBt/SRqqt_purity"trainer/global_step2:JP ß‹HŸbC Purity/0FRt1SRqqt*FBt/FBt_purity"trainer/global_step2:JPµ[Æ N bI &Purity/0FRt1SRqqt2FBt/event_proportion"trainer/global_step2:JPx°ÃJ¡bE "Purity/1FRt2SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP«¾r–N¢bI &Purity/2FRt1SRqqt2FBt/event_proportion"trainer/global_step2:JP®[DtH£bC Purity/2FRt1SRqqt2FBt/FBt_purity"trainer/global_step2:JP™<9H¤bC Purity/*FRt2SRqqt0FBt/FRt_purity"trainer/global_step2:JPyi¼1J¥bE "Purity/0FRt*SRqqt*FBt/event_purity"trainer/global_step2:JP›v:J¦bE "Purity/2FRt*SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPkš%óH§bC Purity/*FRt*SRqqt2FBt/FBt_purity"trainer/global_step2:JPG5ó N¨bI &Purity/2FRt*SRqqt1FBt/event_proportion"trainer/global_step2:JP»ŸJ©bE "Purity/2FRt*SRqqt2FBt/event_purity"trainer/global_step2:JPÉDüJªbE "Purity/*FRt0SRqqt2FBt/event_purity"trainer/global_step2:JP IJ«bE "Purity/*FRt1SRqqt2FBt/event_purity"trainer/global_step2:JP% ÔCJ¬bE "Purity/0FRt0SRqqt1FBt/event_purity"trainer/global_step2:JPEÎÕN­bI &Purity/0FRt1SRqqt*FBt/event_proportion"trainer/global_step2:JP:)[ZH®bC Purity/*FRt0SRqqt2FBt/FBt_purity"trainer/global_step2:JP‡ÿÅÚH¯bC Purity/*FRt1SRqqt*FBt/FBt_purity"trainer/global_step2:JPqF#SJ°bE "Purity/*FRt1SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPY¥×J±bE "Purity/1FRt*SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPXkЇJ²bE "Purity/1FRt*SRqqt2FBt/SRqqt_purity"trainer/global_step2:JP›úeH³bC Purity/1FRt1SRqqt2FBt/FBt_purity"trainer/global_step2:JPp ¿J´bE "Purity/2FRt1SRqqt1FBt/event_purity"trainer/global_step2:JP6K²Å;µb6 jet/accuracy_5_of_5"trainer/global_step2:JP JéJ¶bE "Purity/*FRt0SRqqt0FBt/event_purity"trainer/global_step2:JPZüZ«J·bE "Purity/0FRt0SRqqt0FBt/event_purity"trainer/global_step2:JPí>}v;¸b6 jet/accuracy_2_of_5"trainer/global_step2:JP…¸þ-H¹bC Purity/1FRt*SRqqt1FBt/FRt_purity"trainer/global_step2:JPq“ÏNºbI &Purity/1FRt1SRqqt1FBt/event_proportion"trainer/global_step2:JPQ¢ ;J»bE "Purity/2FRt1SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPyò@©H¼bC Purity/*FRt*SRqqt1FBt/FBt_purity"trainer/global_step2:JP®qcJ½bE "Purity/0FRt*SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPºŸH¾bC Purity/1FRt*SRqqt2FBt/FRt_purity"trainer/global_step2:JPc #†H¿bC Purity/1FRt0SRqqt*FBt/FBt_purity"trainer/global_step2:JPx­¥@Àb; particle/accuracy_1_of_1"trainer/global_step2:JPTpQ×JÁbE "Purity/*FRt2SRqqt0FBt/event_purity"trainer/global_step2:JP4Œ jNÂbI &Purity/0FRt*SRqqt1FBt/event_proportion"trainer/global_step2:JPò’RNÃbI &Purity/0FRt2SRqqt2FBt/event_proportion"trainer/global_step2:JP>iÉ NÄbI &Purity/1FRt*SRqqt1FBt/event_proportion"trainer/global_step2:JP_­ÖJJÅbE "Purity/1FRt1SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPr‡½JÆbE "Purity/2FRt0SRqqt0FBt/event_purity"trainer/global_step2:JP™"ð·JÇbE "Purity/*FRt*SRqqt2FBt/event_purity"trainer/global_step2:JP:-fHÈbC Purity/*FRt0SRqqt1FBt/FBt_purity"trainer/global_step2:JP©™ãÄJÉbE "Purity/1FRt0SRqqt1FBt/event_purity"trainer/global_step2:JPm¢>›HÊbC Purity/1FRt1SRqqt1FBt/FBt_purity"trainer/global_step2:JPZ ËJËbE "Purity/1FRt2SRqqt1FBt/event_purity"trainer/global_step2:JPˆ¯îHÌbC Purity/2FRt*SRqqt*FBt/FRt_purity"trainer/global_step2:JPùb?NÍbI &Purity/2FRt*SRqqt0FBt/event_proportion"trainer/global_step2:JP¹Œ GÎbB validation_average_jet_accuracy"trainer/global_step2:JP›Ä HÏbC Purity/*FRt*SRqqt1FBt/FRt_purity"trainer/global_step2:JPz±}JÐbE "Purity/*FRt1SRqqt*FBt/event_purity"trainer/global_step2:JPQÓ ‹NÑbI &Purity/2FRt*SRqqt*FBt/event_proportion"trainer/global_step2:JP„ VýNÒbI &Purity/2FRt2SRqqt0FBt/event_proportion"trainer/global_step2:JPÒ¾B;HÓbC Purity/*FRt*SRqqt2FBt/FRt_purity"trainer/global_step2:JP—hJÔbE "Purity/0FRt0SRqqt2FBt/event_purity"trainer/global_step2:JP3¤LHÕbC Purity/1FRt*SRqqt1FBt/FBt_purity"trainer/global_step2:JPÏ&­–JÖbE "Purity/1FRt0SRqqt2FBt/event_purity"trainer/global_step2:JPš·ôH×bC Purity/2FRt1SRqqt*FBt/FBt_purity"trainer/global_step2:JPbúg’@Øb; particle/accuracy_3_of_3"trainer/global_step2:JPé-çìHÙbC Purity/0FRt1SRqqt2FBt/FBt_purity"trainer/global_step2:JPç8îÔHÚbC Purity/*FRt0SRqqt1FBt/FRt_purity"trainer/global_step2:JP\²•NÛbI &Purity/1FRt0SRqqt*FBt/event_proportion"trainer/global_step2:JPº×ø÷HÜbC Purity/1FRt1SRqqt1FBt/FRt_purity"trainer/global_step2:JPñÌü}JÝbE "Purity/2FRt*SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPH¼›oHÞbC Purity/2FRt*SRqqt0FBt/FRt_purity"trainer/global_step2:JPÙÐÊ;ßb6 jet/accuracy_1_of_2"trainer/global_step2:JP„ý.©;àb6 jet/accuracy_3_of_3"trainer/global_step2:JP8¶¯JábE "Purity/0FRt1SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPÙ“.}HâbC Purity/*FRt0SRqqt2FBt/FRt_purity"trainer/global_step2:JPôñ×JãbE "Purity/1FRt*SRqqt*FBt/event_purity"trainer/global_step2:JP„|8^HäbC Purity/1FRt1SRqqt2FBt/FRt_purity"trainer/global_step2:JPèj´åbê{ë¦FE "Purity/1FRt2SRqqt0FBt/event_purity"trainer/global_step2:JPC«˜ @æb; particle/accuracy_3_of_4"trainer/global_step2:JP–Éo;çb6 jet/accuracy_4_of_5"trainer/global_step2:JP:°eøHèbC Purity/0FRt1SRqqt1FBt/FBt_purity"trainer/global_step2:JPzœ¿NNébI &Purity/1FRt*SRqqt0FBt/event_proportion"trainer/global_step2:JPVëŽHêbC Purity/1FRt*SRqqt2FBt/FBt_purity"trainer/global_step2:JPÞ}Á–;ëb6 jet/accuracy_3_of_5"trainer/global_step2:JP¾ü(JìbE "Purity/*FRt*SRqqt1FBt/event_purity"trainer/global_step2:JPúGÓ”NíbI &Purity/1FRt1SRqqt*FBt/event_proportion"trainer/global_step2:JPU@È JîbE "Purity/0FRt*SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPô–åÑJïbE "Purity/2FRt1SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPé.‰NðbI &Purity/*FRt2SRqqt*FBt/event_proportion"trainer/global_step2:JPÑj§ÂJñbE "Purity/2FRt*SRqqt1FBt/event_purity"trainer/global_step2:JP*»àŸ;òb6 jet/accuracy_1_of_1"trainer/global_step2:JPŸƒtF@ób; particle/accuracy_3_of_5"trainer/global_step2:JPåÜMNôbI &Purity/*FRt*SRqqt*FBt/event_proportion"trainer/global_step2:JP!:¸ªJõbE "Purity/*FRt*SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP|ÊàûNöbI &Purity/*FRt2SRqqt1FBt/event_proportion"trainer/global_step2:JPt(³„0÷+ _wandbruntime‚660  _runtime‚660胾0>øª| ("Purity/*FRt2SRqqt1FBt/event_purity‚0 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 9 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.023568956181406975 ,jet/accuracy_2_of_2‚0.051340989768505096 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.1119791716337204 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.06134391948580742 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.24758881330490112 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0012454213574528694 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 *jet/accuracy_1_of_3‚0.5794954895973206 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.24599874019622803 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.05970693379640579 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.2245798408985138 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_2_of_3‚1 +particle/sensitivity‚0.5386502146720886 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.02613985165953636 :"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.21978260576725006 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.2325982302427292 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.1518324911594391 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.14358274638652802 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.03936391696333885 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.2199791669845581 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.5648148059844971 (particle/accuracy‚0.7958868145942688 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.10412339866161346 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.7692508697509766 ,jet/accuracy_3_of_4‚0.012017344124615192 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9444444179534912 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4000000059604645 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9647400975227356 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.2221650779247284 :"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.23205289244651794 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 particle/accuracy_1_of_2‚1 + Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9375 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.48148149251937866 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.09550683945417404 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 8"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.180070161819458 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.11540386825799942 9 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.013564098626375198 particle/accuracy_1_of_5‚1 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.2266857922077179 6 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.977568507194519 8 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.20591457188129425 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 8 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.30000001192092896 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.15483619272708893 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.11966816335916519 8"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.238542377948761 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.11154573410749435 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 /particle/accuracy_4_of_4‚0.9640492796897888 particle/accuracy_5_of_5‚0 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.48148149251937866 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.3402777910232544 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9692595601081848 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 :"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.24165447056293488 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 ( Purity/2FRt1SRqqt2FBt/FBt_purity‚0.5 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.13088122010231018 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.3088071346282959 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9337990283966064 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 ("Purity/2FRt*SRqqt2FBt/event_purity‚0 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.6806460618972778 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9856082201004028 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9469119310379028 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9686850309371948 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.4476345479488373 :"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.48566871881484985 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.6666666865348816 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.6111111044883728 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 jet/accuracy_5_of_5‚0 ;"Purity/*FRt0SRqqt0FBt/event_purity‚0.022662905976176262 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 jet/accuracy_2_of_5‚0 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.1174616813659668 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.2050388902425766 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9861127138137817 :"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.40500640869140625 8 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.21266062557697296 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9836961627006531 particle/accuracy_1_of_1‚1 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.020417217165231705 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.6666666865348816 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.002311334013938904 trainer/global_step‚399 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.6374609470367432 6 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.986639142036438 ;"Purity/1FRt0SRqqt1FBt/event_purity‚0.058048348873853683 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9922696352005005 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 9 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.057470113039016724 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 7validation_average_jet_accuracy‚0.15392984449863434 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.11763635277748108 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.1273384392261505 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.2142857164144516 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.8743829131126404 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9883795976638794 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.12622827291488647 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9736841917037964 /particle/accuracy_3_of_3‚0.9984673857688904 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.7692307829856873 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.11785677820444107 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.11727948486804962 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.1990366280078888 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.0536186508834362 )jet/accuracy_1_of_2‚0.408681720495224 -jet/accuracy_3_of_3‚0.0062730517238378525 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.3461538553237915 8 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.20612244307994843 ;"Purity/1FRt*SRqqt*FBt/event_purity‚0.018360117450356483 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.3333333432674408 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_3_of_4‚1 jet/accuracy_4_of_5‚0 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9805300831794739 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 6 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.938066840171814 jet/accuracy_3_of_5‚0 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.5546444654464722 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.3461538553237915 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 ("Purity/2FRt*SRqqt1FBt/event_purity‚0 +jet/accuracy_1_of_1‚0.12319111824035645 particle/accuracy_3_of_5‚1 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 :"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.22368642687797546 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 +particle/specificity‚0.8662404417991638 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9852121472358704 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.2221650779247284 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.001655382919125259 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.15700840950012207 :"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.19296228885650635 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.2329237312078476 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.12987680733203888 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.8250102996826172 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9696428179740906 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.2339056134223938 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.1267469972372055 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.2221384346485138 :"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.49039649963378906 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9743033647537231 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.6990740299224854 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9706740379333496 <"Purity/1FRt1SRqqt0FBt/event_purity‚0.0014020794769749045 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.15740741789340973 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 ;"Purity/0FRt2SRqqt*FBt/event_purity‚0.044705379754304886 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 :"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.18736037611961365 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.01913236640393734 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9729613065719604 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 ;"Purity/0FRt2SRqqt0FBt/event_purity‚0.045758143067359924 8"Purity/1FRt*SRqqt0FBt/event_purity‚0.017346216365695 7 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.9875974059104919 :"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.19296228885650635 *loss/val_total_loss‚21.275293947607985 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 jet/accuracy_4_of_4‚0 ("Purity/2FRt0SRqqt2FBt/event_purity‚0 & Purity/2FRt1SRqqt2FBt/FRt_purity‚0 jet/accuracy_1_of_5‚1 particle/accuracy_2_of_5‚1 :"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.23538056015968323 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9856082201004028 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt0SRqqt1FBt/event_purity‚0 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.020065905526280403 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.8999999761581421 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.4341323971748352 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9704977869987488 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9371915459632874 9 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.061898402869701385 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.05896572396159172 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.7533417344093323 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.20404060184955597 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 +jet/accuracy_2_of_4‚0.12858335673809052 , Purity/2FRt*SRqqt2FBt/FRt_purity‚0.21875 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.02279055491089821 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.15309497714042664 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.14041666686534882 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.5648148059844971 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.00213325722143054 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 epoch‚0 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.23675914108753204 6 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.987500011920929 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 :"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.42148062586784363 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.02668113447725773 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 % _timestamp‚1.7842904759626365e+09 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.09922734647989273 _runtime‚ 660.180403157 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.14459779858589172 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.02497893013060093 <"Purity/1FRt1SRqqt1FBt/event_purity‚0.0064745196141302586 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.08979976922273636 +jet/accuracy_2_of_3‚0.14490120112895966 *jet/accuracy_1_of_4‚0.5415709614753723 /particle/accuracy_2_of_2‚0.9998962879180908 ;"Purity/1FRt0SRqqt0FBt/event_purity‚0.023568956181406975 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9444444179534912 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.1454940140247345 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.08979976922273636 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.06582764536142349 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.8999999761581421 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 particle/accuracy_4_of_5‚1 5 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.92950439453125 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9706061482429504 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 <"Purity/2FRt*SRqqt*FBt/event_purity‚0.0011621315497905016 particle/accuracy_1_of_3‚1 _step‚32 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.3046875 :"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.23425376415252686 :"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.23922765254974365 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 particle/accuracy_2_of_4‚1 &particle/f_score‚0.531285285949707 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.2312779724597931 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 6 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.984639585018158 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.10049835592508316 particle/accuracy_1_of_4‚1 ) Purity/2FRt0SRqqt2FBt/FRt_purity‚0.25 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344]“eY<ù®| :"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.42148062586784363 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.02668113447725773 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 % _timestamp‚1.7842904759626365e+09 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.09922734647989273 _runtime‚ 660.180403157 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.14459779858589172 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.02497893013060093 <"Purity/1FRt1SRqqt1FBt/event_purity‚0.0064745196141302586 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.08979976922273636 +jet/accuracy_2_of_3‚0.14490120112895966 *jet/accuracy_1_of_4‚0.5415709614753723 /particle/accuracy_2_of_2‚0.9998962879180908 ;"Purity/1FRt0SRqqt0FBt/event_purity‚0.023568956181406975 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9444444179534912 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.1454940140247345 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.08979976922273636 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.06582764536142349 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.8999999761581421 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 particle/accuracy_4_of_5‚1 5 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.92950439453125 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9706061482429504 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 <"Purity/2FRt*SRqqt*FBt/event_purity‚0.0011621315497905016 particle/accuracy_1_of_3‚1 _step‚32 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.3046875 :"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.23425376415252686 :"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.23922765254974365 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 particle/accuracy_2_of_4‚1 &particle/f_score‚0.531285285949707 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.2312779724597931 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 6 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.984639585018158 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.10049835592508316 particle/accuracy_1_of_4‚1 ) Purity/2FRt0SRqqt2FBt/FRt_purity‚0.25 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 9 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.023568956181406975 ,jet/accuracy_2_of_2‚0.051340989768505096 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.1119791716337204 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.06134391948580742 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.24758881330490112 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0012454213574528694 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 *jet/accuracy_1_of_3‚0.5794954895973206 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.24599874019622803 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.05970693379640579 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.2245798408985138 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_2_of_3‚1 +particle/sensitivity‚0.5386502146720886 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.02613985165953636 :"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.21978260576725006 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.2325982302427292 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.1518324911594391 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.14358274638652802 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.03936391696333885 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.2199791669845581 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.5648148059844971 (particle/accuracy‚0.7958868145942688 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.10412339866161346 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.7692508697509766 ,jet/accuracy_3_of_4‚0.012017344124615192 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9444444179534912 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4000000059604645 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9647400975227356 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.2221650779247284 :"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.23205289244651794 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 particle/accuracy_1_of_2‚1 + Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9375 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.48148149251937866 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.09550683945417404 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 8"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.180070161819458 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.11540386825799942 9 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.013564098626375198 particle/accuracy_1_of_5‚1 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.2266857922077179 6 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.977568507194519 8 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.20591457188129425 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 8 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.30000001192092896 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.15483619272708893 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.11966816335916519 8"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.238542377948761 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.11154573410749435 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 /particle/accuracy_4_of_4‚0.9640492796897888 particle/accuracy_5_of_5‚0 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.48148149251937866 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.3402777910232544 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9692595601081848 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 :"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.24165447056293488 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 ( Purity/2FRt1SRqqt2FBt/FBt_purity‚0.5 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.13088122010231018 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.3088071346282959 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9337990283966064 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 ("Purity/2FRt*SRqqt2FBt/event_purity‚0 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.6806460618972778 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9856082201004028 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9469119310379028 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9686850309371948 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.4476345479488373 :"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.48566871881484985 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.6666666865348816 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.6111111044883728 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 jet/accuracy_5_of_5‚0 ;"Purity/*FRt0SRqqt0FBt/event_purity‚0.022662905976176262 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 jet/accuracy_2_of_5‚0 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.1174616813659668 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.2050388902425766 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9861127138137817 :"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.40500640869140625 8 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.21266062557697296 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9836961627006531 particle/accuracy_1_of_1‚1 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.020417217165231705 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.6666666865348816 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.002311334013938904 trainer/global_step‚399 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.6374609470367432 6 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.986639142036438 ;"Purity/1FRt0SRqqt1FBt/event_purity‚0.058048348873853683 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9922696352005005 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 9 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.057470113039016724 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 7validation_average_jet_accuracy‚0.15392984449863434 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.11763635277748108 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.1273384392261505 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.2142857164144516 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.8743829131126404 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9883795976638794 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.12622827291488647 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9736841917037964 /particle/accuracy_3_of_3‚0.9984673857688904 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.7692307829856873 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.11785677820444107 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.11727948486804962 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.1990366280078888 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.0536186508834362 )jet/accuracy_1_of_2‚0.408681720495224 -jet/accuracy_3_of_3‚0.0062730517238378525 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.3461538553237915 8 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.20612244307994843 ;"Purity/1FRt*SRqqt*FBt/event_purity‚0.018360117450356483 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.3333333432674408 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_3_of_4‚1 jet/accuracy_4_of_5‚0 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9805300831794739 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 6 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.938066840171814 jet/accuracy_3_of_5‚0 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.5546444654464722 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.3461538553237915 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 ("Purity/2FRt*SRqqt1FBt/event_purity‚0 +jet/accuracy_1_of_1‚0.12319111824035645 particle/accuracy_3_of_5‚1 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 :"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.22368642687797546 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 +particle/specificity‚0.8662404417991638 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9852121472358704 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.2221650779247284 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.001655382919125259 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.15700840950012207 :"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.19296228885650635 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.2329237312078476 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.12987680733203888 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.8250102996826172 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9696428179740906 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.2339056134223938 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.1267469972372055 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.2221384346485138 :"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.49039649963378906 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9743033647537231 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.6990740299224854 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9706740379333496 <"Purity/1FRt1SRqqt0FBt/event_purity‚0.0014020794769749045 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.15740741789340973 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 ;"Purity/0FRt2SRqqt*FBt/event_purity‚0.044705379754304886 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 :"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.18736037611961365 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.01913236640393734 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9729613065719604 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 ;"Purity/0FRt2SRqqt0FBt/event_purity‚0.045758143067359924 8"Purity/1FRt*SRqqt0FBt/event_purity‚0.017346216365695 7 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.9875974059104919 :"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.19296228885650635 *loss/val_total_loss‚21.275293947607985 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 jet/accuracy_4_of_4‚0 ("Purity/2FRt0SRqqt2FBt/event_purity‚0 & Purity/2FRt1SRqqt2FBt/FRt_purity‚0 jet/accuracy_1_of_5‚1 particle/accuracy_2_of_5‚1 :"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.23538056015968323 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9856082201004028 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt0SRqqt1FBt/event_purity‚0 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.020065905526280403 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.8999999761581421 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.4341323971748352 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9704977869987488 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9371915459632874 9 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.061898402869701385 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.05896572396159172 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.7533417344093323 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.20404060184955597 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 +jet/accuracy_2_of_4‚0.12858335673809052 , Purity/2FRt*SRqqt2FBt/FRt_purity‚0.21875 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.02279055491089821 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526ÑšK&791036129 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.15309497714042664 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.14041666686534882 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.5648148059844971 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.00213325722143054 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 epoch‚0 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.23675914108753204 6 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.987500011920929 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 hJ+IŸú‚(:” º¹èÒ•ýäø gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚41.0 gpu.0.powerWatts‚263.424' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚52.684799999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0–«ÊÖû‚(:Ë º¹èÒžŽ÷ú disk.sda3.out‚23.3125$ disk./.usageGB‚554.9293022155762 disk.rbd0.in‚0 proc.cpu.threads‚17 disk.sda.in‚0* proc.memory.percent‚17.347288131713867 cpu‚25.576281349280315% disk.nvme0n1.out‚4507.74951171875 disk.rbd1.out‚ 1.64453125 network.sent‚670699) disk./.usagePercent‚37.24177108838511 disk.nvme0c0n1.in‚ 3.66796875 disk.rbd1.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda.out‚23.3125 disk.dm-1.in‚0 disk.dm-0.in‚ 3.66796875" disk.dm-0.out‚4507.74951171875 disk.sda3.in‚0" proc.memory.rssMB‚ 2842.1796875 network.recv‚223703 disk.sda2.out‚0) proc.memory.availableMB‚ 1946.38671875 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.nvme0n1.in‚ 3.66796875 disk.rbd0.out‚0# memory_percent‚88.1201982498169' disk.nvme0c0n1.out‚4507.74951171875 disk.dm-1.out‚23.3125Žayü‚(:’ ɹèÒ×â‰ù gpu.0.gpu‚30.0 gpu.0.memory‚9, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚40.0 gpu.0.powerWatts‚268.84' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚53.767999999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‹`œfÙý‚(:Î ɹèÒżÄû disk.sda1.out‚0 disk.rbd3.in‚0 disk.rbd1.in‚0 proc.cpu.threads‚17 disk.rbd1.out‚ 1.70703125 network.sent‚715421 network.recv‚228378 disk.dm-1.out‚ 23.578125 disk.rbd0.in‚0 disk.drbd2165.out‚0$ disk.nvme0n1.out‚4521.7998046875 disk.dm-1.in‚0 disk.rbd0.out‚0! disk.dm-0.out‚4521.7998046875 disk.sda1.in‚0 disk.nvme0c0n1.in‚ 3.96484375 disk.drbd2165.in‚0 disk.sda2.out‚0" proc.memory.rssMB‚ 2842.1796875 cpu‚25.572178654355547 disk.nvme0n1.in‚ 3.96484375* proc.memory.percent‚17.347288131713867 disk.sda.in‚0 disk.sda3.in‚0 disk.sda2.in‚0* disk./.usagePercent‚37.242207581230986 disk.sda3.out‚ 23.578125' proc.memory.availableMB‚ 1946.171875 disk.dm-0.in‚ 3.96484375 disk.rbd3.out‚0& disk.nvme0c0n1.out‚4521.7998046875$ memory_percent‚88.12150955200195$ disk./.usageGB‚554.9358062744141 disk.sda.out‚ 23.578125Ž0÷àžþ‚(:“ عèҫ̾ø gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚297.575' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚59.51499999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0üÔeãÿ‚(:Ø عèÒ•ÝÅú network.sent‚720698 disk.drbd2165.out‚0$ memory_percent‚88.12310695648193 cpu‚25.568017888490072 disk.sda2.out‚0 disk.rbd0.in‚0 disk.drbd2165.in‚0 network.recv‚232221 disk.dm-0.in‚ 3.99609375" disk.dm-0.out‚4533.07763671875) disk./.usagePercent‚37.24265738646865 disk.sda.out‚ 24.06640625 disk.dm-1.in‚0 disk.sda3.in‚0 disk.rbd3.in‚0) proc.memory.availableMB‚ 1945.91015625 disk.rbd1.in‚0 disk.dm-1.out‚ 24.06640625 disk.sda1.in‚0$ disk./.usageGB‚554.9425086975098 disk.rbd1.out‚ 1.70703125 disk.sda1.out‚0 disk.nvme0c0n1.in‚ 3.99609375' disk.nvme0c0n1.out‚4533.07763671875" proc.memory.rssMB‚ 2842.1796875* proc.memory.percent‚17.347288131713867 proc.cpu.threads‚17 disk.sda2.in‚0 disk.sda.in‚0 disk.sda3.out‚ 24.06640625 disk.nvme0n1.in‚ 3.99609375% disk.nvme0n1.out‚4533.07763671875 disk.rbd0.out‚0 disk.rbd3.out‚0‘3žŸ€‚(:” ç¹èÒ©¨àø gpu.0.gpu‚86.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚40.0 gpu.0.powerWatts‚146.709' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚29.341800000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÄÄ íÖ‚(:Ë ç¹èÒõȼú% disk.nvme0n1.out‚4546.88720703125 disk.rbd0.out‚0 disk.rbd0.in‚0 disk.sda3.in‚0 disk.sda3.out‚24.28125 disk.drbd2165.out‚0 disk.rbd3.in‚0* proc.memory.percent‚17.347288131713867* disk./.usagePercent‚37.243105911668636 disk.sda1.in‚0 disk.nvme0c0n1.in‚ 4.29296875 network.recv‚235873 disk.sda2.in‚0 disk.rbd1.in‚0" disk.dm-0.out‚4546.88720703125" proc.memory.rssMB‚ 2842.1796875$ disk./.usageGB‚554.9491920471191 disk.sda1.out‚0' disk.nvme0c0n1.out‚4546.88720703125 network.sent‚725673 disk.rbd1.out‚ 1.83203125 disk.sda.out‚24.28125 disk.dm-0.in‚ 4.29296875$ memory_percent‚99.94173049926758$ proc.memory.availableMB‚9.546875 disk.sda.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚24.28125 disk.rbd3.out‚0 disk.nvme0n1.in‚ 4.29296875 cpu‚25.565977573874218 disk.sda2.out‚0 disk.drbd2165.in‚0 proc.cpu.threads‚17 ?OsŸ‚‚(:” ö¹èÒ¡›Ñø gpu.0.gpu‚90.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚274.213' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚54.842600000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0í43«áƒ‚(:Ö ö¹èÒìóú disk.rbd3.in‚0 cpu‚25.567610308561832 proc.cpu.threads‚17 disk.rbd0.out‚0 disk.dm-1.in‚0 disk.sda3.in‚0 network.recv‚238986 disk.nvme0n1.in‚ 4.29296875 disk.sda2.in‚0 disk.dm-0.in‚ 4.29296875$ disk./.usageGB‚554.9526481628418$ memory_percent‚99.94027614593506* proc.memory.percent‚17.347288131713867 disk.rbd1.out‚ 1.83203125" disk.dm-0.out‚4551.42236328125 disk.nvme0c0n1.in‚ 4.29296875' disk.nvme0c0n1.out‚4551.42236328125 disk.sda2.out‚0& proc.memory.availableMB‚ 9.78515625" proc.memory.rssMB‚ 2842.1796875 network.sent‚732748 disk.sda.out‚ 24.44921875 disk.sda1.in‚0 disk.drbd2165.in‚0 disk.sda.in‚0 disk.sda3.out‚ 24.44921875 disk.drbd2165.out‚0 disk.rbd3.out‚0* disk./.usagePercent‚37.243337854494655 disk.rbd1.in‚0 disk.dm-1.out‚ 24.44921875 disk.rbd0.in‚0 disk.sda1.out‚0% disk.nvme0n1.out‚4551.42236328125±ì%0„+ _wandbruntime‚743  _runtime‚743J~ÝÄ…¾ trainer/global_step‚449 % _timestamp‚1.7842905596375206e+09 _runtime‚ 743.854891591 _step‚33 & lr-AdamW/pg1‚0.0006918377827674013 & lr-AdamW/pg2‚0.0006918377827674013ý» Ȇ _step‚33 & lr-AdamW/pg1‚0.0006918377827674013 & lr-AdamW/pg2‚0.0006918377827674013 trainer/global_step‚449 % _timestamp‚1.7842905596375206e+09 _runtime‚ 743.854891591!ñ®¿0‡+ _wandbruntime‚743  _runtime‚743—ÕÄàÒ'ˆÌO VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402628608 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 542971573760 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚128959197515776 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1280304 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12152 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚35109115154432 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚3897003 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚248 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12095 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345388544 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚253 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9971 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028282 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚129015812227072 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9977 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚7726080 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚129439138723840 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371561984 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚35651718637056 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471111 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚3829077 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 423326496768 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 _runtime‚ 743.856233985 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚129015786053632 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1280552 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚129439138723840 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚3829077 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚3887032 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371561984 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _step‚34 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚129381825472967 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚129015812227072 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545123328 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9977 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 421257928909 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7842905596388936e+09 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680115712 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12095 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600314 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 HDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚129437767161856 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚3887032 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1157472 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚35652121265664 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 421981108224 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚3829071 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚129015786053632 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 423326496768 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 421981108224 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚2437771 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚2438024 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 543006111232 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚128959222943744 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚3829071 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318918 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345388544¢ «CÖ'‰ÐO E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12152 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚35109115154432 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚3897003 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚248 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12095 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345388544 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚253 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9971 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028282 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚129015812227072 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9977 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚7726080 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚129439138723840 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371561984 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚35651718637056 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471111 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚3829077 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 423326496768 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 _runtime‚ 743.856233985 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚129015786053632 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1280552 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚129439138723840 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚3829077 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚3887032 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371561984 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _step‚34 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚129381825472967 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚129015812227072 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545123328 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9977 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 421257928909 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7842905596388936e+09 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680115712 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12095 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600314 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 HDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚129437767161856 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚3887032 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1157472 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚35652121265664 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 421981108224 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚3829071 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚129015786053632 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 423326496768 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 421981108224 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚2437771 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚2438024 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 543006111232 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚128959222943744 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚3829071 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318918 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345388544 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402628608 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 542971573760 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚128959197515776 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1280304 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820""(×!0Š+ _wandbruntime‚745  _runtime‚745ÇÁP70‹ÖM F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚7742778 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1160010 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1283307 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚35186601562112 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚129660630717501 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚7732787 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚3837283 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚3837277 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚3837283 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚129293786783744 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680115712 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371569152 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680115712 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12152 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚129293786783744 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚3905495 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚3895510 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345395712 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚35186233470976 % _timestamp‚1.7842905610266476e+09 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚129716692927488 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚129293812957184 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚7742778 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1160015 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318918 _runtime‚ 745.244070457 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9985 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚449 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545123328 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402621440 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚3895510 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1283061 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 424251539456 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345395712 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚246 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 544209453568 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚129237079293952 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚3905495 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 424251539456 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 544174923264 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚129659260689107 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚129293812957184 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚3837277 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371569152 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545123328 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9991 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚35730408394240 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚129718064496640 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚2443071 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚129718064496640 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9985 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471111 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 _step‚35 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚3837283 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚129293786783744 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680115712 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371569152 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680115712 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12152 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚129293786783744 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚3905495 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚3895510 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345395712 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚35186233470976 % _timestamp‚1.7842905610266476e+09 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚129716692927488 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚129293812957184 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚7742778 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1160015 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318918 _runtime‚ 745.244070457 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9985 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚449 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545123328 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402621440 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚3895510 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1283061 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 424251539456 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345395712 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚246 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 544209453568 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚129237079293952 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚3905495 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 424251539456 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 544174923264 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚129659260689107 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚129293812957184 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚3837277 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371569152 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545123328 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9991 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚35730408394240 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚129718064496640 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚2443071 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚129718064496640 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9985 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471111 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 _step‚35 EDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9977 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚4322755 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚39586438894592 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 603003356160 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12095 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1419729 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471111 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318918 _step‚38 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚4322755 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 YÅ9TÞM EDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚143385491160064 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚143384119598080 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9971 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚4312784 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 % _timestamp‚1.7842906388520129e+09 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚143384119598080 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 602966721536 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚142853286424576 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12095 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545123328 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 468297038848 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345388544 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚2704070 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚143385491160064 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚8202=‚ÙÖ'ŸÐO bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 603003356160 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12095 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1419729 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471111 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318918 _step‚38 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚4322755 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚143385491160064 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚143384119598080 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9971 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚4312784 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 % _timestamp‚1.7842906388520129e+09 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚143384119598080 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 602966721536 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚142853286424576 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12095 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545123328 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 468297038848 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345388544 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚2704070 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚143385491160064 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371561984 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 468297038848 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028282 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9971 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680115712 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12152 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1284622 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚143320781466593 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404725760 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚143322151494875 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚4239377 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚4312784 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚2704351 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚281 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9977 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚4322755 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚39586438894592&æy‘y0 + _wandbruntime‚824  _runtime‚824ëaUÜ&¡ÖM H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚271 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1422192 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚39060553854976 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028394 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚8578830 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚143193823289344 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚2709352 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚39060921946112 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471111 _runtime‚ 824.456174371 _step‚39 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚4247583 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680115712 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚4247583 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 604158887936 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚143664416932864 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚143193849462784 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12095 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚143600956739409 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚143131168202752 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12152 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚143599586711015 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 % _timestamp‚1.7842906402387645e+09 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚39665117461504 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545123328 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 469222074368 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚4331247 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 604195515392 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚143193823289344 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚39664712742912 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚143131193630720 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371569152 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚143663045363712 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚8578830 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318918&mÀ/à&¢ÚM F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 trainer/global_step‚499 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371569152 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1422458 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600426 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚4321262 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545123328 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚143193849462784 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚2709623 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36627456 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚143664416932864 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1287165 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚8568839 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404718592 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚4247583 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680115712 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚4247583 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 604158887936 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚143664416932864 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚143193849462784 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12095 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚143600956739409 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚143131168202752 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12152 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚143599586711015 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 % _timestamp‚1.7842906402387645e+09 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚39665117461504 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545123328 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 469222074368 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚4331247 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 604195515392 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚143193823289344 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚39664712742912 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚143131193630720 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371569152 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚143663045363712 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚8578830 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318918 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚271 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1422192 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚39060553854976 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028394 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚8578830 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚143193823289344 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚2709352 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚39060921946112 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471111 _runtime‚ 824.456174371 _step‚39 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚157330472034304 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680115712 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545123328 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚156815885238272 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 514612969472 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚273 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚157330472034304 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚4649677 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9971 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚42857387831296 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚156747400761344 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9977 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚4649671 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚4738536 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680115712 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚157331843596288 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 662550407168 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 HDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9977 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 513732155125 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12095 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚4748507 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1557713 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚43520306329600 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600314 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12152 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12095 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1411772 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318918 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 515958358016 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚43519901603840 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 515076755439 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 514612969472 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1411767 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚9388207 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12152 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9971 HDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 662513772544 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚4649671Cñ‰Ö'·ÐO L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚9398184 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚2969753 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚156815859064832 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404725760 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 % _timestamp‚1.7842907182238405e+09 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚157330472034304 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680115712 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545123328 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚156815885238272 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 514612969472 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚273 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚157330472034304 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚4649677 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9971 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚42857387831296 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚156747400761344 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9977 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚4649671 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚4738536 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680115712 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚157331843596288 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 662550407168 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 HDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9977 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 513732155125 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12095 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚4748507 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1557713 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚43520306329600 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600314 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12152 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12095 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1411772 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318918 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 515958358016 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚43519901603840 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 515076755439 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 514612969472 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1411767 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚9388207 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12152 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9971 HDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 662513772544 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚4649671 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371561984 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345388544 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚4738536 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚268 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚157261107488501 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚156815885238272 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545123328 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚2969480 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _runtime‚ 902.441060073 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471111 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚156747375333376 _step‚42 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0*d$0¸+ _wandbruntime‚903  _runtime‚903BÅNv¹ÖM C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12152 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚4657883 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 trainer/global_step‚549 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚9404891 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚4747014 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545123328 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 516883400704 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680115712 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1414315 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚157610769369088 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1414310 GADeviceStatsMonitor.on_tágwfrain_batch_end/oversize_segments.allocated‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 516000221797 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318918 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12152 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚4657877 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345395712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚4756999 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚4747014 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12095 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371569152 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚157093859794944 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 515538004992 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1560425 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36627456 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚4657877 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680115712 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 514655621371 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚2974735 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚265 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471111 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚4756999 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚9404891 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 663737568256 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9985 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 516883400704 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚9414882 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9985 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9991 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545123328 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚2975005 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600426 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚157609397799936 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚42935242330112 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚42934874238976 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 _step‚43 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚2974735 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚265 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471111 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚4756999 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚9404891 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 663737568256 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9985 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 516883400704 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚9414882 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9985 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9991 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545123328 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚2975005 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600426 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚157609397799936 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚42935242330112 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚42934874238976 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 _step‚43 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680115712 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1414315 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚157610769369088 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1414310 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 516000221797 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318918 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12152 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚4657877 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345395712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚4756999 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚4747014 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12095 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371569152 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚157093859794944 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 515538004992 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1560425 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36627456 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚4657877 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680115712 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0+¹y¯¬0»+ _wandbruntime‚903  _runtime‚903ർû &loss/total_loss‚17.271099541513028 _step‚44 0loss/FRt1/assignment_loss‚3.5398937932252883 2loss/SRqqt1/assignment_loss‚1.6501827901582928 2loss/SRqqt2/assignment_loss‚1.3188794431541957 1loss/FBt2/assignment_loss‚0.24001708857599588 1loss/SRqqt2/detection_loss‚0.6587612541745523 .loss/FRt1/detection_loss‚0.612417808547616 0loss/SRqqt1/detection_loss‚2.015101279733302 .loss/FBt2/detection_loss‚2.116711886972189 % _timestamp‚1.7842907196961536e+09 epoch‚1 /loss/FRt2/assignment_loss‚2.274898119434911 1loss/FBt1/assignment_loss‚0.12419090228462926 /loss/FRt2/detection_loss‚1.7188542424188311 trainer/global_step‚549 _runtime‚ 903.913070677 /loss/FBt1/detection_loss‚1.0011909328332242©þàl½ÿ epoch‚1 /loss/FRt2/assignment_loss‚2.274898119434911 1loss/FBt1/assignment_loss‚0.12419090228462926 /loss/FRt2/detection_loss‚1.7188542424188311 trainer/global_step‚549 _runtime‚ 903.913070677 /loss/FBt1/detection_loss‚1.0011909328332242 &loss/total_loss‚17.271099541513028 _step‚44 0loss/FRt1/assignment_loss‚3.5398937932252883 2loss/SRqqt1/assignment_loss‚1.6501827901582928 2loss/SRqqt2/assignment_loss‚1.3188794431541957 1loss/FBt2/assignment_loss‚0.24001708857599588 1loss/SRqqt2/detection_loss‚0.6587612541745523 .loss/FRt1/detection_loss‚0.612417808547616 0loss/SRqqt1/detection_loss‚2.015101279733302 .loss/FBt2/detection_loss‚2.116711886972189 % _timestamp‚1.7842907196961536e+09,?(¤$”¾‚(:‰ ª»èÒÁŸ÷ø gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚270.943' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.1886 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Êõ¸à¿‚(:Õ ª»èÒÆòŒû network.recv‚281618* disk./.usagePercent‚37.247434231050036 disk.dm-0.in‚ 5.93359375 disk.rbd1.in‚0 disk.rbd1.out‚ 2.2265625 disk.drbd2165.in‚0 disk.rbd3.out‚0' proc.memory.availableMB‚ 10.30859375$ disk./.usageGB‚555.0136871337891 disk.rbd0.in‚0 disk.sda3.out‚ 30.44921875 disk.rbd3.in‚0 disk.sda2.out‚0# memory_percent‚99.9370813369751 disk.dm-1.out‚ 30.44921875 disk.rbd0.out‚0 disk.sda1.in‚0 disk.sda1.out‚0' disk.nvme0c0n1.out‚4683.88623046875 cpu‚25.59539517924929 disk.sda.in‚0 disk.sda.out‚ 30.44921875" disk.dm-0.out‚4683.88623046875 disk.sda3.in‚0 disk.nvme0c0n1.in‚ 5.93359375 network.sent‚918481 disk.drbd2165.out‚0# proc.memory.rssMB‚ 2842.18359375* proc.memory.percent‚17.347311973571777 disk.sda2.in‚0 proc.cpu.threads‚17 disk.dm-1.in‚0 disk.nvme0n1.in‚ 5.93359375% disk.nvme0n1.out‚4683.88623046875X0MŒ•À‚(:Š ¹»èÒ‹Ë©ø gpu.0.gpu‚100.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚277.374' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.4748 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ÓªaÝÁ‚(:Ò ¹»èÒçßžú disk.rbd3.in‚0 disk.sda1.out‚0$ memory_percent‚99.93643760681152 disk.sda3.in‚0& proc.memory.availableMB‚ 10.4140625 disk.drbd2165.in‚0) disk./.usagePercent‚37.24766284577811$ disk.nvme0n1.out‚4698.8759765625 disk.sda2.in‚0 disk.sda.in‚0# proc.memory.rssMB‚ 2842.18359375* proc.memory.percent‚17.347311973571777! disk.dm-0.out‚4698.8759765625$ disk./.usageGB‚555.0170936584473 disk.rbd3.out‚0 disk.nvme0n1.in‚ 5.96484375 disk.dm-1.out‚ 30.98046875 disk.sda3.out‚ 30.98046875 network.sent‚968541 disk.sda.out‚ 30.98046875 disk.sda1.in‚0 disk.drbd2165.out‚0 network.recv‚286363 disk.nvme0c0n1.in‚ 5.96484375& disk.nvme0c0n1.out‚4698.8759765625 disk.rbd1.in‚0 disk.rbd1.out‚ 2.2890625 disk.dm-1.in‚0 disk.rbd0.out‚0 cpu‚25.597876188504806 disk.dm-0.in‚ 5.96484375 disk.sda2.out‚0 disk.rbd0.in‚0 proc.cpu.threads‚17aM¤˜žÂ‚(:“ È»èÒû¶¯ø gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚275.487' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚55.09740000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0†/qÜÂ(:Ñ È»è҇νú network.recv‚289611 disk.sda2.in‚0 disk.sda.in‚0 disk.rbd3.in‚0 proc.cpu.threads‚17* disk./.usagePercent‚37.248109066910295 disk.dm-1.out‚ 31.2265625 disk.drbd2165.in‚0$ disk./.usageGB‚555.0237426757812 disk.sda.out‚ 31.2265625$ memory_percent‚99.93641376495361 disk.sda3.out‚ 31.2265625 disk.nvme0n1.in‚ 6.26171875 disk.sda2.out‚0 disk.rbd0.out‚0 disk.sda1.out‚0 disk.sda3.in‚0 network.sent‚973510 disk.rbd1.out‚ 2.2890625 disk.drbd2165.out‚0 disk.rbd3.out‚0 disk.nvme0c0n1.in‚ 6.26171875$ disk.nvme0n1.out‚4708.3955078125 disk.rbd1.in‚0 disk.dm-0.in‚ 6.26171875' proc.memory.availableMB‚ 10.41796875# proc.memory.rssMB‚ 2842.18359375& disk.nvme0c0n1.out‚4708.3955078125! disk.dm-0.out‚4708.3955078125 cpu‚25.599187062500526 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.sda1.in‚0* proc.memory.percent‚17.3473119735717771dé’Ä‚(:‡ ×»èÒ°›ýø gpu.0.gpu‚97.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚217.38' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚43.476 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0öêKoÞÅ‚(:Ó ×»èÒ«Šû* disk./.usagePercent‚37.248579352750696 disk.rbd0.in‚0' disk.nvme0c0n1.out‚4719.50732421875 cpu‚25.601454739139875 disk.sda1.in‚0 disk.rbd3.out‚0 disk.rbd1.out‚ 2.3515625 disk.dm-1.in‚0 disk.dm-1.out‚ 31.5859375$ disk./.usageGB‚555.0307502746582 disk.rbd1.in‚0" disk.dm-0.out‚4719.50732421875$ memory_percent‚99.93605613708496 disk.dm-0.in‚ 6.29296875& proc.memory.availableMB‚ 10.4765625 disk.rbd0.out‚0 disk.sda1.out‚0 disk.nvme0c0n1.in‚ 6.29296875# proc.memory.rssMB‚ 2842.18359375 disk.sda2.in‚0 disk.sda.in‚0 disk.sda3.in‚0 disk.sda3.out‚ 31.5859375 disk.rbd3.in‚0 disk.sda.out‚ 31.5859375 network.sent‚978524 network.recv‚293414 disk.nvme0n1.in‚ 6.29296875% disk.nvme0n1.out‚4719.50732421875 disk.sda2.out‚0 disk.drbd2165.in‚0 disk.drbd2165.out‚0* proc.memory.percent‚17.347311973571777 proc.cpu.threads‚17^d žÆ‚(:“ æ»èÒûúïö gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚279.79' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚55.958000000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0v¸ÆÑÇ‚(:Æ æ»èÒ†—ûø disk.rbd3.out‚0 disk.rbd1.in‚0 network.sent‚983604 disk.dm-0.out‚ 4732.07421875 disk.drbd2165.in‚0 disk.rbd3.in‚0 disk.dm-1.in‚0 disk.sda1.in‚0 disk.sda3.out‚ 31.734375 disk.sda2.out‚0 network.recv‚296665 disk.sda.in‚0 disk.dm-0.in‚ 6.58984375 disk.sda1.out‚0 disk.sda3.in‚0" disk.nvme0n1.out‚ 4732.07421875# memory_percent‚99.9375581741333' proc.memory.availableMB‚ 10.23046875* disk./.usagePercent‚37.249006629325336 disk.rbd0.out‚0 disk.drbd2165.out‚0$ disk.nvme0c0n1.out‚ 4732.07421875* proc.memory.percent‚17.347311973571777 proc.cpu.threads‚17 disk.sda.out‚ 31.734375 disk.sda2.in‚0 disk.rbd1.out‚ 2.3515625# proc.memory.rssMB‚ 2842.18359375 cpu‚25.60154707645478$ disk./.usageGB‚555.0371170043945 disk.rbd0.in‚0 disk.nvme0n1.in‚ 6.58984375 disk.nvme0c0n1.in‚ 6.58984375 disk.dm-1.out‚ 31.734375d?0È+ _wandbruntime‚981  _runtime‚981y„mêÄɾ & lr-AdamW/pg1‚0.0006916724943312366 & lr-AdamW/pg2‚0.0006916724943312366 trainer/global_step‚599 % _timestamp‚1.7842907973267338e+09 _runtime‚ 981.544008378 _step‚45»A¥õÈÊ _step‚45 & lr-AdamW/pg1‚0.0006916724943312366 & lr-AdamW/pg2‚0.0006916724943312366 trainer/global_step‚599 % _timestamp‚1.7842907973267338e+09 _runtime‚ 981.544008378-µï„0Ë+ _wandbruntime‚981  _runtime‚981ÒUÉåÍÌÐO LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12095 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028282 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚170641464242176 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚171276824470528 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345388544 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚171201433510409 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 560928900096 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚3234630 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371561984 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12152 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12095 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚170641489670144 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600314 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9977 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1695708 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚5174259 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371561984 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚170715895570432 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚274 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚170715895570432 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680115712 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 562274288640 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚5164288 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚279 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚47454022079488 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚5059971 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345388544 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚10234236 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚3234351 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 YEDevi·8 ceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚171202803538691 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471111 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 _step‚46 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404725760 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 721909138432 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9971 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚170715921743872 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318918 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680115712 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚5059971 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1538922 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚171276824470528 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345388544 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚171201433510409 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 560928900096 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚3234630 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371561984 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12152 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12095 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚170641489670144 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600314 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9977 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1695708 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚5174259 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371561984 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚170715895570432 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚274 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚170715895570432 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680115712 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 562274288640 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚5164288 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚279 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚47454022079488 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚5059971 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345388544 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚10234236 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚3234351 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚171202803538691 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471111 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 _step‚46 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404725760 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 721909138432 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9971 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚170715921743872 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318918 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680115712 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚5059971 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1538922 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371569152 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680115712 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚264 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚10250934 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚170919346020352 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 % _timestamp‚1.7842907987133405e+09 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚170993896300544 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545123328 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3239642 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9991 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚10240943 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚5182751 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471111 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚ 982.93068464 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚5068183 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚259 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9985^ì1·ëÐÝM FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚170919346020352 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 % _timestamp‚1.7842907987133405e+09 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚170993896300544 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545123328 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3239642 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9991 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚10240943 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚5182751 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471111 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚ 982.93068464 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚5068183 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚259 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9985 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1698182 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚5182751 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12095 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1698441 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9985 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚3239906 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚47532285728768 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚171555750236160 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371569152 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1541465 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318918 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 _step‚47 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12152 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371569152 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680115712 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚264 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚10250934 EDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚185223176906752 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404725760 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚5470271 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371561984 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9977 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _step‚50 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9977 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚5600011 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9971 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚51387623782912 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚184535553150976 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚269 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545123328 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 607244830720 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 606206381341 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚185223176906752 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 608590219264 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600314 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚5470271 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚51387219057152 % _timestamp‚1.7842908766438975e+09 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚50606028599296 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1666067 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚185141759532317 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚11070288 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12095 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371561984 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12095 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚184615932076032 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9971 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚5470277 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 trainer/global_step‚649 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 607244830720 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1666072 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚184615958249472 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345388544 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚184615932076032 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚184615958249472 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚50606396690432 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚185143129560599 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028282 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471111 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12152 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1832988 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36634624 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318918 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 781190457856 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚11060311 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚5590040 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 608590219264 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚184535578578944ÍãµÚ'ãÔO YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚185143129560599 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028282 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471111 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12152 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1832988 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36634624 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318918 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 781190457856 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚11060311 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚5590040 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 608590219264 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚184535578578944 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚185223176906752 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404725760 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚5470271 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371561984 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9977 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _step‚50 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9977 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚5600011 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9971 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚51387623782912 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚184535553150976 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚269 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545123328 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 607244830720 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 606206381341 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚185223176906752 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 608590219264 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600314 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚5470271 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚51387219057152 % _timestamp‚1.7842908766438975e+09 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚50606028599296 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1666067 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚185141759532317 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚11070288 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12095 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371561984 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12095 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚184615932076032 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9971 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚5470277 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 trainer/global_step‚649 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 607244830720 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1666072 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚184615958249472 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345388544 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚184615932076032 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚184615958249472 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚50606396690432 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚02B¼Ý2ä- _wandbruntime‚1062  _runtime‚1062{)¼‰åÛM K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚11086986 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600426 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 607129847587 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚184893958979584 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545123328 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚185502102672384 RBDeviceStatsMonitor.on_train_batch_end/actt}‘?Æ"ive_bytes.large_pool.peak‚ 46141448192 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1668610 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371569152 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 608474448013 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚5478477 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 782409773568 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 782373146112 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1835464 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚184893932806144 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚5608503 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12095 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚5598518 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 608169866240 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚184893958979584 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 609515261952 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚5598518 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚11076995 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚3504334 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚51465888153088 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 _runtime‚1062.247284763 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚185503474241536 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680115712 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚51466292871680 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚50683515006976 trainer/global_step‚649 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471111 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318918 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚185421934805133 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9985 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚255 % _timestamp‚1.7842908780298727e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12152 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1668615 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3504074 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚5478477 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028394 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚11086986 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9985 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345395712 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404718592 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚50683883098112 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371569152 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545123328 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 609515261952 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345395712³Ó|vå&æßM 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028394 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚11086986 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9985 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345395712 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404718592 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚50683883098112 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371569152 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545123328 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 609515261952 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345395712 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚11086986 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600426 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 607129847587 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚184893958979584 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545123328 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚185502102672384 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1668610 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371569152 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 608474448013 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚5478477 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 782409773568 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 782373146112 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1835464 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚184893932806144 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚5608503 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12095 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚5598518 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 608169866240 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚184893958979584 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 609515261952 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚5598518 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚11076995 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚3504334 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚51465888153088 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 _runtime‚1062.247284763 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚185503474241536 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680115712 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚51466292871680 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚50683515006976 trainer/global_step‚649 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471111 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318918 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚185421934805133 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9985 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚255 % _timestamp‚1.7842908780298727e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12152 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1668615 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3504074 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚54784773z5%—2ç- _wandbruntime‚1062  _runtime‚1062rwD)èû 0loss/FRt2/assignment_loss‚1.8612429776508148 /loss/FRt1/detection_loss‚0.6156368220954908 /loss/FRt2/detection_loss‚1.8297862325468925 1loss/SRqqt1/detection_loss‚1.4495925350622698 trainer/global_step‚649 _step‚52 /loss/FRt1/assignment_loss‚3.334753553286457 2loss/SRqqt1/assignment_loss‚1.2796324539590966 1loss/SRqqt2/detection_loss‚0.8116029626349124 .loss/FBt2/detection_loss‚1.282730587193686 2loss/SRqqt2/assignment_loss‚1.1769177764415053 1loss/FBt1/assignment_loss‚0.10217472373383277 0loss/FBt2/assignment_loss‚0.2929823936491329 &loss/total_loss‚15.184445841047669 /loss/FBt1/detection_loss‚1.1473928227935781 epoch‚1 $ _timestamp‚1.784290878116093e+09 _runtime‚ 1062.33304241è™éÿ /loss/FRt2/detection_loss‚1.8297862325468925 1loss/SRqqt1/detection_loss‚1.4495925350622698 trainer/global_step‚649 _step‚52 /loss/FRt1/assignment_loss‚3.334753553286457 2loss/SRqqt1/assignment_loss‚1.2796324539590966 1loss/SRqqt2/detection_loss‚0.8116029626349124 .loss/FBt2/detection_loss‚1.282730587193686 2loss/SRqqt2/assignment_loss‚1.1769177764415053 1loss/FBt1/assignment_loss‚0.10217472373383277 0loss/FBt2/assignment_loss‚0.2929823936491329 &loss/total_loss‚15.184445841047669 /loss/FBt1/detection_loss‚1.1473928227935781 epoch‚1 $ _timestamp‚1.784290878116093e+09 _runtime‚ 1062.33304241 0loss/FRt2/assignment_loss‚1.8612429776508148 /loss/FRt1/detection_loss‚0.61563682209549084+E‰žê‚(:“ À¼èÒ¸¡çö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚264.47' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚52.894000000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0&¶¦Uäë‚(:Ù À¼èÒÇä‡ù) disk./.usagePercent‚37.25124695125748 disk.rbd0.out‚0 disk.sda1.in‚0 disk.drbd2165.out‚0 disk.nvme0c0n1.in‚ 7.27734375 cpu‚25.609224429087945 disk.sda3.out‚ 33.03515625 disk.rbd3.in‚0 network.sent‚ 1.084631e+06 disk.rbd3.out‚0 disk.rbd1.in‚0 disk.sda.in‚0 disk.rbd0.in‚0" disk.dm-0.out‚4792.00439453125$ memory_percent‚99.94235038757324# proc.memory.rssMB‚ 2842.18359375 disk.sda.out‚ 33.03515625 disk.dm-1.in‚0 disk.sda1.out‚0' disk.nvme0c0n1.out‚4792.00439453125% disk.nvme0n1.out‚4792.00439453125 disk.sda2.in‚0* proc.memory.percent‚17.347311973571777 proc.cpu.threads‚17 disk.dm-0.in‚ 7.27734375 disk.drbd2165.in‚0% proc.memory.availableMB‚ 9.4453125 network.recv‚318326 disk.sda3.in‚0 disk.rbd1.out‚ 2.6015625# disk./.usageGB‚555.070499420166 disk.dm-1.out‚ 33.03515625 disk.nvme0n1.in‚ 7.27734375 disk.sda2.out‚0„ÉcfŸì‚(:” ϼèÒЀÒö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚292.983' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚58.596599999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0l[p±Þí‚(:Ó ϼèÒÞ§ªø disk.sda.in‚0 disk.dm-1.out‚ 33.421875 disk.sda2.out‚0 disk.rbd1.out‚ 2.6640625$ disk./.usageGB‚555.0756187438965" disk.dm-0.out‚4799.16455078125 disk.drbd2165.in‚0 proc.cpu.threads‚17 disk.rbd3.out‚0' disk.nvme0c0n1.out‚4799.16455078125) disk./.usagePercent‚37.25159051336843 disk.rbd3.in‚0 disk.nvme0n1.in‚ 7.27734375 disk.sda2.in‚0$ memory_percent‚99.94244575500488 disk.dm-1.in‚0* proc.memory.percent‚17.347311973571777 network.sent‚ 1.134679e+06 disk.sda.out‚ 33.421875 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.dm-0.in‚ 7.27734375 disk.sda3.in‚0 disk.nvme0c0n1.in‚ 7.27734375# proc.memory.rssMB‚ 2842.18359375 network.recv‚323421 disk.sda1.in‚0 disk.rbd1.in‚0% proc.memory.availableMB‚ 9.4296875 cpu‚25.61140055962637 disk.sda1.out‚0 disk.sda3.out‚ 33.421875 disk.drbd2165.out‚0% disk.nvme0n1.out‚4799.16455078125hTµÁ”î‚(:‰ Þ¼èÒ» óö gpu.0.gpu‚93.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚278.198' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.6396 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0´…×Aåï‚(:Ú Þ¼èÒ­ÅÖø disk.drbd2165.in‚0 disk.nvme0n1.in‚ 7.57421875 disk.sda2.in‚0 disk.sda.out‚ 33.54296875 proc.cpu.threads‚17 network.sent‚ 1.139668e+06 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd3.in‚0 disk.sda2.out‚0 disk.dm-1.in‚0 disk.dm-0.in‚ 7.57421875 cpu‚25.61257262797326 disk.nvme0c0n1.in‚ 7.57421875' disk.nvme0c0n1.out‚4811.03759765625 disk.rbd1.out‚ 2.6640625 disk.sda3.in‚0 disk.sda3.out‚ 33.54296875% disk.nvme0n1.out‚4811.03759765625 disk.rbd1.in‚0# memory_percent‚99.9413251876831# proc.memory.rssMB‚ 2842.18359375* disk./.usagePercent‚37.251997565347864 disk.sda.in‚0 disk.rbd0.in‚0& proc.memory.availableMB‚ 9.61328125$ disk./.usageGB‚555.0816841125488 disk.drbd2165.out‚0 disk.rbd3.out‚0 disk.dm-1.out‚ 33.54296875 disk.rbd0.out‚0" disk.dm-0.out‚4811.03759765625* proc.memory.percent‚17.347311973571777 network.recv‚327018WÛÀ‰žð‚(:“ í¼èÒèÛáö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚296.857' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚59.37140000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0T_r`åñ‚(:Ú í¼èÒ•çÒø network.sent‚ 1.144691e+06$ disk./.usageGB‚555.0859527587891 disk.sda.out‚ 33.99609375 disk.dm-1.out‚ 33.99609375 disk.dm-0.in‚ 7.60546875 disk.sda1.in‚0 disk.sda2.out‚0$ memory_percent‚99.94199275970459 disk.rbd1.out‚ 2.7265625 network.recv‚330624 disk.rbd0.in‚0 disk.rbd0.out‚0" disk.dm-0.out‚4821.33642578125 disk.rbd1.in‚0& proc.memory.availableMB‚ 9.50390625 cpu‚25.61373111879852) disk./.usagePercent‚37.25228403777868 disk.dm-1.in‚0 disk.sda3.in‚0 disk.drbd2165.out‚0 disk.sda3.out‚ 33.99609375' disk.nvme0c0n1.out‚4821.33642578125 disk.nvme0n1.in‚ 7.60546875 disk.sda2.in‚0# proc.memory.rssMB‚ 2842.18359375 disk.sda.in‚0 disk.drbd2165.in‚0 disk.sda1.out‚0 disk.rbd3.in‚0% disk.nvme0n1.out‚4821.33642578125* proc.memory.percent‚17.347311973571777 proc.cpu.threads‚17 disk.rbd3.out‚0 disk.nvme0c0n1.in‚ 7.60546875|«!iŸò‚(:” ü¼èÒ¥Òö gpu.0.gpu‚90.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚289.503' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚57.900600000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0x‡i&Ñó‚(:Æ ü¼èÒÓ±ø$ disk./.usageGB‚555.0919036865234% disk.nvme0c0n1.out‚4833.259765625 disk.sda.in‚0 disk.rbd3.in‚0" memory_percent‚99.945068359375 disk.dm-0.out‚4833.259765625 disk.rbd3.out‚0 proc.cpu.threads‚17 disk.sda2.out‚0 disk.drbd2165.in‚0 disk.drbd2165.out‚0* proc.memory.percent‚17.347311973571777 disk.sda.out‚ 34.1953125 disk.sda1.in‚0 disk.sda1.out‚0 proc.memory.availableMB‚9 cpu‚25.6146524036139 network.recv‚334050 disk.nvme0n1.in‚ 7.90234375 disk.sda2.in‚0 disk.rbd1.out‚ 2.7265625 disk.sda3.out‚ 34.1953125 network.sent‚ 1.149671e+06* disk./.usagePercent‚37.252683409532104 disk.nvme0c0n1.in‚ 7.90234375# disk.nvme0n1.out‚4833.259765625 disk.sda3.in‚0 disk.rbd1.in‚0 disk.rbd0.out‚0# proc.memory.rssMB‚ 2842.18359375 disk.dm-1.in‚0 disk.dm-1.out‚ 34.1953125 disk.rbd0.in‚0 disk.dm-0.in‚ 7.90234375¡2ô- _wandbruntime‚1139  _runtime‚1139’`ý¦Ãõ½ % lr-AdamW/pg2‚0.000691451262316168 trainer/global_step‚699 % _timestamp‚1.7842909557586637e+09 _runtime‚1139.975837858 _step‚53 % lr-AdamW/pg1‚0.000691451262316168,¨ÍÇöÁ _step‚53 % lr-AdamW/pg1‚0.000691451262316168 % lr-AdamW/pg2‚0.000691451262316168 trainer/global_step‚699 % _timestamp‚1.7842909557586637e+09 _runtime‚1139.9758378585`Ö„L2÷- _wandbruntime‚1139  _runtime‚113991¶k8øÑO ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚55321425883648 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471111 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9971 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚199082085554225 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚3762590 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12095 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371561984 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12152 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9977 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚198515994755072 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1793217 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚54480348983296 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚198515994755072 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680115712 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚11896363 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1969373 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚55321023255040 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚199083455582507 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680115712 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚199169529342976 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚11906340 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚3762852 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 652443494449 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 trainer/global_step‚699 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚6015792 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1793222 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LBDeviceStatsMonitor.on_train_batch_start/allocatÓØŸion.small_pool.peak‚12095 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚262 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345388544 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 653560761344 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚198515968581632 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9977 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371561984 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚199170900904960 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚199169529342976 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1969630(zPÛ'ùÕO R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚199169529342976 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1969630 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚55321425883648 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471111 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9971 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚199082085554225 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚3762590 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12095 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371561984 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12152 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9977 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚198515994755072 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1793217 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚54480348983296 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚198515994755072 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680115712 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚11896363 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1969373 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚55321023255040 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚199083455582507 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680115712 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚199169529342976 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚11906340 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚3762852 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 652443494449 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 trainer/global_step‚699 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚6015792 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1793222 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12095 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚262 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345388544 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 653560761344 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚198515968581632 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9977 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371561984 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚199170900904960 IDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545123328 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚5888783 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471111 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 655831192576 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚6034255 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚54558203482112 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚11923038 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1972383 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 653366960695 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9985 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚199449826677760 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371569152 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680115712 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3767882 % _timestamp‚1.7842909571479657e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 654485796864 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371569152 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚11913047 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12095 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9991 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 _step‚55 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12152 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚5888783 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚6024270 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚55399695342080 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028394 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚11923038 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚6024270 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1972122 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚199449826677760 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚54557835390976 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚11913047 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9985 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648:pÔÈþßM H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚266 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 654711561121 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680115712 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚199448455108608 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚261 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36627456 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚198793969311744 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12152 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚5888777 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1795765 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545123328 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚5888783 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471111 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 655831192576 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚6034255 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚54558203482112 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚11923038 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1972383 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 653366960695 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9985 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚199449826677760 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371569152 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680115712 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3767882 % _timestamp‚1.7842909571479657e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 654485796864 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371569152 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚11913047 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12095 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9991 ?6DeviceStatsMonitor.on_train_batch_end/num_devicJôe_alloc‚1995 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 _step‚55 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12152 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚5888783 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚6024270 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚55399695342080 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028394 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚11923038 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚6024270 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1972122 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚199449826677760 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚54557835390976 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚11913047 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9985 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚6034255 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404718592 _runtime‚1141.365382567 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚198707523837952 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345395712 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚5888777 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚3768148 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1795760 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚199362260827041 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 655831192576 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚198707549265920 trainer/global_step‚699 FÃ@2- _wandbruntime‚1219  _runtime‚1219ç-Z×'ÑO D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9977 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚6451515 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚12742392 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚58354669367296 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028282 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚212323756396544 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 700025207871 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404725760 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 698680607557 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7842910350528715e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚213117253341184 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680115712 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚12732415 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚6451515 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345388544 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚213023781604415 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚213115881779200 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12152 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 699876691968 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680115712 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9971 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚6441544 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12095 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345388544 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12152 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545123328 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚212416031260672 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _runtime‚1219.270216717 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 699876691968 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚212323730968576 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚212416005087232 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚212416031260672 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚4027497 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 trainer/global_step‚749 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 701222080512 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚213115881779200 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 701222080512 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600314 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚266 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2106859 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471111 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚6290877 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚6290871 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1920367 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 900073071616 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚212416005087232 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318918 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9971 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚213022411576133 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚59255147164672 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 900109706240 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12095 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2107125 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648/¿+#‘ÕO YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚213023781604415 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚213115881779200 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12152 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 699876691968 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680115712 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9971 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚6441544 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12095 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345388544 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12152 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545123328 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚212416031260672 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _runtime‚1219.270216717 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 699876691968 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚212323730968576 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚212416005087232 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚212416031260672 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚4027497 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 trainer/global_step‚749 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 701222080512 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚213115881779200 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 701222080512 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600314 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚266 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2106859 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471111 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚6290877 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚6290871 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1920367 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 900073071616 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚212416005087232 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318918 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9971 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚213022411576133 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚59255147164672 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 900109706240 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12095 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2107125 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9977 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚6451515 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚12742392 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚58354669367296 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 RC"ÆbP¾DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028282 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚212323756396544 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 700025207871 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404725760 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 698680607557 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7842910350528715e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚213117253341184 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680115712 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚12732415 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚6451515 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345388544::’cw2’- _wandbruntime‚1220  _runtime‚1220;5M:á&“ÛM TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 700801727488 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚212694031990784 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 702147123200 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 901288868864 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚213396179113984 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345395712 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4032485 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚213302586848949 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚6299083 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 901252241408 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _runtime‚1220.657558009 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12095 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545123328 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345395712 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12152 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚6450022 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚12749099 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2109575 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36627456 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9991 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚12759090 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1922915 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545123328 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371569152 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1922910 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚59333408016384 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚212601638174720 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚212694005817344 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚6450022 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚6460007 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚212694031990784 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028394 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚59333812734976 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚6299083 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12152 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471111 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680115712 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚212601612746752 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 700948674229 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚262 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 699604073803 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 702147123200 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680115712 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚4032752 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚213396179113984 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345395712 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4032485 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚213302586848949 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚6299083 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 901252241408 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _runtime‚1220.657558009 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12095 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545123328 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345395712 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12152 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚6450022 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚12749099 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2109575 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36627456 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9991 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚12759090 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1922915 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545123328 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371569152 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1922910 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚59333408016384 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚212601638174720 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚212694005817344 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚6450022 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚6460007 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚212694031990784 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028394 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚59333812734976 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚6299083 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12152 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471111 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680115712 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚212601612746752 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 700948674229 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚262 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 699604073803 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 702147123200 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680115712 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚4032752 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 EDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚226962737598041 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371561984 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9971 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚226217845305344 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 746192622592 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371561984 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680115712 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚63188888471040 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚13568467 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚6701171 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471111 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚6701171 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚6877267 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12095 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12152 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 ZLDeviðˆ¹žceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 trainer/global_step‚799 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9977 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚227062234215424 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545123328 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36634624 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚227063605777408 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚13578444 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600314 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚269 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚4291946 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 746262320979 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9977 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12095 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚6867296 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318918 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2047517 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _runtime‚1298.569073053 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545123328 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚6701177 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚226316041592832 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚63188483745280 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 959493993984 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2244693åΡÛ'§ÕO C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9977 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚227062234215424 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545123328 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36634624 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚227063605777408 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚13578444 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600314 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚269 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚4291946 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 746262320979 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9977 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12095 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚6867296 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318918 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2047517 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _runtime‚1298.569073053 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545123328 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚6701177 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚226316041592832 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚63188483745280 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 959493993984 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2244693 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚226962737598041 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371561984 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9971 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚226217845305344 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 746192622592 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371561984 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680115712 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚63188888471040 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚13568467 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚6701171 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471111 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚6701171 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚6877267 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12095 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12152 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 trainer/global_step‚799>‹ÂŒ]2¨- _wandbruntime‚1299  _runtime‚1299íœzëá&©ÛM TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680115712 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9985 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545123328 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚4297513 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 960713300480 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9991 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚6875774 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 747117658112 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚226594068496384 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12152 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2050065 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚6885759 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 748463053824 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2247186 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚227242912870857 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318918 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028394 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚6709383 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404718592 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12095 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 745841186911 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚6709377 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚62306844250112 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36627456 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371569152 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345395712 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471111 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚226594068496384 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚6709377 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545123328 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚6875774 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚226495701655552 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚13585151 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚226594042322944 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚1299.955106222 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚227342531550208 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 960676673024 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚226594042322944 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2050060 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚63267557550592 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600426 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚63267152832000 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4297246 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 747185787337 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680115712 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345395712 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚13585151 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 trainer/global_step‚799 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚227342531550208 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚13595142 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚227241542842463 _step‚63 % _timestamp‚1.7842911157377403e+09 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚226495727083520 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12152 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 748463053824 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚6885759 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚262 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚13595142 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 747117658112 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4297246 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 747185787337 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680115712 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345395712 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚13585151 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 trainer/global_step‚799 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚227342531550208 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚13595142 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚227241542842463 _step‚63 % _timestamp‚1.7842911157377403e+09 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚226495727083520 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12152 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 748463053824 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚6885759 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚262 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚13595142 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 747117658112 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545123328 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚4297513 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 960713300480 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9991 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚6875774 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 747117658112 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚226594068496384 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12152 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2050065 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚6885759 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 748463053824 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2247186 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚227242912870857 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318918 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 UHDeviceStatsMonitor3À˜”.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028394 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚6709383 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404718592 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12095 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 745841186911 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚6709377 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚62306844250112 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36627456 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371569152 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345395712 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471111 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚226594068496384 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚6709377 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545123328 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚6875774 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚226495701655552 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚13585151 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚226594042322944 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚1299.955106222 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚227342531550208 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 960676673024 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚226594042322944 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2050060 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚63267557550592 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600426 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0?ùßuë2«- _wandbruntime‚1300  _runtime‚1300U•_Mÿ¬ù 1loss/FBt2/assignment_loss‚0.22952640113168551 /loss/FBt1/detection_loss‚0.7090333821076267 /loss/FBt2/detection_loss‚1.4032688970367113 $loss/total_loss‚14.2870709521641 trainer/global_step‚799 /loss/FRt1/assignment_loss‚3.371208928902786 /loss/SRqqt1/detection_loss‚1.31047396233771 % _timestamp‚1.7842911158244076e+09 2loss/SRqqt1/assignment_loss‚1.0862826654504585 .loss/FRt2/detection_loss‚1.746309117040201 epoch‚1 _step‚64 0loss/FRt2/assignment_loss‚1.7371347700898268 2loss/SRqqt2/assignment_loss‚1.1391360571629747 1loss/FBt1/assignment_loss‚0.12490223022712886 /loss/FRt1/detection_loss‚0.6005518327141366 1loss/SRqqt2/detection_loss‚0.8292427079628557 _runtime‚ 1300.041347690اY­ý /loss/FRt1/assignment_loss‚3.371208928902786 /loss/SRqqt1/detection_loss‚1.31047396233771 % _timestamp‚1.7842911158244076e+09 2loss/SRqqt1/assignment_loss‚1.0862826654504585 .loss/FRt2/detection_loss‚1.746309117040201 epoch‚1 _step‚64 0loss/FRt2/assignment_loss‚1.7371347700898268 2loss/SRqqt2/assignment_loss‚1.1391360571629747 1loss/FBt1/assignment_loss‚0.12490223022712886 /loss/FRt1/detection_loss‚0.6005518327141366 1loss/SRqqt2/detection_loss‚0.8292427079628557 _runtime‚ 1300.04134769 1loss/FBt2/assignment_loss‚0.22952640113168551 /loss/FBt1/detection_loss‚0.7090333821076267 /loss/FBt2/detection_loss‚1.4032688970367113 $loss/total_loss‚14.2870709521641 trainer/global_step‚799@ÿ3O•®‚(:Š °¾èÒº­÷ gpu.0.gpu‚100.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚338.056' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚67.6112 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0©D×'寂(:Ú °¾èÒåÉíø disk.nvme0n1.in‚ 9.54296875% disk.nvme0n1.out‚4952.99072265625 disk.sda2.in‚0$ memory_percent‚99.90968704223633% proc.memory.availableMB‚ 14.796875* disk./.usagePercent‚37.257411100666914 disk.dm-0.in‚ 9.54296875# proc.memory.rssMB‚ 2842.18359375* proc.memory.percent‚17.347311973571777 disk.sda.out‚ 37.48046875' disk.nvme0c0n1.out‚4952.99072265625 disk.rbd1.in‚0 proc.cpu.threads‚17 disk.rbd1.out‚ 3.2265625" disk.dm-0.out‚4952.99072265625 disk.sda.in‚0 disk.dm-1.in‚0 disk.rbd3.in‚0$ disk./.usageGB‚555.1623497009277 disk.sda3.out‚ 37.48046875 network.sent‚ 1.36152e+06 disk.dm-1.out‚ 37.48046875 disk.sda3.in‚0 cpu‚25.634104827991553 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 9.54296875 disk.sda2.out‚0 network.recv‚376969 disk.rbd0.in‚0 disk.drbd2165.out‚0 disk.rbd3.out‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd2165.in‚0¼“¼`•°‚(:Š ¿¾èÒø£öö gpu.0.gpu‚100.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚345.747' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚69.1494 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0qê¢â±‚(:× ¿¾èÒó–Ýø disk.sda.in‚0 disk.sda.out‚ 37.7890625 disk.dm-1.in‚0 network.sent‚ 1.424705e+06 disk.rbd0.in‚0 disk.sda1.in‚0 disk.rbd3.out‚0' disk.nvme0c0n1.out‚4961.38916015625' proc.memory.availableMB‚ 14.65234375 disk.sda3.out‚ 37.7890625 disk.nvme0n1.in‚ 9.54296875 disk.dm-0.in‚ 9.54296875 disk.sda1.out‚0% disk.nvme0n1.out‚4961.38916015625 disk.dm-1.out‚ 37.7890625# proc.memory.rssMB‚ 2842.18359375) disk./.usagePercent‚37.25761821076212 disk.rbd0.out‚0 disk.sda3.in‚0 disk.nvme0c0n1.in‚ 9.54296875 disk.rbd1.in‚0 disk.rbd1.out‚ 3.2890625" disk.dm-0.out‚4961.38916015625 disk.drbd2165.in‚0 disk.rbd3.in‚0" memory_percent‚99.910569190979 cpu‚25.663880053079126 proc.cpu.threads‚17 network.recv‚381819 disk.drbd2165.out‚0$ disk./.usageGB‚555.1654357910156 disk.sda2.in‚0 disk.sda2.out‚0* proc.memory.percent‚17.347311973571777F§ô2²- _wandbruntime‚1327  _runtime‚1327Ð0 >³ý{ 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9909065961837769 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.005662260577082634 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4166816473007202 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.4589044153690338 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 ,jet/accuracy_4_of_4‚0.009931244887411594 /particle/accuracy_3_of_3‚0.9963866472244263 _runtime‚1327.014430537 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.6990929841995239 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.2397262006998062 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 +jet/accuracy_2_of_3‚0.22851160168647766 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.4528283178806305 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.008957412093877792 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.10559604316949844 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5421232581138611 :"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.21789434552192688 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.4793686270713806 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 6 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.991727352142334 particle/accuracy_1_of_2‚1 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9814109802246094 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.18328721821308136 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.12861506640911102 :"Purity/0FRt*SRqqt*FBt/event_purity‚0.31848299503326416 :"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.24484188854694366 :"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.23185864090919495 /particle/accuracy_2_of_4‚0.9968253374099731 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.11538461595773697 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.05774269625544548 :"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.49977508187294006 particle/accuracy_5_of_5‚0 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 particle/accuracy_2_of_3‚1 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.28240740299224854 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.48570340871810913 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9948257207870483 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9959829449653625 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.07110532373189926 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.8657407164573669 9"Purity/*FRt2SRqqt0FBt/event_purity‚0.0294208787381649 :"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.43965208530426025 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.17143148183822632 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.2345517873764038 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.2734375 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.06839869916439056 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ,jet/accuracy_3_of_4‚0.051045190542936325 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9748831987380981 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 *jet/accuracy_1_of_2‚0.4398585855960846 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.0930686891078949 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9945489764213562 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.13648822903633118 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.3153091073036194 :"Purity/0FRt1SRqqt0FBt/event_purity‚0.22244426608085632 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 particle/accuracy_1_of_5‚1 % _timestamp‚1.7842911427966464e+09 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9062219262123108 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.6666666865348816 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.06399862468242645 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.6965985894203186 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 :"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.27656418085098267 particle/accuracy_3_of_5‚1 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.4766666293144226 7validation_average_jet_accuracy‚0.17445512115955353 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.1296675205230713 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.08494149893522263 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.07833333313465118 :"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.27656418085098267 +jet/accuracy_1_of_1‚0.13679295778274536 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9906969666481018 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.24081766605377197 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 :"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.25815558433532715 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 particle/accuracy_1_of_3‚1 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.13482218980789185 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.2222222238779068 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.4774475693702698 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 particle/accuracy_2_of_2‚1 )loss/val_total_loss‚15.73146826937897 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.6880574226379395 :"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.22263917326927185 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.004578232299536467 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 jet/accuracy_2_of_5‚1 epoch‚1 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.6499999761581421 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9924145936965942 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.12964951992034912 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.14454549551010132 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.1958005726337433 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9901702404022217 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.2222222238779068 /particle/accuracy_3_of_4‚0.9870250225067139 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9808476567268372 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.0713128075003624 :"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.22543534636497498 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.34403857588768005 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.18841460347175598 particle/accuracy_2_of_5‚1 particle/accuracy_4_of_5‚0 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9971277117729187 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 :"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.25018224120140076 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.010064174421131611 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.06625206768512726 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 :"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.18855755031108856 particle/accuracy_1_of_4‚1 jet/accuracy_5_of_5‚0 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 :"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.24780060350894928 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.5286458730697632 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.6448553800582886 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.1944444477558136 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.0906282514333725 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.0681820958852768 +jet/accuracy_2_of_2‚0.07535846531391144 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.27184805274009705 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9682255983352661 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.011633043177425861 7 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.1854173243045807 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 )jet/accuracy_1_of_4‚0.768945574760437 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.2598334848880768 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.7777777910232544 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.3203752636909485 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.20933791995048523 jet/accuracy_4_of_5‚0 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.43518519401550293 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9616955518722534 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.17152689397335052 6 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.942307710647583 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.20621994137763977 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.4459412395954132 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 +particle/sensitivity‚0.5324462056159973 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9963602423667908 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.36384791135787964 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.05000000074505806 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.6898148059844971 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.5625 :"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.24344287812709808 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.22830429673194885 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 (particle/accuracy‚0.8069235682487488 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.11538461595773697 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9940094947814941 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4075143337249756 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.2750000059604645 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.17288100719451904 jet/accuracy_3_of_5‚1 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9776639342308044 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 ;"Purity/1FRt0SRqqt0FBt/event_purity‚0.058832019567489624 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.18257449567317963 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.4285714328289032 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8048663139343262 8"Purity/1FRt0SRqqt2FBt/event_purity‚0.601738452911377 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.20933791995048523 *jet/accuracy_1_of_3‚0.6775723099708557 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8350961804389954 8"Purity/*FRt0SRqqt2FBt/event_purity‚0.871371328830719 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.18403135240077972 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 :"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.22244426608085632 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.10254856199026108 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.5519652366638184 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.04575027897953987 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 ;"Purity/*FRt0SRqqt0FBt/event_purity‚0.056699447333812714 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 ;"Purity/1FRt*SRqqt*FBt/event_purity‚0.052058275789022446 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.6916855573654175 'particle/f_score‚0.5421954393386841 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.05000000074505806 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 +jet/accuracy_2_of_4‚0.29505911469459534 _step‚65 8"Purity/*FRt*SRqqt2FBt/event_purity‚0.815018355846405 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.3157455027103424 *jet/accuracy_3_of_3‚0.0448039248585701 trainer/global_step‚799 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9911687970161438 9 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.058832019567489624 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.6898148059844971 particle/accuracy_1_of_1‚1 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.19468571245670319 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.32181864976882935 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9833332896232605 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.3200882375240326 6 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.986784815788269 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 /particle/accuracy_4_of_4‚0.9308899641036987 /"Purity/2FRt0SRqqt1FBt/event_purity‚0.171875 jet/accuracy_1_of_5‚1 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.02878374420106411 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9959829449653625 +particle/specificity‚0.8819905519485474 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.375 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.5714285969734192 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.2533623278141022 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.019401410594582558 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 ;"Purity/1FRt*SRqqt0FBt/event_purity‚0.043885715305805206 6 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.986586332321167 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5603371858596802 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9865365028381348 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.32283836603164673 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 :"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.23611575365066528 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.1944444477558136 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9915344715118408 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.112808614969253540‡Ì»Å´| :"Purity/*FRt0SRqqt*FBt/event_purity‚0.13648822903633118 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.3153091073036194 :"Purity/0FRt1SRqqt0FBt/event_purity‚0.22244426608085632 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 particle/accuracy_1_of_5‚1 % _timestamp‚1.7842911427966464e+09 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9062219262123108 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.6666666865348816 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.06399862468242645 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.6965985894203186 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 :"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.27656418085098267 particle/accuracy_3_of_5‚1 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.4766666293144226 7validation_average_jet_accuracy‚0.17445512115955353 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.1296675205230713 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.08494149893522263 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.07833333313465118 :"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.27656418085098267 +jet/accuracy_1_of_1‚0.13679295778274536 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9906969666481018 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.24081766605377197 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 :"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.25815558433532715 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 particle/accuracy_1_of_3‚1 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.13482218980789185 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.2222222238779068 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.4774475693702698 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 particle/accuracy_2_of_2‚1 )loss/val_total_loss‚15.73146826937897 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.6880574226379395 :"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.22263917326927185 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.004578232299536467 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 jet/accuracy_2_of_5‚1 epoch‚1 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.6499999761581421 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9924145936965942 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.12964951992034912 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.14454549551010132 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.1958005726337433 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9901702404022217 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.2222222238779068 /particle/accuracy_3_of_4‚0.9870250225067139 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9808476567268372 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.0713128075003624 :"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.22543534636497498 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.34403857588768005 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.18841460347175598 particle/accuracy_2_of_5‚1 particle/accuracy_4_of_5‚0 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9971277117729187 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 :"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.25018224120140076 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.010064174421131611 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.06625206768512726 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 :"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.18855755031108856 particle/accuracy_1_of_4‚1 jet/accuracy_5_of_5‚0 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 :"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.24780060350894928 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.5286458730697632 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.6448553800582886 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.1944444477558136 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.0906282514333725 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.0681820958852768 +jet/accuracy_2_of_2‚0.07535846531391144 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.27184805274009705 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9682255983352661 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.011633043177425861 7 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.1854173243045807 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 )jet/accuracy_1_of_4‚0.768945574760437 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.2598334848880768 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.7777777910232544 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.3203752636909485 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.20933791995048523 jet/accuracy_4_of_5‚0 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.43518519401550293 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9616955518722534 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.17152689397335052 6 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.942307710647583 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.20621994137763977 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.4459412395954132 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 +particle/sensitivity‚0.5324462056159973 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9963602423667908 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.36384791135787964 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.05000000074505806 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.6898148059844971 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.5625 :"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.24344287812709808 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.22830429673194885 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 (particle/accuracy‚0.8069235682487488 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.11538461595773697 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9940094947814941 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4075143337249756 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.2750000059604645 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.17288100719451904 jet/accuracy_3_of_5‚1 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9776639342308044 ("Purity/0FRt2SRqqt1FBt/event_puEûæwB"rity‚0 ;"Purity/1FRt0SRqqt0FBt/event_purity‚0.058832019567489624 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.18257449567317963 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.4285714328289032 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8048663139343262 8"Purity/1FRt0SRqqt2FBt/event_purity‚0.601738452911377 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.20933791995048523 *jet/accuracy_1_of_3‚0.6775723099708557 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8350961804389954 8"Purity/*FRt0SRqqt2FBt/event_purity‚0.871371328830719 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.18403135240077972 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 :"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.22244426608085632 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.10254856199026108 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.5519652366638184 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.04575027897953987 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 ;"Purity/*FRt0SRqqt0FBt/event_purity‚0.056699447333812714 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 ;"Purity/1FRt*SRqqt*FBt/event_purity‚0.052058275789022446 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.6916855573654175 'particle/f_score‚0.5421954393386841 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.05000000074505806 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 +jet/accuracy_2_of_4‚0.29505911469459534 _step‚65 8"Purity/*FRt*SRqqt2FBt/event_purity‚0.815018355846405 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.3157455027103424 *jet/accuracy_3_of_3‚0.0448039248585701 trainer/global_step‚799 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9911687970161438 9 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.058832019567489624 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.6898148059844971 particle/accuracy_1_of_1‚1 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.19468571245670319 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.32181864976882935 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9833332896232605 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.3200882375240326 6 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.986784815788269 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 /particle/accuracy_4_of_4‚0.9308899641036987 /"Purity/2FRt0SRqqt1FBt/event_purity‚0.171875 jet/accuracy_1_of_5‚1 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.02878374420106411 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9959829449653625 +particle/specificity‚0.8819905519485474 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.375 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.5714285969734192 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.2533623278141022 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.019401410594582558 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 ;"Purity/1FRt*SRqqt0FBt/event_purity‚0.043885715305805206 6 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.986586332321167 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5603371858596802 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9865365028381348 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.32283836603164673 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 :"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.23611575365066528 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.1944444477558136 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9915344715118408 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.11280861496925354 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9909065961837769 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.005662260577082634 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4166816473007202 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.4589044153690338 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 ,jet/accuracy_4_of_4‚0.009931244887411594 /particle/accuracy_3_of_3‚0.9963866472244263 _runtime‚1327.014430537 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.6990929841995239 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.2397262006998062 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 +jet/accuracy_2_of_3‚0.22851160168647766 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.4528283178806305 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.008957412093877792 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.10559604316949844 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5421232581138611 :"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.21789434552192688 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.4793686270713806 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 6 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.991727352142334 particle/accuracy_1_of_2‚1 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9814109802246094 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.18328721821308136 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.12861506640911102 :"Purity/0FRt*SRqqt*FBt/event_purity‚0.31848299503326416 :"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.24484188854694366 :"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.23185864090919495 /particle/accuracy_2_of_4‚0.9968253374099731 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.11538461595773697 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.05774269625544548 :"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.49977508187294006 particle/accuracy_5_of_5‚0 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 particle/accuracy_2_of_3‚1 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.28240740299224854 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.48570340871810913 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9948257207870483 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9959829449653625 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.07110532373189926 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.8657407164573669 9"Purity/*FRt2SRqqt0FBt/event_purity‚0.0294208787381649 :"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.43965208530426025 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.17143148183822632 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.2345517873764038 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.2734375 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.06839869916439056 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ,jet/accuracy_3_of_4‚0.051045190542936325 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9748831987380981 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 *jet/accuracy_1_of_2‚0.4398585855960846 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.0930686891078949 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9945489764213562AÐH³I”µ‚(:‰ ξèÒÙ€¯÷ gpu.0.gpu‚100.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚41.0 gpu.0.powerWatts‚335.265' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚67.053 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Å?5ú綂(:Ü ξèÒøòœù disk.sda.in‚0 disk.rbd0.out‚0 cpu‚25.675302990030303* disk./.usagePercent‚37.325109733021414 disk.drbd2165.in‚0' disk.nvme0c0n1.out‚5780.53564453125 disk.rbd1.in‚0 disk.sda1.in‚0 disk.sda3.out‚ 38.03515625 disk.rbd0.in‚0 disk.dm-0.in‚ 9.57421875" disk.dm-0.out‚5780.53564453125 network.sent‚ 1.42969e+06 network.recv‚385116$ disk./.usageGB‚556.1711082458496% disk.nvme0n1.out‚5780.53564453125 disk.sda2.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 38.03515625 disk.nvme0c0n1.in‚ 9.57421875 disk.nvme0n1.in‚ 9.57421875$ memory_percent‚88.12181949615479" proc.memory.rssMB‚ 2842.0078125 disk.drbd2165.out‚0 disk.rbd3.out‚0 disk.sda.out‚ 38.03515625 disk.sda3.in‚0) proc.memory.availableMB‚ 1946.12109375 proc.cpu.threads‚17 disk.rbd3.in‚0 disk.sda1.out‚0) proc.memory.percent‚17.34623908996582 disk.sda2.out‚0 disk.rbd1.out‚ 3.2890625½Ê¼”·‚(:‰ ݾèÒªØÿö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚260.461' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.0922 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0÷aÞ7߸‚(:Ô ݾèÒÒÜŠù disk.sda2.out‚0 disk.nvme0n1.in‚ 9.87109375# disk.nvme0n1.out‚7142.513671875 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.dm-0.in‚ 9.87109375 disk.sda2.in‚0) disk./.usagePercent‚37.32541591803237 disk.sda.in‚0 disk.rbd3.out‚0 disk.nvme0c0n1.in‚ 9.87109375 disk.rbd1.in‚0 disk.dm-1.in‚0 disk.sda1.out‚0 cpu‚25.675032162644648$ disk./.usageGB‚556.1756706237793 disk.dm-1.out‚ 38.2578125 disk.drbd2165.in‚0$ memory_percent‚88.12344074249268) proc.memory.percent‚17.34769344329834 disk.sda1.in‚0 disk.drbd2165.out‚0) proc.memory.availableMB‚ 1945.85546875 network.recv‚390015 disk.sda.out‚ 38.2578125 disk.rbd0.out‚0 disk.rbd3.in‚0 disk.rbd1.out‚ 3.4140625# proc.memory.rssMB‚ 2842.24609375 network.sent‚ 1.487391e+06 disk.dm-0.out‚7142.513671875 disk.sda3.in‚0 disk.sda3.out‚ 38.2578125% disk.nvme0c0n1.out‚7142.5136718756ÊßQ“¹‚(:ˆ ì¾èÒÿžÜö gpu.0.gpu‚14.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚37.0 gpu.0.powerWatts‚109.884' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.9768 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0­qÐbߺ‚(:Ô ì¾èҾݶø disk.drbd2165.out‚0 disk.sda2.out‚0 disk.rbd1.out‚ 3.4140625( proc.memory.availableMB‚ 1945.7734375 disk.drbd2165.in‚0 disk.rbd3.in‚0# disk.nvme0n1.out‚7152.181640625 disk.sda2.in‚0 cpu‚25.67604768991412$ disk./.usageGB‚556.1827354431152 disk.rbd0.out‚0 disk.sda.in‚0# proc.memory.rssMB‚ 2842.24609375) proc.memory.percent‚17.34769344329834 network.sent‚ 1.49248e+06 network.recv‚393098 disk.dm-1.in‚0 disk.dm-0.in‚ 9.90234375 disk.rbd3.out‚0% disk.nvme0c0n1.out‚7152.181640625 disk.rbd1.in‚0 disk.sda.out‚ 38.42578125$ memory_percent‚88.12394142150879 disk.sda3.in‚0 disk.sda3.out‚ 38.42578125 disk.sda1.in‚0 disk.sda1.out‚0) disk./.usagePercent‚37.32589004398578 disk.dm-1.out‚ 38.42578125 disk.nvme0c0n1.in‚ 9.90234375 proc.cpu.threads‚17 disk.nvme0n1.in‚ 9.90234375 disk.rbd0.in‚0 disk.dm-0.out‚7152.181640625(â+>”»‚(:‰ û¾èÒÔ»Ïö gpu.0.gpu‚90.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚268.551' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.7102 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0™iÝì漂(:Û û¾èҷŨø network.sent‚ 1.497454e+06 disk.drbd2165.out‚0 disk.rbd3.out‚0! disk.nvme0c0n1.in‚ 10.19921875 network.recv‚396700 disk.dm-1.in‚0 disk.rbd0.out‚0& disk.nvme0c0n1.out‚7165.9697265625 disk.nvme0n1.in‚ 10.19921875 disk.dm-1.out‚ 38.69140625 disk.rbd1.in‚0 disk.sda.out‚ 38.69140625# proc.memory.rssMB‚ 2842.24609375) proc.memory.percent‚17.34769344329834$ disk./.usageGB‚556.1893348693848# memory_percent‚88.1239652633667 disk.sda1.in‚0 disk.sda3.out‚ 38.69140625 disk.rbd3.in‚0 disk.sda2.in‚0) disk./.usagePercent‚37.32633293702002 disk.drbd2165.in‚0 disk.sda.in‚0) proc.memory.availableMB‚ 1945.76953125 cpu‚25.67612985126689 proc.cpu.threads‚17 disk.rbd0.in‚0! disk.dm-0.out‚7165.9697265625 disk.sda1.out‚0 disk.rbd1.out‚ 3.4765625 disk.dm-0.in‚ 10.19921875 disk.sda3.in‚0$ disk.nvme0n1.out‚7165.9697265625 disk.sda2.out‚0tÖpŸ½‚(:” Š¿èÒ楈ø gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚299.991' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚59.998200000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¥&†’Û¾‚(:Ð Š¿èÒ®¬äø# proc.memory.rssMB‚ 2842.24609375 disk.dm-0.out‚ 7172.1328125 disk.sda1.out‚0 disk.sda3.in‚0$ memory_percent‚88.12370300292969 network.sent‚ 1.502572e+06# disk.nvme0c0n1.out‚ 7172.1328125 cpu‚25.676188530619648 disk.rbd0.in‚0 disk.sda3.out‚ 38.98828125 disk.rbd3.out‚0) proc.memory.percent‚17.34769344329834 disk.dm-1.in‚0 disk.drbd2165.out‚0 disk.rbd3.in‚0! disk.nvme0c0n1.in‚ 10.19921875 disk.sda.in‚0 disk.dm-0.in‚ 10.19921875 disk.sda1.in‚0! disk.nvme0n1.out‚ 7172.1328125 disk.sda2.in‚0 network.recv‚399933$ disk./.usageGB‚556.1923904418945 proc.cpu.threads‚17) disk./.usagePercent‚37.32653799905494 disk.sda.out‚ 38.98828125 disk.dm-1.out‚ 38.98828125 disk.rbd0.out‚0 disk.drbd2165.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 3.4765625 disk.nvme0n1.in‚ 10.19921875 disk.sda2.out‚0% proc.memory.availableMB‚ 1945.8125TQÑ”¿‚(:‰ ™¿èÒ‡ìïö gpu.0.gpu‚90.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚263.118' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.6236 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0f»ó¨×À‚(:Ì ™¿èÒ»íÍø disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd2165.out‚0 disk.nvme0c0n1.in‚ 10.5078125) proc.memory.percent‚17.34769344329834 disk.nvme0n1.in‚ 10.5078125 disk.rbd1.in‚0 disk.drbd2165.in‚0 disk.sda2.in‚0 disk.sda2.out‚0) disk./.usagePercent‚37.32697090779534 disk.dm-0.out‚ 7182.2734375 disk.sda3.out‚ 39.3203125# disk.nvme0c0n1.out‚ 7182.2734375 disk.sda.in‚0 disk.dm-0.in‚ 10.5078125 disk.rbd1.out‚ 3.5390625$ disk./.usageGB‚556.1988410949707 disk.dm-1.out‚ 39.3203125 disk.rbd0.out‚0 disk.sda3.in‚0 disk.rbd3.out‚0! disk.nvme0n1.out‚ 7182.2734375# proc.memory.rssMB‚ 2842.24609375 cpu‚25.67644405684675 network.sent‚ 1.507553e+06 network.recv‚403534 disk.sda.out‚ 39.3203125 proc.cpu.threads‚17 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.rbd3.in‚0$ memory_percent‚88.12699317932129( proc.memory.availableMB‚ 1945.2734375µfŒ2Á- _wandbruntime‚1410  _runtime‚1410Í¡7äÅ¿ & lr-AdamW/pg1‚0.0006909529967948398 & lr-AdamW/pg2‚0.0006909529967948398 trainer/global_step‚849 % _timestamp‚1.7842912266136909e+09 _runtime‚1410.830908053 _step‚664sÉÃà & lr-AdamW/pg1‚0.0006909529967948398 & lr-AdamW/pg2‚0.0006909529967948398 trainer/global_step‚849 % _timestamp‚1.7842912266136909e+09 _runtime‚1410.830908053 _step‚66BŠ,ÍS2Ä- _wandbruntime‚1410  _runtime‚1410‘Ði×Ù'ÅÓO SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116224 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚7243477 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028286 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚7243483 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545123840 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚7362002 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12096 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚14605485 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12153 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚243937817718784 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 798351164928 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚4598981 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚221 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 796983965207 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚226 trainer/global_step‚849 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389056 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚244044085706752 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9978 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚7243483 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚4598755 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2183585 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚7352030 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚244736171712277 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 798351164928 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2415170 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚244842410698240 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 799696553984 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚7352030 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2183590 _runtime‚1410.832107648 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚244044059533312 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚32439808 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1027147287040 % _timestamp‚1.7842912266147418e+09 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12096 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371562496 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2415391 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚67178970200064 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 400530944 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚14595507 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚66151422382080 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚243937843146752 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9972 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚244843782260736 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚7243477 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚7362002 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 798328565525 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚244044059533312 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 799696553984 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚244843782260736 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚244842410698240 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116224 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600318 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545123840 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9978 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1027179726848 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318922 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371562496 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚244044085706752 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9972 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _step‚67 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚66151790473216 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚244842410698240 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116224 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600318 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545123840 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9978 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1027179726848 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318922 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371562496 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚244044085706752 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9972 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _step‚67 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚66151790473216 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12096 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚14605485 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12153 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚243937817718784 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 798351164928 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚4598981 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚221 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 796983965207 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚226 trainer/global_step‚849 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389056 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚244044085706752 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008C^L”=2Ç- _wandbruntime‚1412  _runtime‚1412ÄšÚœâ&ÈÜM NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚4604250 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2186133 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 797907431453 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12153 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚66228908789760 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚14622183 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚7251689 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 799276200448 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471115 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚7370494 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚244322086436864 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚14612191 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚244322060263424 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9986 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4604034 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚67257671066624 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371569664 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚216 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 799252031883 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545123840 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12096 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _runtime‚1412.218208666 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116224 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2417906 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1028361753088 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600430 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028398 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚244215724924928 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318922 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚14622183 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚66229276880896 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚244215699496960 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚7360508 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚7360508 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2186128 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚245014976956811 _step‚68 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371569664 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 800621596672 trainer/global_step‚849 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545123840 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9992 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚245122708033536 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚7370494X5Éæ&ÉàM PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545123840 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9992 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚245122708033536 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚7370494 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚4604250 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2186133 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 797907431453 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12153 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚66228908789760 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚14622183 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚7251689 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 799276200448 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471115 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚7370494 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚244322086436864 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚14612191 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚244322060263424 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9986 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4604034 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚67257671066624 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371569664 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚216 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 799252031883 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545123840 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12096 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _runtime‚1412.218208666 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116224 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2417906 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1028361753088 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600430 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028398 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚244215724924928 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318922 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚14622183 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚66229276880896 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚244215699496960 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚7360508 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚7360508 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2186128 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚245014976956811 _step‚68 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371569664 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 800621596672 trainer/global_step‚849DܸØC2Ê- _wandbruntime‚1412  _runtime‚1412u8¥;Ëþ 1loss/SRqqt2/detection_loss‚0.7922258428595168 /loss/FBt2/detection_loss‚1.8528388440076793 &loss/total_loss‚15.935829480068517 0loss/FRt1/assignment_loss‚3.3093308260043464 0loss/FBt2/assignment_loss‚0.7922199131173568 /loss/FRt2/detection_loss‚1.8168469740490263 1loss/SRqqt1/detection_loss‚1.3370559168422325 trainer/global_step‚849 2loss/SRqqt2/assignment_loss‚1.1590748894003888 1loss/FBt1/assignment_loss‚0.16683481191716215 /loss/FBt1/detection_loss‚1.1979920828309332 % _timestamp‚1.7842912280871773e+09 _runtime‚1412.304111264 /loss/FRt2/assignment_loss‚1.733664379792782 2loss/SRqqt1/assignment_loss‚1.2035759163176656 /loss/FRt1/detection_loss‚0.5741690829294271 epoch‚2 _step‚69pïÌ‚ 2loss/SRqqt2/assignment_loss‚1.1590748894003888 1loss/FBt1/assignment_loss‚0.16683481191716215 /loss/FBt1/detection_loss‚1.1979920828309332 % _timestamp‚1.7842912280871773e+09 _runtime‚1412.304111264 /loss/FRt2/assignment_loss‚1.733664379792782 2loss/SRqqt1/assignment_loss‚1.2035759163176656 /loss/FRt1/detection_loss‚0.5741690829294271 epoch‚2 _step‚69 1loss/SRqqt2/detection_loss‚0.7922258428595168 /loss/FBt2/detection_loss‚1.8528388440076793 &loss/total_loss‚15.935829480068517 0loss/FRt1/assignment_loss‚3.3093308260043464 0loss/FBt2/assignment_loss‚0.7922199131173568 /loss/FRt2/detection_loss‚1.8168469740490263 1loss/SRqqt1/detection_loss‚1.3370559168422325 trainer/global_step‚849EN~þ0ŸÍ‚(:” ¨¿èÒ¸‹·÷ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚274.958' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚54.991600000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0—×`øß΂(:Ô ¨¿èÒáÂøø disk.drbd2165.out‚0 disk.rbd3.in‚0 disk.sda3.out‚ 39.453125# proc.memory.rssMB‚ 2842.24609375 network.sent‚ 1.512534e+06 disk.dm-1.in‚0 disk.dm-0.in‚ 10.8046875$ memory_percent‚88.12553882598877$ disk./.usageGB‚556.1490821838379 disk.sda2.in‚0 disk.rbd1.in‚0 disk.sda.in‚0 proc.cpu.threads‚17 disk.drbd2165.in‚0& disk.nvme0c0n1.out‚7193.9248046875! disk.dm-0.out‚7193.9248046875) proc.memory.availableMB‚ 1945.51171875 network.recv‚406645 disk.rbd3.out‚0 disk.nvme0c0n1.in‚ 10.8046875 disk.nvme0n1.in‚ 10.8046875 disk.sda2.out‚0 disk.sda1.in‚0 cpu‚25.67650300285899 disk.rbd1.out‚ 3.5390625 disk.rbd0.in‚0* disk./.usagePercent‚37.323631545518715$ disk.nvme0n1.out‚7193.9248046875 disk.dm-1.out‚ 39.453125 disk.rbd0.out‚0 disk.sda1.out‚0) proc.memory.percent‚17.34769344329834 disk.sda.out‚ 39.453125 disk.sda3.in‚0}/”Ï‚(:‰ ·¿èÒ™š”÷ gpu.0.gpu‚99.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚331.111' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚66.2222 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ç!”ÃêЂ(:ß ·¿èÒñö¢ù network.sent‚ 1.575294e+06 disk.nvme0n1.in‚ 11.11328125 disk.rbd0.out‚0 disk.dm-0.in‚ 11.11328125) disk./.usagePercent‚37.32407546258308 disk.rbd1.out‚ 3.6015625 disk.sda.out‚ 39.69921875 disk.dm-1.out‚ 39.69921875" disk.dm-0.out‚7204.39892578125 disk.drbd2165.in‚0 disk.sda1.out‚0) proc.memory.percent‚17.34769344329834 disk.drbd2165.out‚0 disk.sda2.in‚0 disk.rbd1.in‚0 disk.sda.in‚0 disk.sda3.in‚0$ disk./.usageGB‚556.1556968688965% disk.nvme0n1.out‚7204.39892578125 disk.sda2.out‚0 cpu‚25.676560879835417! disk.nvme0c0n1.in‚ 11.11328125$ memory_percent‚88.12804222106934 disk.rbd3.in‚0' disk.nvme0c0n1.out‚7204.39892578125 disk.dm-1.in‚0 disk.sda1.in‚0 proc.cpu.threads‚17 network.recv‚410941 disk.rbd3.out‚0 disk.rbd0.in‚0 disk.sda3.out‚ 39.69921875( proc.memory.availableMB‚ 1945.1015625# proc.memory.rssMB‚ 2842.24609375¼ÎãÀžÑ‚(:“ Æ¿èÒèä§÷ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚288.679' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚57.73579999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ZâÞßÒ‚(:Ô Æ¿èÒ¿ª’ù# memory_percent‚88.1296157836914$ disk./.usageGB‚556.1615676879883 disk.dm-0.out‚ 7213.27734375 disk.sda3.in‚0" disk.nvme0n1.out‚ 7213.27734375 disk.sda.in‚0 disk.dm-1.in‚0 disk.rbd0.out‚0 disk.drbd2165.out‚0 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.dm-1.out‚ 39.91796875 disk.sda3.out‚ 39.91796875$ disk.nvme0c0n1.out‚ 7213.27734375# proc.memory.rssMB‚ 2842.24609375 network.recv‚414115* disk./.usagePercent‚37.324469458178285 disk.dm-0.in‚ 11.41015625 disk.sda1.out‚0! disk.nvme0c0n1.in‚ 11.41015625 disk.sda2.in‚0 disk.drbd2165.in‚0 disk.nvme0n1.in‚ 11.41015625& proc.memory.availableMB‚ 1944.84375) proc.memory.percent‚17.34769344329834 cpu‚25.676127003978806 proc.cpu.threads‚17 network.sent‚ 1.580286e+06 disk.sda.out‚ 39.91796875 disk.rbd1.in‚0 disk.rbd0.in‚0 disk.sda1.in‚0 disk.sda2.out‚0 disk.rbd1.out‚ 3.6015625‚-õŸÓ‚(:” Õ¿èÒùÝäö gpu.0.gpu‚100.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚328.786' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚65.75720000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0u<¦ êÔ‚(:ß Õ¿èÒ™ÐÇø' disk.nvme0c0n1.out‚7220.14013671875) proc.memory.percent‚17.34769344329834 disk.rbd1.in‚0 disk.dm-1.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda3.in‚0 disk.rbd3.in‚0! disk.nvme0c0n1.in‚ 11.41796875 disk.sda3.out‚ 40.18359375 disk.drbd2165.in‚0 disk.nvme0n1.in‚ 11.41796875 disk.sda2.out‚0$ memory_percent‚88.12816143035889) proc.memory.availableMB‚ 1945.08203125 network.recv‚417805 disk.dm-1.out‚ 40.18359375# proc.memory.rssMB‚ 2842.24609375 cpu‚25.67689000926905 network.sent‚ 1.585315e+06$ disk./.usageGB‚556.1652755737305 disk.sda.out‚ 40.18359375 disk.rbd0.in‚0 disk.rbd3.out‚0 disk.sda2.in‚0 proc.cpu.threads‚17) disk./.usagePercent‚37.32471829750156 disk.rbd1.out‚ 3.6640625 disk.dm-0.in‚ 11.41796875" disk.dm-0.out‚7220.14013671875 disk.drbd2165.out‚0% disk.nvme0n1.out‚7220.14013671875 disk.rbd0.out‚0 disk.sda.in‚02.$(Õ‚(:‰ ä¿èÒ®çðö gpu.0.gpu‚¯„*/l99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚268.774' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.7548 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0™keqÊÖ‚(:¿ ä¿èÒÚ‹ûø disk.sda3.out‚40.4375 disk.rbd3.in‚0% disk.nvme0c0n1.out‚7231.634765625' proc.memory.availableMB‚ 1945.328125# disk./.usageGB‚556.170768737793 disk.sda2.in‚0 disk.rbd1.out‚ 3.6640625 disk.drbd2165.in‚0 disk.rbd3.out‚0 disk.dm-1.out‚40.4375 disk.dm-0.in‚11.71875 disk.drbd2165.out‚0$ memory_percent‚88.12665939331055 disk.rbd0.out‚0 disk.sda1.in‚0 proc.cpu.threads‚17 disk.nvme0n1.in‚11.71875 disk.rbd1.in‚0 disk.rbd0.in‚0 disk.sda1.out‚0 disk.sda3.in‚0 disk.nvme0c0n1.in‚11.71875 proc.memory.rssMB‚ 2841.78125) disk./.usagePercent‚37.32508694835087 disk.sda2.out‚0 disk.dm-1.in‚0) proc.memory.percent‚17.34485626220703 disk.sda.in‚0 cpu‚25.676608428040353 network.sent‚ 1.590301e+06 network.recv‚421219 disk.sda.out‚40.4375 disk.dm-0.out‚7231.634765625# disk.nvme0n1.out‚7231.634765625ýÀl2×- _wandbruntime‚1490  _runtime‚1490‚ö͈Äؾ & lr-AdamW/pg1‚0.0006907425715334865 & lr-AdamW/pg2‚0.0006907425715334865 trainer/global_step‚899 % _timestamp‚1.7842913059293985e+09 _runtime‚ 1490.14652236 _step‚70ø°*ÈÙ _step‚70 & lr-AdamW/pg1‚0.0006907425715334865 & lr-AdamW/pg2‚0.0006907425715334865 trainer/global_step‚899 % _timestamp‚1.7842913059293985e+09 _runtime‚ 1490.14652236FðŒ2Ú- _wandbruntime‚1490  _runtime‚1490idþÙ'ÛÓO M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚15441537 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚257831906627584 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 _step‚71 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚7787754 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚70025742766080 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 % _timestamp‚1.7842913059304564e+09 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389056 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1087614487552 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318922 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚7653783 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471115 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚7653777 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚70026110857216 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 843221078315 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116224 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚257944122212352 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚7653783 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9978 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚71113322716672 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 846012484608 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12096 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545123840 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12096 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2554119 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536960 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚7777782 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚258788763134464 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 846012484608 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚257944122212352 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12153 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚15431559 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12153 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371562496 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚4864609 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚258788763134464 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚250 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402628096 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚257944096038912 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9978 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚71113322716672 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 846012484608 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12096 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545123840 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12096 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2554119 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536960 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚7777782 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚258788763134464 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 846012484608 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚257944122212352 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12153 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚15431559 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12153 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371562496 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚4864609 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚258788763134464 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚250 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402628096 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚257944096038912 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚257831906627584 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 _step‚71 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚7787754 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚70025742766080 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 % _timestamp‚1.7842913059304564e+09 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389056 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1087614487552 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318922 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚7653783 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471115 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚7653777 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚70026110857216 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 843221078315 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116224 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚257944122212352 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚7653783 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504GT:†2Ý- _wandbruntime‚1491  _runtime‚1491#Šzâ&ÞÜM SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9986 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚237 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471115 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116224 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371569664 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12153 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚7796246 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚71192416479232 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚258222122942464 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4869898 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2313283 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚7661989 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371569664 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚232 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚258953932950321 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚259067688900096 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 844144544561 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚7786260 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2313278 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12153 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚7796246 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚15448243 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚15458235 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚71192013858304 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9992 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545123840 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1088819214336 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚258109788405760 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318922 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 845592131072 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚258955302978719 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028398 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9986 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 845592131072 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116224 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚4870135 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚70103597264896 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396224 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚15448243 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚15458235 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚258222096769024 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529792 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 $ _timestamp‚1.784291307317421e+09 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚258222096769024 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396224 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚7661983 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 trainer/global_step‚899 _step‚72 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚259069060469760 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚258109813833728M¹-…ÃßàM LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚7661983 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚117ÚL'F#%5 trainer/global_step‚899 _step‚72 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚259069060469760 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚258109813833728 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9986 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚237 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471115 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116224 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371569664 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12153 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚7796246 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚71192416479232 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚258222122942464 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4869898 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2313283 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚7661989 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371569664 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚232 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚258953932950321 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚259067688900096 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 844144544561 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚7786260 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2313278 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12153 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚7796246 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚15448243 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚15458235 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚71192013858304 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9992 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545123840 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1088819214336 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚258109788405760 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318922 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 845592131072 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚258955302978719 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028398 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9986 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 845592131072 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116224 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚4870135 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚70103597264896 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396224 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚15448243 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚15458235 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚258222096769024 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529792 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 $ _timestamp‚1.784291307317421e+09 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚258222096769024 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396224 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1H¬{2à- _wandbruntime‚1491  _runtime‚1491ïñëáý 2loss/SRqqt1/assignment_loss‚1.1272451813737487 1loss/SRqqt2/detection_loss‚0.8293584515910823 epoch‚2 1loss/FBt1/assignment_loss‚0.14210865496175526 &loss/total_loss‚14.917363331761859 % _timestamp‚1.7842913074045775e+09 _step‚73 /loss/FRt2/assignment_loss‚1.845679993273446 2loss/SRqqt2/assignment_loss‚1.1698870441866849 1loss/FBt2/assignment_loss‚0.31152170537194857 /loss/FBt1/detection_loss‚1.1187221609023525 /loss/FBt2/detection_loss‚1.2928109953800837 trainer/global_step‚899 /loss/FRt1/assignment_loss‚3.330739456758008 /loss/FRt1/detection_loss‚0.6216459882362345 /loss/FRt2/detection_loss‚1.7721398822642964 0loss/SRqqt1/detection_loss‚1.355503817462217 _runtime‚1491.621521695àœ”â 1loss/FBt1/assignment_loss‚0.14210865496175526 &loss/total_loss‚14.917363331761859 % _timestamp‚1.7842913074045775e+09 _step‚73 /loss/FRt2/assignment_loss‚1.845679993273446 2loss/SRqqt2/assignment_loss‚1.1698870441866849 1loss/FBt2/assignment_loss‚0.31152170537194857 /loss/FBt1/detection_loss‚1.1187221609023525 /loss/FBt2/detection_loss‚1.2928109953800837 trainer/global_step‚899 /loss/FRt1/assignment_loss‚3.330739456758008 /loss/FRt1/detection_loss‚0.6216459882362345 /loss/FRt2/detection_loss‚1.7721398822642964 0loss/SRqqt1/detection_loss‚1.355503817462217 _runtime‚1491.621521695 2loss/SRqqt1/assignment_loss‚1.1272451813737487 1loss/SRqqt2/detection_loss‚0.8293584515910823 epoch‚2Iç„’Õ•ã‚(:Š ó¿èÒò×Ú÷ gpu.0.gpu‚100.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚340.764' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚68.1528 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0—rXàä‚(:Õ ó¿èÒÛÿ¨ù disk.rbd3.in‚0' disk.nvme0c0n1.out‚7242.06494140625 disk.sda2.in‚0 disk.sda.out‚ 40.828125$ memory_percent‚88.12863826751709) proc.memory.percent‚17.34769344329834 network.recv‚424377$ disk./.usageGB‚556.1745796203613 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 12.015625 disk.rbd1.out‚ 3.7265625 disk.dm-1.out‚ 40.828125 proc.cpu.threads‚17 disk.rbd3.out‚0 disk.rbd1.in‚0# proc.memory.rssMB‚ 2842.24609375 network.sent‚ 1.595234e+06 disk.drbd2165.out‚0) proc.memory.availableMB‚ 1945.00390625* disk./.usagePercent‚37.325342699877574 disk.dm-0.in‚ 12.015625 disk.sda1.out‚0 disk.sda1.in‚0 disk.sda3.out‚ 40.828125 disk.sda2.out‚0 disk.sda.in‚0 disk.rbd0.out‚0" disk.dm-0.out‚7242.06494140625 disk.sda3.in‚0 disk.nvme0n1.in‚ 12.015625% disk.nvme0n1.out‚7242.06494140625 disk.drbd2165.in‚0 disk.dm-1.in‚0 cpu‚25.674199954120482Ž¾eN”å‚(:‰ ‚ÀèÒ˜Ù˜÷ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚288.968' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚57.7936 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0·§±ãæ‚(:Ø ‚ÀèÒ˜Èù disk.sda2.in‚0 disk.rbd1.out‚ 3.7265625 network.recv‚429077* disk./.usagePercent‚37.325375980857025 disk.dm-1.out‚ 41.140625$ disk.nvme0n1.out‚7257.0166015625# proc.memory.rssMB‚ 2842.24609375 disk.sda.in‚0 disk.dm-1.in‚0 disk.rbd0.out‚0 disk.rbd1.in‚0) proc.memory.percent‚17.34769344329834 network.sent‚ 1.657487e+06 disk.sda.out‚ 41.140625 disk.rbd0.in‚0! disk.dm-0.out‚7257.0166015625 disk.sda3.out‚ 41.140625 disk.nvme0n1.in‚ 12.35546875$ memory_percent‚88.12873363494873 disk.drbd2165.in‚0 disk.dm-0.in‚ 12.35546875 disk.rbd3.out‚0! disk.nvme0c0n1.in‚ 12.35546875& disk.nvme0c0n1.out‚7257.0166015625 disk.drbd2165.out‚0 disk.sda2.out‚0 disk.sda1.in‚0 disk.rbd3.in‚0) proc.memory.availableMB‚ 1944.98828125 cpu‚25.671192567852504$ disk./.usageGB‚556.1750755310059 disk.sda1.out‚0 disk.sda3.in‚0 proc.cpu.threads‚17‘Ú-Ÿç‚(:” ‘ÀèÒùæ­ø gpu.0.gpu‚100.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚329.321' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚65.86420000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ÿ•véè‚(:Þ ‘ÀèÒÐòŽù disk.rbd1.out‚ 3.7890625 disk.sda3.in‚0! disk.nvme0c0n1.in‚ 12.65234375 disk.sda2.in‚0 disk.sda.out‚ 41.33203125 disk.dm-1.out‚ 41.33203125# proc.memory.rssMB‚ 2842.24609375) proc.memory.percent‚17.34769344329834 disk.rbd0.out‚0" disk.dm-0.out‚7267.65380859375% disk.nvme0n1.out‚7267.65380859375 disk.rbd1.in‚0 network.recv‚432756( disk./.usagePercent‚37.3258475467351$ disk./.usageGB‚556.1821022033691 disk.sda.in‚0 disk.dm-1.in‚0 disk.sda1.in‚0 disk.rbd3.out‚0 disk.rbd0.in‚0 disk.dm-0.in‚ 12.65234375 disk.sda1.out‚0' disk.nvme0c0n1.out‚7267.65380859375 disk.nvme0n1.in‚ 12.65234375$ memory_percent‚88.12804222106934 network.sent‚ 1.662506e+06 disk.sda3.out‚ 41.33203125 disk.drbd2165.out‚0 proc.cpu.threads‚17 disk.drbd2165.in‚0 disk.rbd3.in‚0 disk.sda2.out‚0( proc.memory.availableMB‚ 1945.1015625 cpu‚25.668589122224585‚0é‚(:’  ÀèÒ¾ºàö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚297.04' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚59.40800000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0uaÆ{Úê‚(:Ï  ÀèÒàŒ‚ù) disk./.usagePercent‚37.32643866813166 disk.sda1.out‚0 disk.rbd3.in‚0 disk.nvme0n1.in‚ 12.9609375 disk.rbd1.in‚0& proc.memory.availableMB‚ 1945.28125 network.sent‚ 1.667486e+06 disk.drbd2165.out‚0$ disk.nvme0n1.out‚7279.5087890625 disk.sda2.in‚0" proc.memory.rssMB‚ 2841.9765625 cpu‚25.66554563866802 network.recv‚435691$ disk./.usageGB‚556.1909103393555! disk.dm-0.out‚7279.5087890625 disk.rbd3.out‚0 disk.nvme0c0n1.in‚ 12.9609375 disk.sda1.in‚0 disk.dm-0.in‚ 12.9609375 disk.sda3.in‚0 disk.sda2.out‚0 disk.rbd1.out‚ 3.7890625 disk.dm-1.in‚0 disk.dm-1.out‚ 41.453125 disk.rbd0.in‚0 disk.sda3.out‚ 41.453125$ memory_percent‚88.12694549560547) proc.memory.percent‚17.34604835510254 disk.sda.in‚0 disk.rbd0.out‚0 proc.cpu.threads‚17 disk.sda.out‚ 41.453125 disk.drbd2165.in‚0& disk.nvme0c0n1.out‚7279.5087890625AN‘”ë‚(:‰ ¯ÀèÒè׉÷ gpu.0.gpu‚95.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚288.439' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚57.6878 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0jÛ_˜Þì‚(:Ó ¯ÀèÒˆô›ù disk.sda3.in‚0 disk.nvme0c0n1.in‚ 12.9609375 disk.sda2.out‚0 disk.sda.in‚0) proc.memory.availableMB‚ 1945.06640625 cpu‚25.663180486191308 network.sent‚ 1.67253e+06& disk.nvme0c0n1.out‚7286.2216796875$ disk.nvme0n1.out‚7286.2216796875 proc.cpu.threads‚17 disk.sda.out‚ 41.703125 disk.dm-1.out‚ 41.703125 disk.rbd0.out‚0 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.sda2.in‚0 disk.dm-0.in‚ 12.9609375 disk.drbd2165.in‚0$ memory_percent‚88.12825679779053 network.recv‚439325$ disk./.usageGB‚556.1977996826172 disk.drbd2165.out‚0 disk.rbd3.in‚0 disk.rbd3.out‚0! disk.dm-0.out‚7286.2216796875 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda3.out‚ 41.703125 disk.nvme0n1.in‚ 12.9609375 disk.rbd1.in‚0 disk.rbd1.out‚ 3.8515625# proc.memory.rssMB‚ 2842.24609375) proc.memory.percent‚17.34769344329834) disk./.usagePercent‚37.32690101773849Ô042í- _wandbruntime‚1568  _runtime‚1568›èi<Åî¿ trainer/global_step‚949 % _timestamp‚1.7842913845079904e+09 _runtime‚1568.725182257 _step‚74 & lr-AdamW/pg1‚0.0006905099992341415 & lr-AdamW/pg2‚0.0006905099992341415ùˆïÉïà _runtime‚1568.725182257 _step‚74 & lr-AdamW/pg1‚0.0006905099992341415 & lr-AdamW/pg2‚0.0006905099992341415 trainer/global_step‚949 % _timestamp‚1.7842913845079904e+09J+¸ÃÒ2ð- _wandbruntime‚1568  _runtime‚1568ëíaÙ'ñÓO G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚272735115570688 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9978 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536960 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 890983026176 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9972 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚8203534 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚272736487133184 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12096 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 890802791741 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2692282 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚272736487133184 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚242 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371562496 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2437890 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚271844132544512 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389056 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚5130414 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚73900063150080 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚272735115570688 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚271844132544512 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 892328415232 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 890983026176 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚16277589 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚8203534 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545123840 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚8213506 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12153 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚247 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚16267611 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9972 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚8064083 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚272615453727807 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚8064083 _step‚75 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389056 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028286 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚8064077 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471115 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600318 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚75048278898688 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚271726020964352 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚271725995536384 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 889458191423 _runtime‚1568.726454293€=Éíò×O E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12153 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚247 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚16267611 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9972 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚8064083 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚272615453727807 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚8064083 _step‚75 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389056 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028286 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚8064077 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471115 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600318 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚75048278898688 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚271726020964352 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚271725995536384 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 889458191423 _runtime‚1568.726454293 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚272735115570688 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9978 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536960 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 890983026176 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9972 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚8203534 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚272736487133184 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12096 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 890802791741 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2692282 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚272736487133184 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚242 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371562496 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2437890 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚271844132544512 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389056 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚5130414 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚73900063150080 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚272735115570688 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚271844132544512 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 892328415232 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 890983026176 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚16277589 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚8203534 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545123840 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚8213506K#›2ó- _wandbruntime‚1570  _runtime‚1570gŸÉ¯â&ôÜM E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471115 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚272122159448064 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 _runtime‚ 1570.11264883 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371569664 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12153 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚8072283 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2440433 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028398 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396224 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚273014041336320 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12153 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2440428 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚75126571459072 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 trainer/global_step‚949 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚16284295 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116224 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 % _timestamp‚1.7842913858952348e+09 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 893253457920 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318922 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚73977917648896 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 891726258099 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚272122159448064 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚16284295 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚5135656 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620928 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚8221998 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚75126974080000 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚8212012 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚273015412905984 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚272122133274624 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚272894258972229 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 890381657669 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 _step‚76 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚8221998 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2694992 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚273015412905984 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 891908061696 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545123840 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1149021901312 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 893253457920 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚272003902742528 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚8212012 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1149056431104 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚8072289 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12096 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545123840 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚236 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚16294287 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚8072289 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5135420 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚272003877314560 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚16294287 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9992 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚16294287 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚8072289 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5135420 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚272003877314560 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚16294287 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9992 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471115 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚272122159448064 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 _runtime‚ 1570.11264883 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371569664 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12153 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚8072283 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2440433 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028398 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396224 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚273014041336320 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12153 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2440428 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚75126571459072 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 trainer/global_step‚949 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚16284295 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116224 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 % _timestamp‚1.7842913858952348e+09 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 893253457920 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318922 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚73977917648896 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 891726258099 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚272122159448064 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚16284295 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚5135656 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620928 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚8221998 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚75126974080000 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚8212012 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚273015412905984 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚272122133274624 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚272894258972229 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 890381657669 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 _step‚76 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚8221998 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2694992 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚273015412905984 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 891908061696 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545123840 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1149021901312 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0Lœº0R2ö- _wandbruntime‚1570  _runtime‚1570ë ƒ÷û /loss/FBt1/detection_loss‚0.7378533138823439 epoch‚2 _runtime‚1570.199456416 2loss/SRqqt2/assignment_loss‚1.0616068433292474 /loss/FBt2/detection_loss‚1.7433685418860665 _step‚77 /loss/FRt1/assignment_loss‚3.220545981620169 0loss/FRt2/assignment_loss‚1.6856800718296634 .loss/FRt1/detection_loss‚0.609904292700895 .loss/FRt2/detection_loss‚1.684598202908365 1loss/SRqqt2/detection_loss‚0.7591607164323184 trainer/global_step‚949 % _timestamp‚1.7842913859825268e+09 1loss/SRqqt1/assignment_loss‚1.140027558667677 1loss/FBt2/assignment_loss‚0.05604408396090062 1loss/SRqqt1/detection_loss‚1.2997646878977291 &loss/total_loss‚14.134537119605373 0loss/FBt1/assignment_loss‚0.1359828244899965Ái)·øÿ _runtime‚1570.199456416 2loss/SRqqt2/assignment_loss‚1.0616068433292474 /loss/FBt2/detection_loss‚1.7433685418860665 _step‚77 /loss/FRt1/assignment_loss‚3.220545981620169 0loss/FRt2/assignment_loss‚1.6856800718296634 .loss/FRt1/detection_loss‚0.609904292700895 .loss/FRt2/detection_loss‚1.684598202908365 1loss/SRqqt2/detection_loss‚0.7591607164323184 trainer/global_step‚949 % _timestamp‚1.7842913859825268e+09 1loss/SRqqt1/assignment_loss‚1.140027558667677 1loss/FBt2/assignment_loss‚0.05604408396090062 1loss/SRqqt1/detection_loss‚1.2997646878977291 &loss/total_loss‚14.134537119605373 0loss/FBt1/assignment_loss‚0.1359828244899965 /loss/FBt1/detection_loss‚0.7378533138823439 epoch‚2MlÚºƒ”ù‚(:‰ ¾ÀèÒ“»ì÷ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚278.726' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.7452 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0n©?cæú‚(:Û ¾ÀèÒô›¤ù disk.rbd1.in‚0 disk.rbd1.out‚ 3.8515625 disk.rbd0.in‚0 disk.rbd3.in‚0) proc.memory.availableMB‚ 1945.01953125 network.sent‚ 1.677511e+06 disk.dm-0.in‚ 13.2578125 disk.sda3.in‚0 disk.rbd0.out‚0 disk.sda1.out‚0# proc.memory.rssMB‚ 2842.24609375 cpu‚25.660395440655652* disk./.usagePercent‚37.327339814652184 disk.sda2.in‚0 disk.dm-1.out‚ 41.89453125 disk.nvme0n1.in‚ 13.2578125 network.recv‚442760 disk.sda1.in‚0 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.sda3.out‚ 41.89453125& disk.nvme0c0n1.out‚7299.2919921875$ memory_percent‚88.12854290008545 disk.sda.in‚0 disk.sda.out‚ 41.89453125 disk.dm-1.in‚0! disk.dm-0.out‚7299.2919921875 disk.nvme0c0n1.in‚ 13.2578125$ disk.nvme0n1.out‚7299.2919921875) proc.memory.percent‚17.34769344329834 proc.cpu.threads‚17 disk.sda2.out‚0 disk.rbd3.out‚0$ disk./.usageGB‚556.2043380737305o¸%iŸû‚(:” ÍÀèÒ•·àö gpu.0.gpu‚90.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚264.778' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚52.955600000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0_ø<–éü‚(:Þ ÍÀèÒ ÅÓø disk.rbd1.in‚0* disk./.usagePercent‚37.327865654127514 disk.sda3.out‚ 44.22265625 network.recv‚447088 disk.nvme0n1.in‚ 13.56640625 disk.sda2.in‚0 disk.dm-1.in‚0 disk.sda1.in‚0 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.sda1.out‚0 disk.sda3.in‚0 disk.rbd1.out‚ 3.9140625 disk.sda.out‚ 44.22265625 disk.dm-0.in‚ 13.56640625# proc.memory.rssMB‚ 2842.24609375$ disk./.usageGB‚556.2121734619141' disk.nvme0c0n1.out‚7319.62158203125 disk.sda.in‚0 disk.dm-1.out‚ 44.22265625 disk.rbd0.in‚0) proc.memory.percent‚17.34769344329834 disk.rbd0.out‚0 proc.cpu.threads‚17 network.sent‚ 1.74023e+06! disk.nvme0c0n1.in‚ 13.56640625% disk.nvme0n1.out‚7319.62158203125 disk.drbd2165.out‚0$ memory_percent‚88.13009262084961 cpu‚25.657995240036005 disk.sda2.out‚0" disk.dm-0.out‚7319.62158203125 disk.drbd2165.in‚0' proc.memory.availableMB‚ 1944.765625Ãükžý‚(:“ ÜÀèÒ«’¥ø gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚295.687' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚59.13740000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0üߢéþ‚(:Þ ÜÀèÒÇ«Úù$ disk./.usageGB‚556.3839912414551 disk.dm-0.in‚ 13.87890625 disk.sda3.out‚ 47.35546875! disk.nvme0c0n1.in‚ 13.87890625 network.recv‚450244 disk.rbd0.out‚0 disk.sda1.in‚0 disk.drbd2165.in‚0 disk.rbd3.in‚0' disk.nvme0c0n1.out‚7366.17724609375 disk.rbd1.in‚0) proc.memory.percent‚17.34769344329834 network.sent‚ 1.745354e+06 disk.rbd0.in‚0 disk.sda1.out‚0 disk.nvme0n1.in‚ 13.87890625 disk.dm-1.in‚0 cpu‚25.65558395698449 proc.cpu.threads‚17) disk./.usagePercent‚37.33939648947723" disk.dm-0.out‚7366.17724609375% disk.nvme0n1.out‚7366.17724609375 disk.rbd1.out‚ 3.9140625 disk.drbd2165.out‚0 disk.rbd3.out‚0 disk.sda2.out‚0# memory_percent‚88.1300687789917) proc.memory.availableMB‚ 1944.76953125 disk.sda.in‚0 disk.dm-1.out‚ 47.35546875# proc.memory.rssMB‚ 2842.24609375 disk.sda3.in‚0 disk.sda2.in‚0 disk.sda.out‚ 47.35546875–˜ @”ÿ‚(:‰ ëÀèÒúóåö gpu.0.gpu‚98.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚268.238' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.6476 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0âŽÉ±X€‚(:× ëÀèÒ¼±Žù) disk./.usagePercent‚37.35011782900368 disk.dm-0.in‚ 1αNóŠ4.19140625! disk.nvme0c0n1.in‚ 14.19140625 network.recv‚453728 disk.sda.out‚ 47.921875" disk.dm-0.out‚7375.06396484375 disk.sda1.out‚0) proc.memory.percent‚17.34769344329834 proc.cpu.threads‚17 disk.sda2.out‚0$ disk./.usageGB‚556.5437469482422 disk.rbd1.out‚ 3.9765625 disk.rbd0.out‚0 disk.drbd2165.in‚0 disk.drbd2165.out‚0# memory_percent‚94.0636157989502 network.sent‚ 1.750355e+06 disk.dm-1.in‚0 disk.dm-1.out‚ 47.921875 disk.sda3.out‚ 47.921875 disk.rbd3.out‚0' proc.memory.availableMB‚ 972.6171875 disk.sda3.in‚0# proc.memory.rssMB‚ 2842.24609375' disk.nvme0c0n1.out‚7375.06396484375 disk.sda2.in‚0 disk.rbd1.in‚0 disk.rbd3.in‚0% disk.nvme0n1.out‚7375.06396484375 disk.sda.in‚0 disk.rbd0.in‚0 disk.sda1.in‚0 disk.nvme0n1.in‚ 14.19140625 cpu‚25.653422247215158xé)¥”‚(:‰ úÀèÒÐÃèö gpu.0.gpu‚93.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚336.329' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚67.2658 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¶gô_ß‚‚(:Ô úÀèÒµ˜Âø disk.sda2.in‚0 disk.sda2.out‚0 disk.drbd2165.out‚0 network.recv‚456996 disk.sda.out‚ 48.890625! disk.dm-0.out‚7607.5869140625 disk.nvme0n1.in‚ 14.19140625 disk.dm-1.out‚ 48.890625 disk.sda1.in‚0 disk.sda3.in‚0 proc.cpu.threads‚17) disk./.usagePercent‚37.37451406697911 disk.sda.in‚0 disk.dm-1.in‚0 disk.rbd0.out‚0 disk.drbd2165.in‚0! disk.nvme0c0n1.in‚ 14.19140625# proc.memory.rssMB‚ 2842.24609375 disk.rbd1.in‚0 disk.rbd0.in‚0 disk.sda1.out‚0 disk.sda3.out‚ 48.890625 disk.rbd3.out‚0$ disk.nvme0n1.out‚7608.3369140625 cpu‚25.65114920866868$ disk./.usageGB‚556.9072685241699 disk.rbd1.out‚ 3.9765625 disk.dm-0.in‚ 14.19140625 disk.rbd3.in‚0& disk.nvme0c0n1.out‚7608.3369140625( proc.memory.availableMB‚ 972.82421875) proc.memory.percent‚17.34769344329834 network.sent‚ 1.75557e+06$ memory_percent‚94.06235218048096M-€b2ƒ- _wandbruntime‚1647  _runtime‚1647Ðú Å„¿ & lr-AdamW/pg1‚0.0006902552948344889 & lr-AdamW/pg2‚0.0006902552948344889 trainer/global_step‚999 % _timestamp‚1.7842914632933815e+09 _runtime‚1647.510527759 _step‚78šJy¦É…à trainer/global_step‚999 % _timestamp‚1.7842914632933815e+09 _runtime‚1647.510527759 _step‚78 & lr-AdamW/pg1‚0.0006902552948344889 & lr-AdamW/pg2‚0.0006902552948344889NÑ´+½2†- _wandbruntime‚1647  _runtime‚1647ëà}zØ'‡ÒO F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1207998608384 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚8474377 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚8639258 trainer/global_step‚999 _step‚79 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚286682839569408 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 937298956800 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚78982750233600 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402628096 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚286681468006912 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚285744169050112 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116224 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 _runtime‚ 1647.51179978 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389056 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371562496 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 935695304531 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9972 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚286681468006912 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116224 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚285744195223552 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9978 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545123840 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚5395219 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028286 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12153 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚286555779749715 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 % _timestamp‚1.7842914632944608e+09 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚285744195223552 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12096 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2829934 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 938644345856 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9978 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚250 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚245 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 937039904849 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 937298956800 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9972 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚8474383 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚8629286 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚286557149778001 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚8629286 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536960 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚8474377 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2565040 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚8474383 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504ÙòdÜ'ˆÖO E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12153 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2830179 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 938644345856 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚17113641 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚78982347605504 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚77774383534080 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚285620109873152 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600318 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚8639258 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚286682839569408 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389056 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2565035 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚286681468006912 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚285744169050112 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116224 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 _runtime‚ 1647.51179978 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389056 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371562496 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 935695304531 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9972 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚286681468006912 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116224 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚285744195223552 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9978 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545123840 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚5395219 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028286 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12153 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚286555779749715 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 % _timestamp‚1.7842914632944608e+09 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚285744195223552 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12096 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2829934 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 938644345856 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9978 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚250 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚245 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 937039904849 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 937298956800 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9972 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚8474383 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚8629286 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚286557149778001 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚8629286 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536960 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚8474377 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2565040 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚8474383 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1207998608384 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚8474377 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚8639258 trainer/global_step‚999 _step‚79 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚286682839569408 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 937298956800 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚78982750233600 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402628096 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318922 % _timestamp‚1.7842914646804216e+09 _runtime‚1648.897818666 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚8482583 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚79061037035008 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚286835955022535 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚286022169780224 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚17120347 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚285897991651328 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396224 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 938223992320 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2567578 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1209201623040 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9986 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396224 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚286961765342208 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12096 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚8647750 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371569664 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚239 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 936618770777 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471115 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚8637764 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚286960393772544 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5400256 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚17120347 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545123840 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529792 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600430 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚77851869941760 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 trainer/global_step‚999 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚17130339 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚286022169780224 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2832678 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620928 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚17130339 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚8482583 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚286022195953664 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requeste¾M³Fd_bytes.large_pool.current‚25427968 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12153 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚79061439655936 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2858979662233608ëç&‹áM N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471115 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚8637764 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚286960393772544 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5400256 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚17120347 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545123840 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529792 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600430 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚77851869941760 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 trainer/global_step‚999 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚17130339 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚286022169780224 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2832678 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620928 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚17130339 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚8482583 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚286022195953664 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12153 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚79061439655936 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚285897966223360 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚8647750 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 939569388544 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2567583 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116224 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚5400500 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚8637764 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318922 % _timestamp‚1.7842914646804216e+09 _runtime‚1648.897818666 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚8482583 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚79061037035008 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚286835955022535 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚286022169780224 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚17120347 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚285897991651328 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396224 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 938223992320 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2567578 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1209201623040 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9986 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396224 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚286961765342208 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12096 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚8647750 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371569664 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚239 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 936618770777Pì-52Œ- _wandbruntime‚1648  _runtime‚1648ªë—ý 1loss/FBt2/assignment_loss‚0.40520185145609966 trainer/global_step‚999 % _timestamp‚1.7842914647675343e+09 /loss/FRt2/detection_loss‚1.7700290129071585 /loss/FBt2/detection_loss‚1.3232677575686704 epoch‚2 1loss/SRqqt1/assignment_loss‚1.040146292271939 1loss/SRqqt1/detection_loss‚1.2397436924375498 _runtime‚1648.984464401 _step‚81 2loss/SRqqt2/assignment_loss‚1.1534503886078242 -loss/FRt1/detection_loss‚0.58465258781053 1loss/SRqqt2/detection_loss‚0.8427079936750672 /loss/FBt1/detection_loss‚0.7845913944037137 &loss/total_loss‚14.367854068086809 0loss/FRt1/assignment_loss‚3.2887599754802737 0loss/FRt2/assignment_loss‚1.7832549730384792 1loss/FBt1/assignment_loss‚0.15204814842950312=>ŠüŽ 2loss/SRqqt2/assignment_loss‚1.1534503886078242 -loss/FRt1/detection_loss‚0.58465258781053 1loss/SRqqt2/detection_loss‚0.8427079936750672 /loss/FBt1/detection_loss‚0.7845913944037137 &loss/total_loss‚14.367854068086809 0loss/FRt1/assignment_loss‚3.2887599754802737 0loss/FRt2/assignment_loss‚1.7832549730384792 1loss/FBt1/assignment_loss‚0.15204814842950312 1loss/FBt2/assignment_loss‚0.40520185145609966 trainer/global_step‚999 % _timestamp‚1.7842914647675343e+09 /loss/FRt2/detection_loss‚1.7700290129071585 /loss/FBt2/detection_loss‚1.3232677575686704 epoch‚2 1loss/SRqqt1/assignment_loss‚1.040146292271939 1loss/SRqqt1/detection_loss‚1.2397436924375498 _runtime‚1648.984464401 _step‚81Qm`ž‚(:“ ‰ÁèÒ’Ðô÷ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚289.405' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚57.88099999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0:.ï˜Þ‚(:Ó ‰ÁèÒ€Ö¡ù disk.rbd3.out‚0 disk.nvme0n1.in‚ 14.48828125 disk.dm-0.out‚ 7727.33203125 disk.sda1.in‚0 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda3.out‚ 49.3515625$ disk./.usageGB‚558.9945182800293 disk.rbd3.in‚0 disk.sda2.out‚0 disk.rbd1.out‚ 4.0390625 disk.sda.in‚0 network.sent‚ 1.760586e+06 network.recv‚460721 disk.drbd2165.out‚0! disk.nvme0c0n1.in‚ 14.48828125$ disk.nvme0c0n1.out‚ 7727.33203125 disk.dm-1.in‚0 disk.dm-1.out‚ 49.3515625 disk.drbd2165.in‚0" disk.nvme0n1.out‚ 7727.33203125 disk.rbd1.in‚0( proc.memory.availableMB‚ 973.28515625* disk./.usagePercent‚37.514591149414024 disk.dm-0.in‚ 14.48828125 disk.sda1.out‚0 cpu‚25.64857503362146 disk.sda.out‚ 49.3515625# proc.memory.rssMB‚ 2842.24609375) proc.memory.percent‚17.34769344329834 disk.sda2.in‚0 disk.sda3.in‚0$ memory_percent‚94.05953884124756U÷³Î”‘‚(:‰ ˜ÁèÒÓ†Ô÷ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚37.0 gpu.0.powerWatts‚109.884' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.9768 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0NµŽ×’‚(:Ì ˜ÁèÒÙÀø# proc.memory.rssMB‚ 2842.24609375 cpu‚25.64624236807845 disk.sda2.out‚0 disk.sda.in‚0 disk.rbd0.in‚0 disk.sda3.out‚ 51.296875 disk.drbd2165.out‚0 disk.rbd1.in‚0 disk.sda1.out‚0! disk.nvme0n1.out‚ 9725.6171875( proc.memory.availableMB‚ 973.46484375 network.sent‚ 1.823413e+06$ disk./.usageGB‚558.8472785949707 disk.dm-1.in‚0 disk.dm-0.out‚ 9725.6171875 disk.rbd3.in‚0 disk.sda1.in‚0 disk.rbd3.out‚0! disk.nvme0c0n1.in‚ 14.80078125# disk.nvme0c0n1.out‚ 9725.6171875 disk.dm-0.in‚ 14.80078125 disk.sda3.in‚0) proc.memory.percent‚17.34769344329834) disk./.usagePercent‚37.50470977060742 disk.sda2.in‚0 disk.dm-1.out‚ 51.296875 disk.rbd0.out‚0$ memory_percent‚94.05844211578369 network.recv‚465051 disk.nvme0n1.in‚ 14.80078125 proc.cpu.threads‚17 disk.rbd1.out‚ 4.0390625 disk.sda.out‚ 51.296875 disk.drbd2165.in‚0I`”“‚(:‰ §ÁèÒò·›÷ gpu.0.gpu‚99.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚290.031' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚58.0062 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚01Lµ×甂(:Ü §ÁèÒªóÓø disk.sda.in‚0 disk.sda.out‚ 53.71484375$ memory_percent‚94.06001567840576( proc.memory.availableMB‚ 973.20703125 proc.cpu.threads‚17 disk.drbd2165.out‚0 disk.rbd0.in‚0) proc.memory.percent‚17.34769344329834 disk.rbd3.in‚0 disk.sda2.in‚0 disk.dm-1.out‚ 53.71484375 disk.dm-0.in‚ 15.109375 network.recv‚468354* disk./.usagePercent‚37.523102887912444$ disk./.usageGB‚559.1213493347168 disk.drbd2165.in‚0 disk.sda1.out‚0 disk.nvme0c0n1.in‚ 15.109375& disk.nvme0n1.out‚10245.85302734375 disk.rbd1.in‚0# proc.memory.rssMB‚ 2842.24609375 disk.rbd3.out‚0( disk.nvme0c0n1.out‚10245.85302734375 disk.rbd0.out‚0 disk.sda1.in‚0 disk.sda3.in‚0 disk.sda3.out‚ 53.71484375 disk.sda2.out‚0 disk.rbd1.out‚ 4.1640625 disk.dm-1.in‚0# disk.dm-0.out‚10245.85302734375 cpu‚25.64392226907135 network.sent‚ 1.828434e+06 disk.nvme0n1.in‚ 15.109375y/¼”•‚(:‰ ¶ÁèÒùÿ™÷ gpu.0.gpu‚65.0 gpu.0.memory‚20, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚331.638' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚66.3276 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0F-¿©é–‚(:Þ ¶ÁèÒ“ºø& proc.memory.availableMB‚ 973.453125 network.sent‚ 1.833333e+06 disk.rbd3.out‚0 disk.rbd1.out‚ 4.1640625 disk.rbd0.in‚0 proc.cpu.threads‚17* disk./.usagePercent‚37.523080615256966$ disk./.usageGB‚559.1210174560547 disk.drbd2165.in‚0( disk.nvme0c0n1.out‚10476.02880859375 disk.sda2.in‚0 disk.dm-1.out‚ 54.4921875# proc.memory.rssMB‚ 2842.24609375 disk.drbd2165.out‚0 disk.nvme0n1.in‚ 15.41796875 disk.dm-0.in‚ 15.41796875 disk.sda1.in‚0) proc.memory.percent‚17.34769344329834& disk.nvme0n1.out‚10476.02880859375 disk.rbd3.in‚0 disk.sda1.out‚0$ memory_percent‚94.05851364135742 cpu‚25.641799096683076 network.recv‚471675 disk.sda.in‚0# disk.dm-0.out‚10476.02880859375! disk.nvme0c0n1.in‚ 15.41796875 disk.rbd1.in‚0 disk.sda2.out‚0 disk.sda.out‚ 54.4921875 disk.dm-1.in‚0 disk.sda3.in‚0 disk.sda3.out‚ 54.4921875 disk.rbd0.out‚0Î2¹Å”—‚(:‰ ÅÁèÒâ›÷ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚283.649' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚56.7298 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0?ÌEÜꘂ(:ß ÅÁèÒƒ¢Èø network.recv‚475397 disk.sda.in‚0 disk.rbd0.out‚0$ memory_percent‚94.05853748321533( proc.memory.availableMB‚ 973.44921875 disk.sda2.out‚0 disk.rbd1.in‚0 disk.sda3.in‚0 disk.rbd3.out‚0 disk.rbd0.in‚0# disk.dm-0.out‚10494.51806640625! disk.nvme0c0n1.in‚ 15.41796875$ disk./.usageGB‚559.1242446899414 disk.dm-1.in‚0& disk.nvme0n1.out‚10494.51806640625 disk.dm-1.out‚ 55.3203125 disk.dm-0.in‚ 15.41796875 disk.sda3.out‚ 55.3203125 disk.rbd1.out‚ 4.2265625 cpu‚25.639270408138223 proc.cpu.threads‚17) disk./.usagePercent‚37.52329719763093 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.rbd3.in‚0( disk.nvme0c0n1.out‚10494.51806640625# proc.memory.rssMB‚ 2842.24609375 disk.sda.out‚ 55.3203125 disk.drbd2165.out‚0 disk.sda2.in‚0 network.sent‚ 1.838514e+06 disk.nvme0n1.in‚ 15.41796875) proc.memory.percent‚17.34769344329834Ô-¿;•™‚(:Š ÔÁèÒ–Ôíö gpu.0.gpu‚100.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚46.0 gpu.0.powerWatts‚334.682' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚66.9364 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ô§‚Táš‚(:Ö ÔÁèÒ£¿Àø! disk.nvme0c0n1.in‚ 15.71484375( disk.nvme0c0n1.out‚10510.53857421875 disk.rbd1.out‚ 4.2265625 cpu‚25.637219686715717 disk.rbd3.in‚0 disk.dm-1.out‚55.4375 disk.dm-0.in‚ 15.71484375 proc.cpu.threads‚17 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda.in‚0 disk.sda.out‚55.4375 disk.sda3.in‚0 disk.dm-1.in‚0# disk.dm-0.out‚10510.53857421875 disk.sda1.in‚0 disk.sda1.out‚0# proc.memory.rssMB‚ 2842.24609375 disk.nvme0n1.in‚ 15.71484375 disk.rbd1.in‚0$ memory_percent‚94.06008720397949$ disk./.usageGB‚559.1273155212402 disk.drbd2165.in‚0 disk.rbd3.out‚0' proc.memory.availableMB‚ 973.1953125) proc.memory.percent‚17.34769344329834 disk.sda3.out‚55.4375 network.sent‚ 1.843536e+06 network.recv‚478599 disk.drbd2165.out‚0& disk.nvme0n1.out‚10510.53857421875 disk.rbd0.in‚0 disk.rbd0.out‚0* disk./.usagePercent‚37.523503283695995I;5ÿ2›- _wandbruntime‚1726  _runtime‚1726‡’óÆœÀ % _timestamp‚1.7842915420464451e+09 _runtime‚1726.264971864 _step‚82 & lr-AdamW/pg1‚0.0006899784746937165 & lr-AdamW/pg2‚0.0006899784746937165 trainer/global_step‚1049v䀌ÊÄ % _timestamp‚1.7842915420464451e+09 _runtime‚1726.264971864 _step‚82 & lr-AdamW/pg1‚0.0006899784746937165 & lr-AdamW/pg2‚0.0006899784746937165 trainer/global_step‚1049RÕ¢ž 2ž- _wandbruntime‚1726  _runtime‚1726ΗÚÙ'ŸÓO JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9972 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚8884677 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371562496 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12153 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 trainer/global_step‚1049 $ _timestamp‚1.784291542047221e+09 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚8884683 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9972 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9978 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12096 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116224 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚248 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028286 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545123840 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚300627820443136 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2692185 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 983614887424 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9978 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2967612 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚81649072009216 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1268111321088 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚299644205555712 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 981932417639 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚299644231729152 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚300497475799909 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402628096 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536960 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚300496105771623 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471115 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371562496 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚300627820443136 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12096 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚82916780702208 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1268076784128 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389056 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚9065010 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚299644231729152 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12153 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚299514198781952 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600318 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116224 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2692190 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚300629192005632 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚9065010 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚9055038 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 984960276480 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚17949693 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚5659802 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚82917183330304 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _runtime‚1726.265053858 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚81648703918080 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚300629192005632 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚243 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545123840 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12096 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116224 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚248 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028286 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545123840 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚300627820443136 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2692185 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 983614887424 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9978 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2967612 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚81649072009216 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1268111321088 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚299644205555712 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 981932417639 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚299644231729152 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚300497475799909 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402628096 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536960 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚300496105771623 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471115 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371562496 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚300627820443136 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12096 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚82916780702208 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1268076784128 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389056 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚9065010 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚299644231729152 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12153 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚299514198781952 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600318 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116224 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2692190 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175S&gÂû2¡- _wandbruntime‚1727  _runtime‚1727m”‘uã&¢ÝM [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 984200484315 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 984539922944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116224 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚17966391 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚237 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116224 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚300906746208768 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚17956399 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚242 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 985885319168 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1269315039232 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚300908117778432 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371569664 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5664828 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1269280509440 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12153 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471115 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318922 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371569664 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9986 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚81726558416896 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396224 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _step‚84 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚299922232459264 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396224 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2694728 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚300774911016045 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545123840 $ _timestamp‚1.784291543431934e+09 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚17966391 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚8892889 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚9063516 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚300776281044443 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600430 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 985885319168 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 982855883885 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12153 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚9073502 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2694733 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529792 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚9063516 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚5665070 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚81726190325760 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620928 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚299792055132160 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚8892889 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚82995873456128 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028398 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚299792080560128 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 trainer/global_step‚1049 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚17956399 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9986 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚299922206285824\;a.ç&£áM ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 985885319168 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1269315039232 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚300908117778432 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371569664 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5664828 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1269280509440 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12153 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471115 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318922 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371569664 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9986 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚81726558416896 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396224 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _step‚84 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚299922232459264 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396224 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2694728 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚300774911016045 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545123840 $ _timestamp‚1.784291543431934e+09 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚17966391 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚8892889 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚9063516 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚300776281044443 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600430 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 985885319168 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 982855883885 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12153 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚9073502 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2694733 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529792 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚9063516 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚5665070 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚81726190325760 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620928 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚299792055132160 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚8892889 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚82995873456128 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028398 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚299792080560128 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 trainer/global_step‚1049 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚17956399 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9986 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚299922206285824 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 984200484315 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 984539922944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116224 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚17966391 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚237 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116224 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚300906746208768 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚17956399 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚242Tºþi$2¤- _wandbruntime‚1727  _runtime‚1727ÏIã÷¥ÿ /loss/FRt1/detection_loss‚0.6202165783959219 1loss/SRqqt2/detection_loss‚0.7865370607868453 /loss/FBt1/detection_loss‚0.5067623453477895 trainer/global_step‚1049 0loss/FRt1/assignment_loss‚3.1405722458651852 0loss/FRt2/assignment_loss‚1.8430041536396624 1loss/SRqqt1/detection_loss‚1.3019978845735642 .loss/FBt2/detection_loss‚0.985018287820782 _runtime‚1727.735633552 _step‚85 1loss/FBt1/assignment_loss‚0.09669737191521893 .loss/FRt2/detection_loss‚1.745104585628153 % _timestamp‚1.7842915435186982e+09 epoch‚2 &loss/total_loss‚13.543350679857086 2loss/SRqqt1/assignment_loss‚1.1597440802032435 2loss/SRqqt2/assignment_loss‚1.1199033415418245 1loss/FBt2/assignment_loss‚0.23779274413889495êÔft ¦ƒ epoch‚2 &loss/total_loss‚13.543350679857086 2loss/SRqqt1/assignment_loss‚1.1597440802032435 2loss/SRqqt2/assignment_loss‚1.1199033415418245 1loss/FBt2/assignment_loss‚0.23779274413889495 /loss/FRt1/detection_loss‚0.6202165783959219 1loss/SRqqt2/detection_loss‚0.7865370607868453 /loss/FBt1/detection_loss‚0.5067623453477895 trainer/global_step‚1049 0loss/FRt1/assignment_loss‚3.1405722458651852 0loss/FRt2/assignment_loss‚1.8430041536396624 1loss/SRqqt1/detection_loss‚1.3019978845735642 .loss/FBt2/detection_loss‚0.985018287820782 _runtime‚1727.735633552 _step‚85 1loss/FBt1/assignment_loss‚0.09669737191521893 .loss/FRt2/detection_loss‚1.745104585628153 % _timestamp‚1.7842915435186982e+09U™DwŸ§‚(:” ãÁèÒòÍŽ÷ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚280.116' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚56.023199999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ݾý'ᨂ(:Ö ãÁèÒåÉø& disk.nvme0n1.out‚10520.52392578125 disk.rbd1.in‚0 disk.rbd1.out‚ 4.2890625( proc.memory.availableMB‚ 973.16796875# proc.memory.rssMB‚ 2842.24609375 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.sda1.in‚0 disk.rbd3.in‚0 disk.rbd3.out‚0( disk.nvme0c0n1.out‚10520.52392578125 disk.sda.out‚55.71875* disk./.usagePercent‚37.524002498387766 disk.dm-0.in‚ 16.0234375 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.sda.in‚0) proc.memory.percent‚17.34769344329834 network.sent‚ 1.848559e+06$ disk./.usageGB‚559.1347541809082 disk.sda3.out‚55.71875 disk.sda2.out‚0 disk.drbd2165.out‚0 disk.nvme0c0n1.in‚ 16.0234375 disk.sda2.in‚0$ memory_percent‚94.06025409698486 cpu‚25.63477391741633 disk.dm-1.in‚0 disk.rbd0.out‚0 disk.nvme0n1.in‚ 16.0234375 network.recv‚482153 disk.dm-1.out‚55.71875# disk.dm-0.out‚10520.52392578125 disk.sda3.in‚0ÄË—”©‚(:‰ òÁèÒÜ…÷ gpu.0.gpu‚99.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚46.0 gpu.0.powerWatts‚343.993' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚68.7986 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0èáÀª‚(:È òÁèҧуù) disk./.usagePercent‚37.52433863628023 disk.drbd2165.in‚0 disk.rbd3.in‚0 disk.rbd3.out‚0# proc.memory.rssMB‚ 2842.24609375 proc.cpu.threads‚17# disk./.usageGB‚559.139762878418! disk.dm-0.out‚10532.029296875 disk.sda3.out‚55.875 disk.rbd1.in‚0 disk.sda.out‚55.875 disk.sda1.out‚0 disk.drbd2165.out‚0 disk.rbd1.out‚ 4.2890625' proc.memory.availableMB‚ 973.1640625 cpu‚25.632920264414423 network.sent‚ 1.911484e+06 disk.sda.in‚0 disk.ÄŽû‘¶dm-1.out‚55.875 disk.rbd0.in‚0 disk.nvme0n1.in‚ 16.3203125 disk.sda2.out‚0$ memory_percent‚94.06027793884277) proc.memory.percent‚17.34769344329834 network.recv‚486217 disk.rbd0.out‚0 disk.dm-0.in‚ 16.3203125 disk.sda3.in‚0& disk.nvme0c0n1.out‚10532.029296875 disk.dm-1.in‚0 disk.sda1.in‚0 disk.nvme0c0n1.in‚ 16.3203125$ disk.nvme0n1.out‚10532.029296875 disk.sda2.in‚0¸`„j”«‚(:‰ ÂèÒÙ÷ôö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚277.374' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.4748 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0-›?û⬂(:× ÂèÒÿ—ù disk.nvme0n1.in‚ 16.62890625 disk.sda2.out‚0 disk.sda.in‚0& disk.nvme0n1.out‚10815.20361328125$ memory_percent‚94.06201839447021 disk.sda3.in‚0 disk.rbd1.in‚0 disk.rbd0.in‚0 disk.drbd2165.out‚0 disk.rbd3.out‚0( disk.nvme0c0n1.out‚10815.20361328125 disk.sda2.in‚0) proc.memory.percent‚17.34769344329834 network.sent‚ 1.916514e+06) disk./.usagePercent‚37.54084164995996# disk./.usageGB‚559.385669708252# proc.memory.rssMB‚ 2842.24609375 proc.cpu.threads‚17 network.recv‚489912 disk.dm-0.in‚ 16.62890625 disk.sda.out‚56.28125 disk.dm-1.in‚0 disk.dm-1.out‚56.28125# disk.dm-0.out‚10815.20361328125 disk.drbd2165.in‚0( proc.memory.availableMB‚ 972.87890625 cpu‚25.63069367234607 disk.rbd1.out‚ 4.3515625 disk.sda1.in‚0 disk.sda3.out‚56.28125 disk.rbd3.in‚0! disk.nvme0c0n1.in‚ 16.62890625 disk.rbd0.out‚0 disk.sda1.out‚0×ILx”­‚(:‰ ÂèÒíËÞö gpu.0.gpu‚98.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚333.913' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚66.7826 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚02õÔO⮂(:× ÂèÒ›æþ÷# proc.memory.rssMB‚ 2842.24609375) proc.memory.percent‚17.34769344329834) disk./.usagePercent‚37.54128787109214 disk.dm-0.in‚ 16.92578125 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda.out‚ 56.359375 disk.sda3.out‚ 56.359375 disk.rbd3.in‚0% disk.nvme0n1.out‚10838.5087890625 disk.sda.in‚0$ disk./.usageGB‚559.3923187255859 disk.sda1.in‚0 disk.nvme0n1.in‚ 16.92578125 network.sent‚ 1.921518e+06' disk.nvme0c0n1.out‚10838.5087890625# memory_percent‚94.0605640411377 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda3.in‚0 disk.drbd2165.in‚0! disk.nvme0c0n1.in‚ 16.92578125' proc.memory.availableMB‚ 973.1171875 cpu‚25.628948553121198 proc.cpu.threads‚17 disk.dm-1.in‚0" disk.dm-0.out‚10838.5087890625 disk.rbd3.out‚0 disk.rbd1.in‚0 network.recv‚493109 disk.dm-1.out‚ 56.359375 disk.drbd2165.out‚0 disk.sda2.out‚0 disk.rbd1.out‚ 4.3515625PSã힯‚(:“ ŸÂèÒü”ëö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚279.79' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚55.958000000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0nã.>ë°‚(:à ŸÂèÒǺÁø) disk./.usagePercent‚37.54149267711953 disk.drbd2165.out‚0 disk.rbd1.in‚0# proc.memory.rssMB‚ 2842.24609375 disk.sda1.in‚0! disk.nvme0c0n1.in‚ 16.92578125& disk.nvme0n1.out‚10850.21435546875 disk.sda2.out‚0 network.sent‚ 1.926492e+06 disk.sda3.in‚0 disk.rbd1.out‚ 4.4140625 disk.dm-1.out‚ 59.03515625 disk.rbd0.out‚0 network.recv‚496423 disk.rbd3.out‚0( disk.nvme0c0n1.out‚10850.21435546875 disk.sda2.in‚0 cpu‚25.626775344686095 proc.cpu.threads‚17 disk.drbd2165.in‚0 disk.rbd3.in‚0 disk.sda.out‚ 59.03515625 disk.dm-1.in‚0 disk.rbd0.in‚0" memory_percent‚94.059157371521( proc.memory.availableMB‚ 973.34765625 disk.sda.in‚0 disk.dm-0.in‚ 16.92578125) proc.memory.percent‚17.34769344329834$ disk./.usageGB‚559.3953704833984 disk.sda3.out‚ 59.03515625 disk.sda1.out‚0 disk.nvme0n1.in‚ 16.92578125# disk.dm-0.out‚10850.21435546875©›éT2±- _wandbruntime‚1804  _runtime‚1804¥|i?ƲÀ % _timestamp‚1.7842916207007618e+09 _runtime‚1804.917922118 _step‚86 & lr-AdamW/pg1‚0.0006896795565914645 & lr-AdamW/pg2‚0.0006896795565914645 trainer/global_step‚1099Œl»YÊ³Ä & lr-AdamW/pg2‚0.0006896795565914645 trainer/global_step‚1099 % _timestamp‚1.7842916207007618e+09 _runtime‚1804.917922118 _step‚86 & lr-AdamW/pg1‚0.0006896795565914645V5B‹2´- _wandbruntime‚1804  _runtime‚1804¤kî,ß'µÙO E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12153 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚85523024302080 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12096 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚314574172879360 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚9480790 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12096 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚9490762 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1029930818048 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚9294983 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116224 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9978 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚314437801821817 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028286 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚314436431793531 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318922 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1328236324864 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚9294977 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚9490762 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371562496 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚314575544441856 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2819340 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 $ _timestamp‚1.784291620701851e+09 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3105041 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536960 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9972 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3104785 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545123840 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1031276207104 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚9294983 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚313408287690752 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚256 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚86851628718080 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2819335 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚313544268234752 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚313544242061312 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9978 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9972 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402628096 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚85523392393216 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚313544268234752 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚9294977 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1328201787904 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1031276207104 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389056 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚87 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1029930818048 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚18785745 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚18775767 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545123840 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚86851226089984 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚313408262262784 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 trainer/global_step‚1099] bã'¶ÝO HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 trainer/global_step‚1099 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12153 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚85523024302080 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12096 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚314574172879360 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚9480790 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12096 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚9490762 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1029930818048 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚9294983 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116224 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9978 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚314437801821817 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028286 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚314436431793531 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318922 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1328236324864 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚9294977 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚9490762 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371562496 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚314575544441856 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2819340 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 $ _timestamp‚1.784291620701851e+09 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3105041 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536960 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9972 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3104785 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545123840 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1031276207104 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚9294983 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚313408287690752 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚256 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚86851628718080 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2819335 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚313544268234752 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚313544242061312 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9978 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9972 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402628096 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚85523392393216 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚313544268234752 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚9294977 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1328201787904 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1031276207104 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389056 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚87 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1029930818048 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚18785745 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚18775767 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545123840 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚86851226089984 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚313408262262784Wábå2·- _wandbruntime‚1806  _runtime‚1806à»S„¸ãM MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚9303183 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545123840 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚251 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404718080 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12153 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396224 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12153 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚313686144040960 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9986 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371569664 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028398 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1030437597423 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396224 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚9499254 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5929402 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚314715237037953 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚18792451 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9986 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600430 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚314854470214656 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚313822242791424 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2821883 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚85600510709760 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _runtime‚1806.313410541 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚314853098644992 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1032201249792 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471115 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371569664 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545123840 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1030855853568 $ _timestamp‚1.784291622096004e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116224 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚9303189 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚313822242791424 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚9489268 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2821878 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9992 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚18802443 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 _step‚88…ØÚ1í&¹çM TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396224 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12153 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚313686144040960 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9986 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371569664 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028398 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1030437597423 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396224 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚9499254 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5929402 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚314715237037953 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚18792451 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9986 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600430 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚314854470214656 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚313822242791424 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2821883 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚85600510709760 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _runtime‚1806.313410541 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚314853098644992 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1032201249792 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471115 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371569664 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545123840 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1030855853568 $ _timestamp‚1.784291622096004e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116224 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚9303189 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚313822242791424 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚9489268 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2821878 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9992 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚18802443 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 _step‚88 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚9303183 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545123840 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚251 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404718080 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12153 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0Xc–W›2º- _wandbruntime‚1806  _runtime‚1806KÜ=Õ»ü /loss/FBt1/detection_loss‚0.5873351939087121 epoch‚2 trainer/global_step‚1099 % _timestamp‚1.7842916221795614e+09 _runtime‚1806.396661286 2loss/SRqqt2/assignment_loss‚0.9835245371761526 1loss/FBt1/assignment_loss‚0.05609801265445535 1loss/SRqqt1/detection_loss‚1.1374517742325279 .loss/FBt2/detection_loss‚1.356007604939597 .loss/FRt1/assignment_loss‚3.04167891535057 0loss/FRt2/assignment_loss‚1.6619978358650653 1loss/SRqqt1/assignment_loss‚1.138917755583922 1loss/FBt2/assignment_loss‚0.12290938273359708 /loss/FRt1/detection_loss‚0.6290796726464878 .loss/FRt2/detection_loss‚1.713149939997414 1loss/SRqqt2/detection_loss‚0.7531478860676658 &loss/total_loss‚13.181298511156168 _step‚89‡ä ¼€ .loss/FRt1/assignment_loss‚3.04167891535057 0loss/FRt2/assignment_loss‚1.6619978358650653 1loss/SRqqt1/assignment_loss‚1.138917755583922 1loss/FBt2/assignment_loss‚0.12290938273359708 /loss/FRt1/detection_loss‚0.6290796726464878 .loss/FRt2/detection_loss‚1.713149939997414 1loss/SRqqt2/detection_loss‚0.7531478860676658 &loss/total_loss‚13.181298511156168 _step‚89 /loss/FBt1/detection_loss‚0.5873351939087121 epoch‚2 trainer/global_step‚1099 % _timestamp‚1.7842916221795614e+09 _runtime‚1806.396661286 2loss/SRqqt2/assignment_loss‚0.9835245371761526 1loss/FBt1/assignment_loss‚0.05609801265445535 1loss/SRqqt1/detection_loss‚1.1374517742325279 .loss/FBt2/detection_loss‚1.356007604939597YžàF?ž½‚(:“ ®ÂèÒ­”Ò÷ gpu.0.gpu‚52.0 gpu.0.memory‚17, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚328.402' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚65.68039999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¸Çb⾂(:× ®ÂèÒÕ÷°ù disk.nvme0n1.in‚ 17.234375 network.recv‚499601# disk.dm-0.out‚10886.90478515625$ memory_percent‚94.06099319458008) proc.memory.percent‚17.34769344329834 disk.sda3.in‚0 disk.sda2.out‚0 disk.sda.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 cpu‚25.625035714051347 network.sent‚ 1.931609e+06 disk.sda3.out‚ 61.2265625 disk.drbd2165.in‚0 disk.rbd3.in‚0 disk.sda.out‚ 61.2265625 disk.rbd3.out‚0 disk.nvme0c0n1.in‚ 17.234375 disk.rbd1.out‚ 4.4140625 disk.dm-1.out‚ 61.2265625 disk.sda1.out‚0& proc.memory.availableMB‚ 973.046875& disk.nvme0n1.out‚10886.90478515625 disk.sda2.in‚0 disk.dm-1.in‚0 proc.cpu.threads‚17) disk./.usagePercent‚37.54200827629349 disk.rbd1.in‚0$ disk./.usageGB‚559.4030532836914 disk.drbd2165.out‚0( disk.nvme0c0n1.out‚10886.90478515625 disk.dm-0.in‚ 17.234375# proc.memory.rssMB‚ 2842.24609375 disk.sda1.in‚0‰POrŸ¿‚(:” ½ÂèÒ×ÄÝö gpu.0.gpu‚96.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚273.071' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚54.614200000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0G‘Ð’ØÀ‚(:Í ½ÂèÒúþø network.recv‚504168 disk.nvme0c0n1.in‚17.53125 disk.sda2.out‚0 disk.sda3.out‚ 61.7890625 cpu‚25.623090582971745 network.sent‚ 1.994612e+06 disk.rbd3.in‚0% disk.nvme0c0n1.out‚15303.47265625# proc.memory.rssMB‚ 2842.24609375) proc.memory.percent‚17.34769344329834 disk.rbd3.out‚0# disk.nvme0n1.out‚15303.47265625 disk.rbd1.out‚ 4.5390625 disk.sda.out‚ 61.7890625 disk.drbd2165.out‚0 disk.drbd2165.in‚0) disk./.usagePercent‚37.83139919299872 disk.sda.in‚0 disk.dm-1.in‚0 disk.dm-0.out‚15303.47265625 disk.sda1.in‚0 disk.sda1.out‚0( proc.memory.availableMB‚ 973.50390625 disk.nvme0n1.in‚17.53125$ memory_percent‚94.05820369720459$ disk./.usageGB‚563.7151870727539 disk.sda2.in‚0 disk.rbd1.in‚0 disk.rbd0.out‚0 disk.dm-1.out‚ 61.7890625 disk.sda3.in‚0 disk.rbd0.in‚0 disk.dm-0.in‚17.53125 proc.cpu.threads‚17gÝ¥ô”Á‚(:‰ ÌÂèÒé’ø gpu.0.gpu‚98.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚262.589' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.5178 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0™DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚328520525315584 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚9705283 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1076246748672 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚19621797 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389056 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116224 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚9906542 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚328521896878080 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9978 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3242276 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1388212570624 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600318 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚89397344686080 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚328520525315584 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚9705277 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1074406643855 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12153 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536960 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 trainer/global_step‚1149 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1077592137728 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12096 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚90785959884800 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1076246748672 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚19611819 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚327444278566912 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318922 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3242037 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028286 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚9705283 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚327302376599552 % _timestamp‚1.7842917017089334e+09 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚327444304740352 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2946485 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚239 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1077592137728 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚90785557256704 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471115 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚328378127843725 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚328376757815439 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚9705277 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12096 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚327444278566912 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116224 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚9916514 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9972 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚9906542 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚9916514 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚6188766 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚244 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _runtime‚1885.926354908 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318922 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3242037 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028286 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚9705283 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚327302376599552 % _timestamp‚1.7842917017089334e+09 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚327444304740352 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2946485 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚239 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1077592137728 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚90785557256704 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471115 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚328378127843725 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚328376757815439 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚9705277 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12096 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚327444278566912 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116224 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚9916514 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9972 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚9906542 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚9916514 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚6188766 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚244 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _runtime‚1885.926354908 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚328520525315584 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚9705283 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1076246748672 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚19621797 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389056 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116224 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚9906542 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚328521896878080 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9978 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3242276 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1388212570624 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600318 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚89397344686080 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚328520525315584 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚9705277 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1074406643855[ÇÆ«2Í- _wandbruntime‚1887  _runtime‚1887f¡'óê&ÎäM ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚89474831093760 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318922 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545123840 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1078517180416 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚89475199184896 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1389442847744 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚328800822650880 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚90864642032640 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 % _timestamp‚1.7842917031232016e+09 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚9925006 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚327580232949760 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚9713483 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116224 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1075330110101 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚327580258377728 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12096 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3244778 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚327722279297024 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2949028 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12153 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3245011 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚9915020 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371569664 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371569664 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚6194044 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9986 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116224 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚233 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028398 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚328800822650880 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚9713489 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚327722305470464 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396224 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396224 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚6193806 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471115 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚19638495 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620928 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚328655563059861 trainer/global_step‚1149Ð3Öî&ÏèM G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚19638495 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1078517180416 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚90864239411712 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚328800822650880 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚9713489 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚327722305470464 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396224 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396224 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚6193806 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471115 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚19638495 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620928 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚328655563059861 trainer/global_step‚1149 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚89474831093760 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318922 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545123840 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1078517180416 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚89475199184896 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1389442847744 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚328800822650880 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚90864642032640 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 % _timestamp‚1.7842917031232016e+09 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚9925006 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚327580232949760 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚9713483 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116224 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1075330110101 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚327580258377728 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12096 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3244778 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚327722279297024 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2949028 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12153 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3245011 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚9915020 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371569664 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371569664 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚6194044 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9986 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116224 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚233 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028398 PDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚342466877751808 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9972 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12096 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9972 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚93272033161216 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545123840 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3073635 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 400530944 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚94720053860864 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545123840 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚20457849 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚10115577 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 _step‚95 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚32439808 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1448388790784 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371562496 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚10115577 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚93271665070080 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚6453124 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚341344341245952 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12153 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚229 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371562496 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600318 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚10342266 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚6453358 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1123908068352 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9978 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471115 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318922 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚10342266åÀ èì'äæO GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545123840 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3073635 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 400530944 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚94720053860864 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545123840 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚20457849 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚10115577 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 _step‚95 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚32439808 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1448388790784 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371562496 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚10115577 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚93271665070080 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚6453124 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚341344341245952 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12153 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚229 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371562496 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600318 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚10342266 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚6453358 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1123908068352 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9978 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471115 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318922 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚10342266 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚10332294 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚342468249314304 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚341344315072512 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3073640 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚94720454391808 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028286 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12096 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3379489 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1448421230592 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389056 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚234 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 trainer/global_step‚1199 % _timestamp‚1.7842917833078659e+09 _runtime‚1967.525304852 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1122562679296 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1123908068352 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116224 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚341196440080384 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1121988357281 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚342466877751808 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9972 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12096 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9972 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚93272033161216 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175_WÄÿ2å- _wandbruntime‚1968  _runtime‚1968YÏÑ—æìM NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚10340772 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3382418 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9986 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚341474347286528 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 _runtime‚1968.936319162 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚341622341976064 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1124833111040 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116224 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1121567223209 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9986 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚342597259110167 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116224 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3076178 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1449635664896 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371569664 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚32432640 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9992 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚6458375 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600430 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396224 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚93349151477760 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚341474321858560 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚342747175087104 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 400523776 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚341622341976064 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚221 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545123840 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1123487714816 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318922 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471115 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚10123783 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚10123783 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1123487714816 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚342745803517440 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚6458601 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚226 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚94799155233792 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚10340772 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12153 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚20474547 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚10123789 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚93349519568896 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371569664 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚32432640 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9992 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚6458375 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600430 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396224 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚93349151477760 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚341474321858560 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚342747175087104 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 400523776 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚341622341976064 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚221 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545123840 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1123487714816 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318922 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471115 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚10123783 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚10123783 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1123487714816 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚342745803517440 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚6458601 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚226 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚94799155233792 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚10340772 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12153 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚20474547 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚10123789 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚93349519568896 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9986 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚341474347286528 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 _runtime‚1968.936319162 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚341622341976064 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1124833111040 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116224 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1121567223209`I¨è“2è- _wandbruntime‚1969  _runtime‚1969µÍéÿ 2loss/SRqqt2/assignment_loss‚0.8660970777086914 /loss/FRt2/detection_loss‚1.6736332121274562 1loss/SRqqt2/detection_loss‚0.7867315648985095 &loss/total_loss‚13.394265571092266 /loss/FRt2/assignment_loss‚1.630762821072144 1loss/FBt1/assignment_loss‚0.09307269053325844 1loss/SRqqt1/detection_loss‚1.2681522503623675 /loss/FBt2/detection_loss‚1.3701054301551159 trainer/global_step‚1199 _runtime‚1969.011256953 _step‚97 /loss/FBt1/detection_loss‚0.6945137794104379 epoch‚2 0loss/FRt1/assignment_loss‚2.8988521350728864 1loss/SRqqt1/assignment_loss‚1.143391274943434 1loss/FBt2/assignment_loss‚0.36545029937299073 /loss/FRt1/detection_loss‚0.6035030354349726 % _timestamp‚1.7842917847940528e+09e.|Þ êƒ 2loss/SRqqt2/assignment_loss‚0.8660970777086914 /loss/FRt2/detection_loss‚1.6736332121274562 1loss/SRqqt2/detection_loss‚0.7867315648985095 &loss/total_loss‚13.394265571092266 /loss/FRt2/assignment_loss‚1.630762821072144 1loss/FBt1/assignment_loss‚0.09307269053325844 1loss/SRqqt1/detection_loss‚1.2681522503623675 /loss/FBt2/detection_loss‚1.3701054301551159 trainer/global_step‚1199 _runtime‚1969.011256953 _step‚97 /loss/FBt1/detection_loss‚0.6945137794104379 epoch‚2 0loss/FRt1/assignment_loss‚2.8988521350728864 1loss/SRqqt1/assignment_loss‚1.143391274943434 1loss/FBt2/assignment_loss‚0.36545029937299073 /loss/FRt1/detection_loss‚0.6035030354349726 % _timestamp‚1.7842917847940528e+09ad2òœë‚(:‘ ÓÃèÒßÖþ÷ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚333.524' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚66.70479999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0*´Îì‚(:à ÓÃèÒÏ”ªù disk.sda1.in‚0 disk.rbd3.in‚0 disk.sda.out‚64.65625 cpu‚25.6208533549005 disk.sda.in‚0 disk.dm-1.in‚0 disk.rbd0.out‚0* disk./.usagePercent‚37.834905984202734 disk.sda1.out‚0 disk.sda3.in‚0 disk.sda3.out‚64.65625 disk.rbd1.out‚ 4.9140625 disk.rbd0.in‚0 disk.dm-0.in‚ 19.953125 network.sent‚ 2.101701e+06 network.recv‚537944 disk.rbd3.out‚0 disk.nvme0n1.in‚ 19.953125" disk.nvme0n1.out‚ 19818.4765625 disk.rbd1.in‚0 proc.memory.rssMB‚ 2841.96875$ disk.nvme0c0n1.out‚ 19818.4765625 disk.sda2.out‚0 disk.dm-0.out‚ 19818.4765625$ memory_percent‚94.04795169830322$ disk./.usageGB‚563.7674407958984 disk.drbd2165.out‚0( proc.memory.availableMB‚ 975.18359375) proc.memory.percent‚17.34600067138672 proc.cpu.threads‚17 disk.drbd2165.in‚0 disk.nvme0c0n1.in‚ 19.953125 disk.sda2.in‚0 disk.dm-1.out‚64.65625 ×ùÅ”í‚(:‰ âÃèÒ¶Áýö gpu.0.gpu‚99.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚340.764' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚68.1528 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¸rsèî‚(:Ý âÃèÒúâ¢ø disk.nvme0n1.in‚ 20.26171875 disk.rbd1.out‚ 4.9140625 disk.sda.in‚0 disk.dm-1.out‚ 64.8203125 network.recv‚542527 disk.sda3.in‚0( disk.nvme0c0n1.out‚19828.10595703125 disk.sda2.in‚0 disk.rbd0.in‚0( proc.memory.availableMB‚ 972.49609375 disk.drbd2165.in‚0 disk.rbd3.out‚0 disk.sda2.out‚0 disk.rbd1.in‚0 disk.dm-1.in‚0 disk.dm-0.in‚ 20.26171875$ memory_percent‚94.06435489654541 cpu‚25.63905937662162) disk./.usagePercent‚37.83536218962875 disk.rbd3.in‚0! disk.nvme0c0n1.in‚ 20.26171875 disk.rbd0.out‚0 proc.cpu.threads‚17$ disk./.usageGB‚563.7742385864258 disk.sda3.out‚ 64.8203125& disk.nvme0n1.out‚19828.10595703125 disk.sda1.in‚0 network.sent‚ 2.164734e+06 disk.sda.out‚ 64.8203125 disk.sda1.out‚0! proc.memory.rssMB‚ 2844.609375# disk.dm-0.out‚19828.10595703125* proc.memory.percent‚17.362117767333984 disk.drbd2165.out‚0ã—/®2ï- _wandbruntime‚1996  _runtime‚1996KèoV 2ð¦| 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.13463500142097473 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.34953704476356506 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.42933550477027893 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.30356699228286743 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.3333333432674408 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.1919509768486023 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 _runtime‚1996.786396303 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.3268868029117584 /particle/accuracy_4_of_4‚0.9349434971809387 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 6 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.968309223651886 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9913836717605591 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.17745304107666016 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.009847652167081833 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.3271480202674866 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9656745195388794 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.009122497402131557 particle/accuracy_1_of_4‚1 + Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9375 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.15849518775939941 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 epoch‚2 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9007561802864075 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 :"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.31640851497650146 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9868519902229309 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.7099395394325256 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.08196485787630081 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.3326355814933777 :"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.35637444257736206 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.15809930860996246 'particle/accuracy‚0.817812979221344 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.39010560512542725 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.8472222089767456 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8296284675598145 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.3333333432674408 :"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.36858582496643066 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9710730314254761 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.06194411963224411 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.18436597287654877 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.018512234091758728 ."Purity/2FRt0SRqqt1FBt/event_purity‚0.21875 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.28703704476356506 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.7777777910232544 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.3103618025779724 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.14128558337688446 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9722222089767456 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9888964891433716 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9783183336257935 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9981371760368347 6 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.987500011920929 *jet/accuracy_1_of_4‚0.7956437468528748 +particle/sensitivity‚0.5673564076423645 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 6 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.989322304725647 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 particle/accuracy_2_of_2‚1 jet/accuracy_1_of_5‚1 jet/accuracy_2_of_5‚1 jet/accuracy_3_of_5‚1 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.3454428017139435 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ,jet/accuracy_4_of_4‚0.007130124606192112 jet/accuracy_5_of_5‚0 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.018908461555838585 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.28703704476356506 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.36989307403564453 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.004940463230013847 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 ,jet/accuracy_3_of_3‚0.052040185779333115 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.09823662787675858 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.6704451441764832 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.09559980034828186 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.4060523211956024 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.10781341791152954 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.07333333790302277 particle/accuracy_1_of_3‚1 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.09710762649774551 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.1573178768157959 +jet/accuracy_3_of_4‚0.06278160959482193 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 jet/accuracy_4_of_5‚0 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.4174024164676666 *jet/accuracy_1_of_2‚0.4996891915798187 /particle/accuracy_3_of_4‚0.9978355169296265 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.2838059365749359 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8413876891136169 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.45883604884147644 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.08378909528255463 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4490741193294525 *loss/val_total_loss‚12.673405558300022 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.46242183446884155 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.3047589659690857 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.14527453482151031 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.47541671991348267 particle/accuracy_2_of_3‚1 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.14649923145771027 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9973781108856201 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 particle/accuracy_2_of_5‚1 particle/accuracy_4_of_5‚0 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9959626793861389 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9919086694717407 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.3695618510246277 ."Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.34375 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 particle/accuracy_2_of_4‚1 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.1625143587589264 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 8"Purity/*FRt*SRqqt2FBt/event_purity‚0.816939651966095 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.6578231453895569 *jet/accuracy_2_of_4‚0.4302608370780945 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.8846153616905212 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.5357142686843872 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5448443293571472 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.06314552575349808 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7592592239379883 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.04493991658091545 :"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.33629298210144043 *jet/accuracy_1_of_3‚0.7112010717391968 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.14968940615653992 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.3675963580608368 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 $ _timestamp‚1.784291812568054e+09 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5114579200744629 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.38383129239082336 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 +jet/accuracy_2_of_3‚0.29856839776039124 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.04577640816569328 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9813492298126221 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 8"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.297135591506958 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7592592239379883 particle/accuracy_1_of_1‚1 particle/accuracy_1_of_5‚1 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7058201432228088 +particle/specificity‚0.8863130211830139 'particle/f_score‚0.5721859931945801 , Purity/2FRt*SRqqt2FBt/FRt_purity‚0.53125 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.3256514072418213 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7201436161994934 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.23076923191547394 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9570783972740173 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.3208308815956116 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.5364583730697632 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.09454037249088287 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.08056802302598953 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3499999940395355 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.15809930860996246 -"Purity/2FRt*SRqqt2FBt/event_purity‚0.3125 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5637455582618713 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9976362586021423 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9785391688346863 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6091049909591675 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 /particle/accuracy_3_of_3‚0.9994933605194092 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.08273428678512573 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.4797712564468384 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.4056529998779297 particle/accuracy_1_of_2‚1 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.6064625978469849 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9981371760368347 5 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.97784423828125 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.14055967330932617 :"Purity/0FRt*SRqqt*FBt/event_purity‚0.38536980748176575 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.8738566637039185 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 )jet/accuracy_2_of_2‚0.087006576359272 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 5 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.99150151014328 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.10066589713096619 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9917927980422974 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 +jet/accuracy_1_of_1‚0.17064696550369263 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 6 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.450166255235672 8"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.316514253616333 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9952840209007263 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.ÌÌRw" 0027204242069274187 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 :"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.38948893547058105 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9902195930480957 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.10920199006795883 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.17543937265872955 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.11089817434549332 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.07173372805118561 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.08378909528255463 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4426479935646057 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.12471207976341248 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.3571428656578064 _step‚98 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5306503176689148 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.3449500799179077 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.7001512050628662 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5942438840866089 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.10335930436849594 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.010857288725674152 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.3561795651912689 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9887093305587769 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.4746018350124359 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.1309596598148346 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.23076923191547394 trainer/global_step‚1199 :"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.35968393087387085 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.4174024164676666 7validation_average_jet_accuracy‚0.20889036357402802 particle/accuracy_3_of_5‚1 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.11874783784151077 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9809938073158264 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.09338869154453278 7 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.9958513975143433 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.14695490896701813 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 particle/accuracy_5_of_5‚0 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9863596558570862 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.3047589659690857 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9285714030265808œ)b0>ñª| 6 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.450166255235672 8"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.316514253616333 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9952840209007263 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 :"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.38948893547058105 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9902195930480957 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.10920199006795883 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.17543937265872955 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.11089817434549332 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.07173372805118561 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.08378909528255463 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4426479935646057 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.12471207976341248 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.3571428656578064 _step‚98 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5306503176689148 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.3449500799179077 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.7001512050628662 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5942438840866089 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.10335930436849594 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.010857288725674152 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.3561795651912689 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9887093305587769 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.4746018350124359 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.1309596598148346 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.23076923191547394 trainer/global_step‚1199 :"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.35968393087387085 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.4174024164676666 7validation_average_jet_accuracy‚0.20889036357402802 particle/accuracy_3_of_5‚1 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.11874783784151077 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9809938073158264 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.09338869154453278 7 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.9958513975143433 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.14695490896701813 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 particle/accuracy_5_of_5‚0 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9863596558570862 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.3047589659690857 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9285714030265808 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.13463500142097473 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.34953704476356506 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.42933550477027893 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.30356699228286743 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.3333333432674408 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.1919509768486023 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 _runtime‚1996.786396303 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.3268868029117584 /particle/accuracy_4_of_4‚0.9349434971809387 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 6 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.968309223651886 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9913836717605591 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.17745304107666016 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.009847652167081833 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.3271480202674866 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9656745195388794 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.009122497402131557 particle/accuracy_1_of_4‚1 + Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9375 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.15849518775939941 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 epoch‚2 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9007561802864075 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 :"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.31640851497650146 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9868519902229309 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.7099395394325256 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.08196485787630081 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.3326355814933777 :"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.35637444257736206 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.15809930860996246 'particle/accuracy‚0.817812979221344 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.39010560512542725 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.8472222089767456 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8296284675598145 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.3333333432674408 :"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.36858582496643066 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9710730314254761 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.06194411963224411 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.18436597287654877 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.018512234091758728 ."Purity/2FRt0SRqqt1FBt/event_purity‚0.21875 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.28703704476356506 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.7777777910232544 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.3103618025779724 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.14128558337688446 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9722222089767456 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9888964891433716 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9783183336257935 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9981371760368347 6 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.987500011920929 *jet/accuracy_1_of_4‚0.7956437468528748 +particle/sensitivity‚0.5673564076423645 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 6 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.989322304725647 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 particle/accuracy_2_of_2‚1 jet/accuracy_1_of_5‚1 jet/accuracy_2_of_5‚1 jet/accuracy_3_of_5‚1 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.3454428017139435 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ,jet/accuracy_4_of_4‚0.007130124606192112 jet/accuracy_5_of_5‚0 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.018908461555838585 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.28703704476356506 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.36989307403564453 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.004940463230013847 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 ,jet/accuracy_3_of_3‚0.052040185779333115 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.09823662787675858 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.6704451441764832 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.09559980034828186 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.4060523211956024 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.10781341791152954 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.07333333790302277 particle/accuracy_1_of_3‚1 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.09710762649774551 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.1573178768157959 +jet/accuracy_3_of_4‚0.06278160959482193 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 jet/accuracy_4_of_5‚0 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.4174024164676666 *jet/accuracy_1_of_2‚0.4996891915798187 /particle/accuracy_3_of_4‚0.9978355169296265 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.2838059365749359 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8413876891136169 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.45883604884147644 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.08378909528255463 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4490741193294525 *loss/val_total_loss‚12.673405558300022 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.46242183446884155 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.3047589659690857 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.14527453482151031 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.47541671991348267 particle/accuracy_2_of_3‚1 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.14649923145771027 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9973781108856201 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 particle/accuracy_2_of_5‚1 particle/accuracy_4_of_5‚0 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9959626793861389 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9919086694717407 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.3695618510246277 ."Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.34375 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 particle/accuracy_2_of_4‚1 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.1625143587589264 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 8"Purity/*FRt*SRqqt2FBt/event_purity‚0.816939651966095 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.6578231453895569 *jet/accuracy_2_of_4‚0.4302608370780945 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.8846153616905212 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.5357142686843872 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5448443293571472 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.06314552575349808 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7592592239379883 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.04493991658091545 :"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.33629298210144043 *jet/accuracy_1_of_3‚0.7112010717391968 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.14968940615653992 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.3675963580608368 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 $ _timestamp‚1.784291812568054e+09 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5114579200744629 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.38383129239082336 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 +jet/accuracy_2_of_3‚0.29856839776039124 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.04577640816569328 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9813492298126221 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 8"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.297135591506958 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7592592239379883 particle/accuracy_1_of_1‚1 particle/accuracy_1_of_5‚1 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7058201432228088 +particle/specificity‚0.8863130211830139 'particle/f_score‚0.5721859931945801 , Purity/2FRt*SRqqt2FBt/FRt_purity‚0.53125 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.3256514072418213 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7201436161994934 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.23076923191547394 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9570783972740173 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.3208308815956116 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.5364583730697632 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.09454037249088287 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.08056802302598953 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3499999940395355 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.15809930860996246 -"Purity/2FRt*SRqqt2FBt/event_purity‚0.3125 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5637455582618713 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9976362586021423 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9785391688346863 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6091049909591675 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 /particle/accuracy_3_of_3‚0.9994933605194092 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.08273428678512573 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.4797712564468384 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.4056529998779297 particle/accuracy_1_of_2‚1 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.6064625978469849 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9981371760368347 5 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.97784423828125 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.14055967330932617 :"Purity/0FRt*SRqqt*FBt/event_purity‚0.38536980748176575 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.8738566637039185 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 )jet/accuracy_2_of_2‚0.087006576359272 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 5 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.99150151014328 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.10066589713096619 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9917927980422974 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 +jet/accuracy_1_of_1‚0.17064696550369263 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0bÒ Ø|”ò‚(:‰ ñÃèÒºòãö gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚41.0 gpu.0.powerWatts‚288.968' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚57.7936 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0†#ý÷çó‚(:Ü ñÃèÒŒ¯Éø disk.sda1.in‚0* disk./.usagePercent‚37.899380969686085 disk.sda2.out‚0 disk.dm-1.in‚0 proc.cpu.threads‚17 network.sent‚ 2.169695e+06$ disk./.usageGB‚564.7281646728516 disk.sda.in‚0 disk.rbd0.in‚0 disk.sda3.out‚ 65.30078125 disk.rbd3.in‚0 disk.rbd3.out‚0) proc.memory.availableMB‚ 1944.88671875 network.recv‚545790! disk.dm-0.out‚21773.572265625 disk.sda2.in‚0* proc.memory.percent‚17.362117767333984 disk.sda.out‚ 65.30078125 disk.rbd0.out‚0 disk.dm-0.in‚ 20.26171875 disk.dm-1.out‚ 65.30078125 disk.sda3.in‚0 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.rbd1.out‚ 4.9765625! proc.memory.rssMB‚ 2844.609375! disk.nvme0c0n1.in‚ 20.26171875& disk.nvme0c0n1.out‚21773.572265625 disk.sda1.out‚0# memory_percent‚88.1293535232544 cpu‚25.635812142576906 disk.nvme0n1.in‚ 20.26171875$ disk.nvme0n1.out‚21773.572265625 disk.rbd1.in‚0Àrßô‚(:„ €ÄèÒà„Õö gpu.0.gpu‚96.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚330.0' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚66.0 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‡õú²êõ‚(:ß €ÄèÒ™Ó¡ù disk.sda1.in‚0( disk.nvme0c0n1.out‚21786.35693359375$ disk./.usageGB‚564.7345161437988 disk.rbd0.in‚0 disk.sda1.out‚0 disk.sda3.out‚ 65.47265625 disk.dm-1.in‚0 disk.dm-1.out‚ 65.47265625 disk.dm-0.in‚ 20.55859375# disk.dm-0.out‚21786.35693359375 disk.drbd2165.out‚0 disk.nvme0n1.in‚ 20.55859375& disk.nvme0n1.out‚21786.35693359375 disk.sda2.out‚0) proc.memory.availableMB‚ 1944.84765625 network.sent‚ 2.227473e+06# memory_percent‚88.1295919418335! proc.memory.rssMB‚ 2844.609375 disk.sda2.in‚0 disk.rbd1.in‚0 network.recv‚550280( disk./.usagePercent‚37.8998072222306 disk.sda.out‚ 65.47265625 disk.sda3.in‚0* proc.memory.percent‚17.362117767333984 cpu‚25.63724069707723 disk.sda.in‚0 disk.rbd0.out‚0 disk.drbd2165.in‚0 disk.rbd3.in‚0 disk.rbd3.out‚0 proc.cpu.threads‚17! disk.nvme0c0n1.in‚ 20.55859375 disk.rbd1.out‚ 4.9765625+sâË”ö‚(:‰ ÄèÒó¢Ÿ÷ gpu.0.gpu‚86.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚332.173' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚66.4346 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚00ÜT—é÷‚(:Þ ÄèÒœæø disk.nvme0c0n1.in‚ 20.8671875 disk.nvme0n1.in‚ 20.8671875 disk.rbd0.in‚0 disk.dm-0.in‚ 20.8671875 disk.drbd2165.in‚0 disk.rbd3.in‚0 disk.sda2.in‚0 disk.rbd1.out‚ 5.0390625 proc.cpu.threads‚17* disk./.usagePercent‚37.899777525356626 disk.rbd1.in‚0 disk.sda3.out‚ 65.68359375$ memory_percent‚88.12968730926514) proc.memory.availableMB‚ 1944.83203125 disk.rbd3.out‚0 disk.sda.in‚0* proc.memory.percent‚17.362117767333984 cpu‚25.63552790006366 network.recv‚553956 disk.drbd2165.out‚0& disk.nvme0n1.out‚22765.78564453125 disk.sda.out‚ 65.68359375# disk.dm-0.out‚22765.78564453125 disk.sda1.in‚0 network.sent‚ 2.232494e+06 disk.sda1.out‚0 disk.sda3.in‚0! proc.memory.rssMB‚ 2844.609375 disk.dm-1.in‚0 disk.dm-1.out‚ 65.68359375 disk.rbd0.out‚0( disk.nvme0c0n1.out‚22765.78564453125 disk.sda2.out‚0# disk./.usageGB‚564.734073638916ù¬ÅŒŸø‚(:” žÄèÒÕÒíö gpu.0.gpu‚88.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚284.867' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚56.973400000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¡ŠÍÝãù‚(:Ø žÄèÒûÁÉø disk.rbd3.in‚0& disk.nvme0c0n1.out‚22776.970703125$ disk.nvme0n1.out‚22776.970703125 network.sent‚ 2.237571e+06 disk.sda2.out‚0 network.recv‚557301) disk./.usagePercent‚37.90023859492579! disk.dm-0.out‚22776.970703125 disk.sda1.out‚0 disk.rbd1.in‚0 disk.dm-1.in‚0 disk.sda1.in‚0$ disk./.usageGB‚564.7409439086914 disk.sda.in‚0 disk.rbd0.in‚0 disk.sda3.out‚ 65.96484375 disk.sda2.in‚0 disk.rbd1.out‚ 5.0390625( proc.memory.availableMB‚ 1944.8671875! proc.memory.rssMB‚ 2844.609375 disk.sda3.in‚0 disk.rbd3.out‚0$ memory_percent‚88.12947273254395* proc.memory.percent‚17.362117767333984 cpu‚25.633833711713088 disk.sda.out‚ 65.96484375 disk.dm-1.out‚ 65.96484375 disk.dm-0.in‚ 21.1640625 disk.drbd2165.in‚0 disk.nvme0n1.in‚ 21.1640625 proc.cpu.threads‚17 disk.rbd0.out‚0 disk.drbd2165.out‚0 disk.nvme0c0n1.in‚ 21.1640625‘—ÊŸú‚(:” ­ÄèÒ‰Øôö gpu.0.gpu‚98.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚293.084' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚58.616800000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0²šâ,àû‚(:Õ ­ÄèÒúãŠù network.recv‚560746 disk.sda.out‚ 66.29296875 disk.rbd0.in‚0 disk.nvme0n1.in‚ 21.47265625 network.sent‚ 2.242695e+06 disk.rbd1.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 66.29296875 disk.rbd0.out‚0 disk.dm-0.in‚ 21.47265625 disk.rbd1.out‚ 5.1015625 disk.drbd2165.in‚0 disk.drbd2165.out‚0$ memory_percent‚88.13140392303467) disk./.usagePercent‚37.90070529666071 disk.dm-0.out‚ 27190.3046875 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda3.out‚ 66.29296875 disk.rbd3.in‚0 disk.rbd3.out‚0$ disk.nvme0c0n1.out‚ 27190.3046875 disk.sda2.out‚0$ disk./.usageGB‚564.7478981018066) proc.memory.availableMB‚ 1944.55078125* proc.memory.percent‚17.362117767333984! proc.memory.rssMB‚ 2844.609375 proc.cpu.threads‚17 disk.sda.in‚0! disk.nvme0c0n1.in‚ 21.47265625 cpu‚25.63214500348781 disk.sda3.in‚0" disk.nvme0n1.out‚ 27190.3046875 disk.sda2.in‚09EÂd2ü- _wandbruntime‚2083  _runtime‚2083…ÇAÄÆýÀ _step‚99 & lr-AdamW/pg1‚0.0006886504135944074 & lr-AdamW/pg2‚0.0006886504135944074 trainer/global_step‚1249 % _timestamp‚1.7842918993620222e+09 _runtime‚2083.579242447ÆhÊþÄ _step‚99 & lr-AdamW/pg1‚0.0006886504135944074 & lr-AdamW/pg2‚0.0006886504135944074 trainer/global_step‚1249 % _timestamp‚1.7842918993620222e+09 _runtime‚2083.579242447cM2[.2ÿ- _wandbruntime‚2083  _runtime‚2083]4Ñ€ âO D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚360090517951590 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚10657889 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚97194465792000 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 $ _timestamp‚1.784291899363266e+09 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1174721223168 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚10817031 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1174721223168 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚359072333012992 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1515928492032 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚360247054236160 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚359072359186432 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚257 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚21474914 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3551535 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚359072333012992 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚10657889 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚98710394284032 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚6760986 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚358916463349760 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚6761243 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚359072359186432 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚2083.580867636 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3551283 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚10827004 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚10657883 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1176066612736 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚10657883 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚10817031 \HDeviceStatsMonitor.on_train_batch_start/rc§equested_bytes.large_pool.freed‚358916437921792 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚21484893 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚252 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1515893955584 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚360089147923300 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 _step‚100 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3209708 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚360248425799168 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚97194097700864 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3209703 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1174054601830 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1172710001508 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 trainer/global_step‚1249 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚10827004 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1176066612736 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚360247054236160 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚2083.580867636 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3551283 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚10827004 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚10657883 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1176066612736 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚10657883 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚10817031 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚358916437921792 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚21484893 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚252 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1515893955584 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚360089147923300 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 _step‚100 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3209708 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚360248425799168 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚97194097700864 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3209703 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1174054601830 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1172710001508 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 trainer/global_step‚1249 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚10827004 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1176066612736 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚360247054236160 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 IDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1176991655424 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚10825509 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3554050 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚360527351571968 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚359194345127936 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1175646258688 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚10835496 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚97271584108544 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1173633467754 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚10666089 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1517101726720 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚359350333743104 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚359350359916544 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚21501591 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚21491598 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚10666095 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚360369323196124 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚10835496 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 % _timestamp‚1.7842919007714293e+09 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚360527351571968 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚6766296 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3212251 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚359350333743104 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚98788685835264 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚97271952199680 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚241 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚6766542 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1517136256000 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 trainer/global_step‚1249 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚21501591 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚10666095 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3212246 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1174978068188 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚359350359916544 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 _step‚101 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚360367953167722 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820dO´µ „ ñM MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚10835496 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 % _timestamp‚1.7842919007714293e+09 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚360527351571968 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚6766296 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3212251 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚359350333743104 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚98788685835264 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚97271952199680 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚241 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚6766542 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1517136256000 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 trainer/global_step‚1249 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚21501591 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚10666095 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3212246 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1174978068188 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚359350359916544 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 _step‚101 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚360367953167722 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1176991655424 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚10825509 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3554050 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚360527351571968 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚359194345127936 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1175646258688 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚10835496 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚97271584108544 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1173633467754 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚10666089 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1517101726720 WHDeviceStatsMonitor.on_train_iÕÆ.Bbatch_end/requested_bytes.small_pool.current‚ 1344600434 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚359350333743104 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚359350359916544 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚21501591 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚21491598 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚10666095 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚360369323196124 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176e35\øŸ… ‚(:” ¼ÄèÒúÙÃø gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚281.431' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚56.286199999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0&A<ðæ† ‚(:Û ¼ÄèҷЗù! disk.nvme0c0n1.in‚ 21.47265625 network.sent‚ 2.247681e+06) disk./.usagePercent‚37.90090882265043 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda2.out‚0$ memory_percent‚88.13130855560303) proc.memory.availableMB‚ 1944.56640625 disk.dm-0.in‚ 21.47265625 disk.sda1.in‚0* proc.memory.percent‚17.362117767333984" disk.dm-0.out‚27199.1982421875 disk.sda3.in‚0 disk.sda.in‚0 disk.dm-1.in‚0 cpu‚25.63049068832862 disk.rbd0.out‚0' disk.nvme0c0n1.out‚27199.1982421875 proc.cpu.threads‚17 disk.sda.out‚ 66.6015625 disk.dm-1.out‚ 66.6015625$ disk./.usageGB‚564.7509307861328 disk.rbd3.in‚0 disk.rbd1.out‚ 5.1015625 network.recv‚564228 disk.rbd0.in‚0% disk.nvme0n1.out‚27199.1982421875 disk.sda3.out‚ 66.6015625 disk.drbd2165.in‚0 disk.rbd3.out‚0 disk.nvme0n1.in‚ 21.47265625 disk.rbd1.in‚0! proc.memory.rssMB‚ 2844.609375 disk.drbd2165.out‚0ž’¢2‡ - _wandbruntime‚2085  _runtime‚20858³ˆ € 2loss/SRqqt1/assignment_loss‚1.1116906001254738 /loss/FRt2/detection_loss‚1.7564066426621543 1loss/SRqqt1/detection_loss‚1.3009149745993671 &loss/total_loss‚12.512562347025629 /loss/FRt1/assignment_loss‚2.808287773146335 2loss/SRqqt2/assignment_loss‚0.8414353590875404 1loss/FBt2/assignment_loss‚0.06321022756190953 1loss/SRqqt2/detection_loss‚0.7952419216341273 epoch‚3 _runtime‚2085.066416043 0loss/FRt2/assignment_loss‚1.4949174342332063 1loss/FBt1/assignment_loss‚0.04789396424976202 trainer/global_step‚1249 % _timestamp‚1.7842919008492162e+09 _step‚102 /loss/FRt1/detection_loss‚0.5719854884030572 .loss/FBt1/detection_loss‚0.667558651207792 /loss/FBt2/detection_loss‚1.0530193101149052F¶ò ‰ „ 2loss/SRqqt2/assignment_loss‚0.8414353590875404 1loss/FBt2/assignment_loss‚0.06321022756190953 1loss/SRqqt2/detection_loss‚0.7952419216341273 epoch‚3 _runtime‚2085.066416043 0loss/FRt2/assignment_loss‚1.4949174342332063 1loss/FBt1/assignment_loss‚0.04789396424976202 trainer/global_step‚1249 % _timestamp‚1.7842919008492162e+09 _step‚102 /loss/FRt1/detection_loss‚0.5719854884030572 .loss/FBt1/detection_loss‚0.667558651207792 /loss/FBt2/detection_loss‚1.0530193101149052 2loss/SRqqt1/assignment_loss‚1.1116906001254738 /loss/FRt2/detection_loss‚1.7564066426621543 1loss/SRqqt1/detection_loss‚1.3009149745993671 &loss/total_loss‚12.512562347025629 /loss/FRt1/assignment_loss‚2.808287773146335fWÒØžŠ ‚(:“ ËÄèÒ‹ó˜÷ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚333.764' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚66.75280000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Œåø”ä‹ ‚(:Ù ËÄèÒʦçø( disk.nvme0c0n1.out‚27209.96923828125 disk.sda.out‚ 66.765625 network.sent‚ 2.309962e+06 disk.rbd1.out‚ 5.1640625 disk.rbd0.out‚0) disk./.usagePercent‚37.90133789127782 disk.rbd3.out‚0 disk.rbd1.in‚0) proc.memory.availableMB‚ 1944.52734375& disk.nvme0n1.out‚27209.96923828125 disk.rbd0.in‚0$ memory_percent‚88.13154697418213! proc.memory.rssMB‚ 2844.609375 network.recv‚568872 disk.sda3.in‚0 disk.drbd2165.out‚0 disk.sda.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 66.765625 disk.sda1.in‚0 disk.rbd3.in‚0 disk.sda2.in‚0 disk.dm-0.in‚ 21.76953125 disk.sda3.out‚ 66.765625 disk.drbd2165.in‚0! disk.nvme0c0n1.in‚ 21.76953125 disk.sda2.out‚0# disk.dm-0.out‚27209.96923828125* proc.memory.percent‚17.362117767333984 proc.cpu.threads‚17" disk./.usageGB‚564.75732421875 disk.sda1.out‚0 disk.nvme0n1.in‚ 21.76953125 cpu‚25.62910748901641 74ÊžŒ ‚(:“ ÚÄèÒØñÎö gpu.0.gpu‚90.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚286.799' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚57.35979999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0|ðÜ‘Ý ‚(:Ò ÚÄèÒÁ·žú disk.sda1.in‚0 disk.sda1.out‚0 disk.sda3.out‚ 66.9296875* proc.memory.percent‚17.362117767333984 cpu‚25.627348129712306 disk.rbd3.out‚0 disk.sda.in‚0 disk.sda.out‚ 66.9296875 disk.dm-1.in‚0 disk.dm-0.in‚ 22.10546875 disk.sda3.in‚0 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.rbd3.in‚0 disk.nvme0n1.in‚ 22.10546875" disk.nvme0n1.out‚ 27223.3515625 disk.rbd1.in‚0 disk.dm-1.out‚ 66.9296875! disk.nvme0c0n1.in‚ 22.10546875$ memory_percent‚88.13176155090332 proc.cpu.threads‚17 network.sent‚ 2.352611e+06$ disk./.usageGB‚560.4576187133789 disk.dm-0.out‚ 27223.3515625$ disk.nvme0c0n1.out‚ 27223.3515625! proc.memory.rssMB‚ 2844.609375) disk./.usagePercent‚37.61278104711915( proc.memory.availableMB‚ 1944.4921875 disk.sda2.out‚0 disk.rbd1.out‚ 5.1640625 disk.rbd0.out‚0 network.recv‚573155 disk.sda2.in‚0 disk.rbd0.in‚0·úYû”Ž ‚(:‰ éÄèÒ϶‘÷ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚194.196' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚38.8392 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0÷/5à ‚(:Õ éÄèÒæȃû$ disk./.usageGB‚560.4651870727539" disk.nvme0n1.out‚ 27241.9140625 disk.rbd0.out‚0 cpu‚25.62586118710822! disk.nvme0c0n1.in‚ 22.40234375 disk.sda2.out‚0 disk.sda.out‚ 68.87890625 disk.sda1.in‚0 disk.rbd3.in‚0 disk.rbd0.in‚0) proc.memory.availableMB‚ 1944.25390625! proc.memory.rssMB‚ 2844.609375* proc.memory.percent‚17.362117767333984 disk.sda.in‚0 disk.dm-0.in‚ 22.40234375 disk.sda3.in‚0 disk.sda3.out‚ 68.87890625 disk.drbd2165.in‚0 network.recv‚576767 disk.sda2.in‚0 disk.drbd2165.out‚0 disk.sda1.out‚0) disk./.usagePercent‚37.61328896606708 disk.rbd3.out‚0$ disk.nvme0c0n1.out‚ 27241.9140625 disk.nvme0n1.in‚ 22.40234375 disk.rbd1.in‚0 proc.cpu.threads‚17 network.sent‚ 2.357595e+06 disk.rbd1.out‚ 5.2265625 disk.dm-1.in‚0 disk.dm-1.out‚ 68.87890625 disk.dm-0.out‚ 27241.9140625$ memory_percent‚88.13321590423584•<œo” ‚(:‰ øÄèÒœýÝö gpu.0.gpu‚98.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚281.966' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚56.3932 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0f\ÎEä‘ ‚(:Ù øÄèÒë¯äø disk.sda.out‚ 69.4296875' disk.nvme0c0n1.out‚27252.9130859375) proc.memory.availableMB‚ 1944.43359375 disk.sda.in‚0 disk.rbd0.in‚0" disk.dm-0.out‚27252.9130859375 disk.drbd2165.in‚0 disk.rbd3.out‚0 network.recv‚580193 disk.sda2.out‚0 disk.rbd1.in‚0 disk.rbd0.out‚0 disk.sda1.out‚0 disk.sda3.in‚0 disk.nvme0c0n1.in‚ 22.7109375$ memory_percent‚88.13211917877197 disk.dm-1.in‚0 disk.dm-1.out‚ 69.4296875 disk.sda2.in‚0 disk.sda3.out‚ 69.4296875* proc.memory.percent‚17.362117767333984 cpu‚25.62412279163211 disk.nvme0n1.in‚ 22.7109375% disk.nvme0n1.out‚27252.9130859375 disk.dm-0.in‚ 22.7109375 proc.cpu.threads‚17* disk./.usagePercent‚37.617669766993004 disk.rbd1.out‚ 5.2265625 disk.sda1.in‚0 disk.drbd2165.out‚0 disk.rbd3.in‚0! proc.memory.rssMB‚ 2844.609375 network.sent‚ 2.362717e+06$ disk./.usageGB‚560.5304641723633g„‹ŸŸ’ ‚(:” ‡ÅèÒ·¨èö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚285.446' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚57.089200000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0gBqJç“ ‚(:Ü ‡ÅèÒëÌÏø disk.rbd3.out‚0% disk.nvme0n1.out‚27259.8701171875 disk.rbd1.out‚ 5.3515625 disk.dm-1.out‚ 70.01953125 disk.nvme0c0n1.in‚ 22.7109375' disk.nvme0c0n1.out‚27259.8701171875! proc.memory.rssMB‚ 2844.609375 network.recv‚582831) disk./.usagePercent‚37.61400527514789" disk.dm-0.out‚27259.8701171875 disk.sda1.out‚0 disk.drbd2165.out‚0 disk.sda2.out‚0) proc.memory.availableMB‚ 1944.44140625* proc.memory.percent‚17.362117767333984 disk.sda1.in‚0$ memory_percent‚88.13207149505615$ disk./.usageGB‚560.4758605957031 disk.sda3.in‚0 cpu‚25.622472155128055 disk.rbd0.in‚0 disk.nvme0n1.in‚ 22.7109375 proc.cpu.threads‚17 disk.sda.out‚ 70.01953125 disk.dm-0.in‚ 22.7109375 disk.rbd1.in‚0 disk.dm-1.in‚0 disk.rbd3.in‚0 disk.sda2.in‚0 network.sent‚ 2.366771e+06 disk.sda.in‚0 disk.rbd0.out‚0 disk.sda3.out‚ 70.01953125 disk.drbd2165.in‚0¾.PF2” - _wandbruntime‚2163  _runtime‚2163Ãcº@Æ• À & lr-AdamW/pg1‚0.0006882633098098136 & lr-AdamW/pg2‚0.0006882633098098136 trainer/global_step‚1299 % _timestamp‚1.7842919793212805e+09 _runtime‚ 2163.53994903 _step‚103Ë/bÊ– Ä & lr-AdamW/pg2‚0.0006882633098098136 trainer/global_step‚1299 % _timestamp‚1.7842919793212805e+09 _runtime‚ 2163.53994903 _step‚103 & lr-AdamW/pg1‚0.0006882633098098136gÊYÉ 2— - _wandbruntime‚2163  _runtime‚2163•wã”í'˜ çO TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚11068189 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚372972369518592 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚7025817 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚374193406672384 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚372810526830592 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3336853 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚102644456967168 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚252 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚247 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚374030843973498 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚11252756 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚374029473945208 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640Ó»‘ñ'™ ëO \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚372972369518592 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚7025817 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚374193406672384 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚372810526830592 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3336853 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚102644456967168 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚252 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚247 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚374030843973498 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚11252756 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚374029473945208 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚11068189 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1h£knAš - _wandbruntßM)ûime‚2164  _runtime‚2164=GÑö&› ðM UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7030842 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚22327650 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1221215181296 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚11076395 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚101145904492544 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚22337643 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚373250396422144 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚11076389 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚373250396422144 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚373250370248704 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1223307586048 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1577268659200 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚374473704008192 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚244 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _step‚105 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚374472332438016 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3691690 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚374309649218032 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚22327650 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚249 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚11261248 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚373250370248704 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1221962189312 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚11261248 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚102723138622464 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚11251261 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚11251261 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚374473704008192Þ·.Uú&œ ôM cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚101146272583680 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1219870580862 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚374308279189630 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚373088408608768 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚22327650 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1221215181296 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚11076395 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚101145904492544 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚22337643 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚373250396422144 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚11076389 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚373250396422144 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚373250370248704 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1223307586048 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1577268659200 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚374473704008192 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚244 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _step‚105 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚374472332438016 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3691690 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚374309649218032 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚22327650 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚249 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚11261248 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚373250370248704 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1221962189312 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚11261248 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚102723138622464 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚11251261 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚11251261 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚374473704008192 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7030842 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 F  ‚(:’ –ÅèÒ§¢…ø gpu.0.gpu‚21.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚41.0 gpu.0.powerWatts‚333.524' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚66.70479999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0³´çç¡ ‚(:Ü –ÅèÒÚíÛù disk.sda3.out‚ 71.3046875 disk.rbd1.in‚0# disk.dm-0.out‚27268.60888671875 network.recv‚586134 disk.nvme0n1.in‚ 22.7109375 disk.sda2.out‚0 disk.rbd1.out‚ 5.3515625$ memory_percent‚88.12968730926514 cpu‚25.621041177642343 proc.cpu.threads‚17 disk.sda1.out‚0 disk.rbd3.in‚0 disk.sda.in‚0 disk.sda.out‚ 71.3046875 network.sent‚ 2.371747e+06 disk.sda3.in‚0 disk.nvme0c0n1.in‚ 22.7109375$ disk./.usageGB‚560.4669151306152 disk.drbd2165.in‚0& disk.nvme0n1.out‚27268.60888671875) disk./.usagePercent‚37.61340493748009 disk.drbd2165.out‚0 disk.rbd3.out‚0 disk.rbd0.out‚0) proc.memory.availableMB‚ 1944.83203125( disk.nvme0c0n1.out‚27268.60888671875 disk.sda2.in‚0 disk.rbd0.in‚0* proc.memory.percent‚17.362117767333984 disk.sda1.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 71.3046875 disk.dm-0.in‚ 22.7109375! proc.memory.rssMB‚ 2844.609375Ô%Óû”¢ ‚(:‰ ¥ÅèÒÀÜ…ø gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚268.238' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.6476 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Êj¥ßᣠ‚(:Ö ¥ÅèÒØêÏø* disk./.usagePercent‚37.613402121397215 disk.rbd1.in‚0 disk.rbd0.out‚0" disk.dm-0.out‚27276.0966796875 disk.sda1.out‚0 disk.sda.in‚0 disk.dm-1.in‚0 disk.sda1.in‚0 disk.drbd2165.out‚0 disk.sda2.in‚0 disk.rbd1.out‚ 5.4140625 disk.sda.out‚ 72.921875 disk.dm-1.out‚ 72.921875 disk.sda3.in‚0 disk.sda3.out‚ 72.921875 disk.rbd3.in‚0 disk.rbd3.out‚0 network.sent‚ 2.434852e+06 network.recv‚590779 disk.nvme0c0n1.in‚ 22.7109375 proc.cpu.threads‚17' disk.nvme0c0n1.out‚27276.0966796875$ memory_percent‚88.13128471374512$ disk./.usageGB‚560.4668731689453 disk.nvme0n1.in‚ 22.7109375% disk.nvme0n1.out‚27276.0966796875 disk.rbd0.in‚0 disk.drbd2165.in‚0( proc.memory.availableMB‚ 1944.5703125* proc.memory.percent‚17.362117767333984 cpu‚25.619343725046598 disk.sda2.out‚0 disk.dm-0.in‚ 22.7109375! proc.memory.rssMB‚ 2844.609375ÑX錔¤ ‚(:‰ ´ÅèÒ×¼Íö gpu.0.gpu‚42.0 gpu.0.memory‚13, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚333.229' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚66.6458 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0=à€á¥ ‚(:Ö ´ÅèÒší±ø disk.dm-1.in‚0 disk.rbd0.out‚0 disk.rbd1.out‚ 5.4140625 disk.sda.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda3.out‚ 73.703125 disk.rbd3.out‚0) proc.memory.availableMB‚ 1944.56640625 disk.sda2.out‚0! proc.memory.rssMB‚ 2844.609375 disk.sda.out‚ 73.703125 disk.rbd3.in‚0 disk.rbd1.in‚0$ memory_percent‚88.13130855560303* proc.memory.percent‚17.362117767333984 network.recv‚593894" disk.dm-0.out‚27283.2275390625 disk.sda3.in‚0 disk.drbd2165.in‚0 disk.nvme0c0n1.in‚ 22.7109375' disk.nvme0c0n1.out‚27283.2275390625 disk.nvme0n1.in‚ 22.7109375 cpu‚25.617872674663477 network.sent‚ 2.439839e+06$ disk./.usageGB‚560.4674072265625 disk.rbd0.in‚0% disk.nvme0n1.out‚27283.2275390625 proc.cpu.threads‚17) disk./.usagePercent‚37.61343796245201 disk.dm-1.out‚ 73.703125 disk.dm-0.in‚ 22.7109375 disk.drbd2165.out‚0 disk.sda2.in‚0C@óLŸ¦ ‚(:” ÃÅèÒ‘–Åö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚263.941' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚52.788199999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0›m½Ùå§ ‚(:Ú ÃÅèÒýÁÅù disk.dm-1.out‚ 74.01171875 disk.sda3.out‚ 74.01171875 disk.drbd2165.out‚0# disk./.usageGB‚560.473217010498( disk.nvme0c0n1.out‚27287.68896484375 disk.rbd0.out‚0 disk.dm-0.in‚ 22.7109375& proc.memory.availableMB‚ 973.703125 disk.sda3.in‚0 network.recv‚597352 disk.rbd3.out‚0& disk.nvme0n1.out‚27287.68896484375 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.sda1.in‚0 network.sent‚ 2.446862e+06 disk.rbd3.in‚0 disk.nvme0c0n1.in‚ 22.7109375 disk.sda2.in‚0 disk.rbd1.out‚ 5.4765625 disk.sda.out‚ 74.01171875 proc.cpu.threads‚17 disk.nvme0n1.in‚ 22.7109375 disk.rbd1.in‚0 disk.sda.in‚0$ memory_percent‚94.05698776245117! proc.memory.rssMB‚ 2843.609375) disk./.usagePercent‚37.61382786192666 disk.sda2.out‚0# disk.dm-0.out‚27287.68896484375 disk.sda1.out‚0 cpu‚25.61616529884152 disk.drbd2165.in‚0* proc.memory.percent‚17.356014251708984êâ·až¨ ‚(:“ ÒÅèҩˈ÷ gpu.0.gpu‚91.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚335.114' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚67.02279999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 }ç© ‚(:Ü ÒÅèÒ¶›¼ù& disk.nvme0n1.out‚27297.88720703125 disk.rbd1.in‚0 disk.rbd1.out‚ 5.4765625 disk.sda.in‚0 disk.rbd0.out‚0# disk.dm-0.out‚27297.88720703125 disk.sda1.in‚0 disk.sda3.in‚0 disk.drbd2165.in‚0* proc.memory.percent‚17.362117767333984 proc.cpu.threads‚17 disk.sda2.in‚0* disk./.usagePercent‚37.613955353678705( disk.nvme0c0n1.out‚27297.88720703125! proc.memory.rssMB‚ 2844.609375 disk.sda3.out‚ 74.28515625 disk.nvme0c0n1.in‚ 23.0078125$ memory_percent‚94.06375885009766$ disk./.usageGB‚560.4751167297363 disk.drbd2165.out‚0 disk.dm-1.in‚0% proc.memory.availableMB‚ 972.59375 cpu‚25.614841960330786 network.recv‚600698 disk.sda.out‚ 74.28515625 disk.dm-1.out‚ 74.28515625 disk.rbd0.in‚0 disk.rbd3.in‚0 disk.sda2.out‚0 network.sent‚ 2.449826e+06 disk.dm-0.in‚ 23.0078125 disk.sda1.out‚0 disk.rbd3.out‚0 disk.nvme0n1.in‚ 23.0078125¸´˜ÿ2ª - _wandbruntime‚2242  _runtime‚2242-²Ç« Á _runtime‚2242.328188682 _step‚107 & lr-AdamW/pg1‚0.0006878542182255756 & lr-AdamW/pg2‚0.0006878542182255756 trainer/global_step‚1349 % _timestamp‚1.7842920581110168e+09´·Ë¬ Å & lr-AdamW/pg1‚0.0006878542182255756 & lr-AdamW/pg2‚0.0006878542182255756 trainer/global_step‚1349 % _timestamp‚1.7842920581110168e+09 _runtime‚2242.328188682 _step‚107kCuòå2­ - _wandbruntime‚2242  _runtime‚2242N9+'­ ® æO D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚11478483 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚386872406024192 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚106578973010944 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚386704641167360 _step‚108 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚230 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3464003 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚7290283 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 $ _timestamp‚1.784292058112156e+09 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚386872406024192 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚387971169995406 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1266528828046 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚388141130671616 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 4®ÿÿ?9071259648 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚11668535 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚23156997 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚387969799967116 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1267353084416 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚11478489 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3464008 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3826275 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚104943106560000 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 trainer/global_step‚1349 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1268698473984 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚235 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 400530432 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚386872432197632 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚388139759108608 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚388139759108608 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚7290048 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1265184227724 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚11478483 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚11478489 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚386872432197632 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0ZÆÙãð'¯ êO A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚388141130671616 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚11668535 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚23156997 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚387969799967116 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1267353084416 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚11478489 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3464008 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3826275 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚104943106560000 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 trainer/global_step‚1349 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1268698473984 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚235 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 400530432 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚386872432197632 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚388139759108608 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚388139759108608 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚7290048 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1265184227724 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚11478483 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚11478489 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚386872432197632 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚11478483 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚386872406024192 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚106578973010944 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚386704641167360 _step‚108 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚230 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3464003 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚7290283 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 $ _timestamp‚1.784292058112156e+09 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚386872406024192 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚387971169995406 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1266528828046 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0llVa2° - _wandbruntime‚2243  _runtime‚2243öD¥)ö&± ðM REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚11486695 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚11486689 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1269623516672 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7295253 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚106657660662784 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚386982497517568 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1269623516672 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3466551 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚32432128 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚387150406754304 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚388248605211538 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚7295479 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚387150432927744 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚105020224876544 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚11677013 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚388249975239940 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3466546 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1637468218368 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚11687000 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1267452294404 _step‚109 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚388418684874240 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚221 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 400523264 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚386982522945536 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1268278119936 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3828707 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚23173695 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1266107693970 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1268278119936 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚ 2243.71581282 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1637435786240 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚387150432927744 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚105020592967680 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚23173695 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚226 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 trainer/global_step‚1349 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚388420056444416 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚23163702 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 % _timestamp‚1.7842920594981863e+09[- ôM J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚11486689 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1269623516672 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7295253 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚106657660662784 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚386982497517568 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1269623516672 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3466551 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚32432128 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚387150406754304 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚388248605211538 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚7295479 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚387150432927744 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚105020224876544 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚11677013 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚388249975239940 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3466546 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1637468218368 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚11687000 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1267452294404 _step‚109 ?6Device?ž%tStatsMonitor.on_train_batch_end/num_device_alloc‚1995 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚388418684874240 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚221 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 400523264 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚386982522945536 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1268278119936 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3828707 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚23173695 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1266107693970 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1268278119936 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚ 2243.71581282 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1637435786240 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚387150432927744 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚105020592967680 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚23173695 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚226 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 trainer/global_step‚1349 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚388420056444416 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚23163702 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 % _timestamp‚1.7842920594981863e+09 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚11486695 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175m!ø[2³ - _wandbruntime‚2243  _runtime‚2243±r½T´ ÿ 1loss/SRqqt1/assignment_loss‚1.078167055049562 1loss/SRqqt2/assignment_loss‚0.859153147071095 /loss/FRt1/detection_loss‚0.5919390516444469 0loss/SRqqt1/detection_loss‚1.241383426473476 trainer/global_step‚1349 % _timestamp‚1.7842920595847611e+09 0loss/FRt1/assignment_loss‚2.7517210833231607 /loss/FRt2/detection_loss‚1.6517493375907364 1loss/FBt1/assignment_loss‚0.13625036495302056 1loss/FBt2/assignment_loss‚0.13164138337203357 &loss/total_loss‚12.988944732858842 epoch‚3 _runtime‚2243.801722623 _step‚110 0loss/FRt2/assignment_loss‚1.4034770124291391 1loss/SRqqt2/detection_loss‚0.8052751035917827 /loss/FBt1/detection_loss‚0.6093719778803952 /loss/FBt2/detection_loss‚1.7288157894799951ÉS µ ƒ 1loss/FBt1/assignment_loss‚0.13625036495302056 1loss/FBt2/assignment_loss‚0.13164138337203357 &loss/total_loss‚12.988944732858842 epoch‚3 _runtime‚2243.801722623 _step‚110 0loss/FRt2/assignment_loss‚1.4034770124291391 1loss/SRqqt2/detection_loss‚0.8052751035917827 /loss/FBt1/detection_loss‚0.6093719778803952 /loss/FBt2/detection_loss‚1.7288157894799951 1loss/SRqqt1/assignment_loss‚1.078167055049562 1loss/SRqqt2/assignment_loss‚0.859153147071095 /loss/FRt1/detection_loss‚0.5919390516444469 0loss/SRqqt1/detection_loss‚1.241383426473476 trainer/global_step‚1349 % _timestamp‚1.7842920595847611e+09 0loss/FRt1/assignment_loss‚2.7517210833231607 /loss/FRt2/detection_loss‚1.6517493375907364nÈáø럶 ‚(:” áÅèÒõýöö gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚257.462' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚51.492399999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚00ûÆá· ‚(:Ö áÅèÒ‡á—ú$ memory_percent‚94.06232833862305& proc.memory.availableMB‚ 972.828125) disk./.usagePercent‚37.61446557669445 disk.nvme0n1.in‚ 23.31640625 disk.sda.out‚ 74.9453125 disk.drbd2165.in‚0! disk.nvme0c0n1.in‚ 23.31640625 disk.sda.in‚0 disk.dm-0.in‚ 23.31640625! disk.dm-0.out‚27309.173828125 disk.sda1.out‚0* proc.memory.percent‚17.362117767333984 cpu‚25.613077564761916 proc.cpu.threads‚17$ disk./.usageGB‚560.4827194213867$ disk.nvme0n1.out‚27309.173828125 disk.rbd1.out‚ 5.5390625 network.recv‚604456 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd3.in‚0 disk.rbd0.out‚0! proc.memory.rssMB‚ 2844.609375& disk.nvme0c0n1.out‚27309.173828125 disk.rbd1.in‚0 disk.dm-1.out‚ 74.9453125 network.sent‚ 2.455015e+06 disk.rbd3.out‚0 disk.dm-1.in‚0 disk.sda1.in‚0 disk.drbd2165.out‚0 disk.rbd0.in‚0 disk.sda3.in‚0 disk.sda3.out‚ 74.9453125ýVI韸 ‚(:” ðÅèÒ¤žÏö gpu.0.gpu‚91.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚241.077' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚48.215399999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0³e“æ¹ ‚(:Û ðÅèÒ“÷¥ø disk.sda1.in‚0 disk.sda1.out‚0 disk.sda3.out‚ 75.1640625 disk.rbd3.out‚0* proc.memory.percent‚17.362117767333984 cpu‚25.611725000194845 disk.dm-1.out‚ 75.1640625 disk.rbd0.in‚0 disk.rbd0.out‚0 proc.cpu.threads‚17 disk.dm-1.in‚0 disk.sda3.in‚0 disk.drbd2165.in‚0$ memory_percent‚94.06237602233887$ disk./.usageGB‚560.4861526489258' disk.nvme0c0n1.out‚27314.9248046875 disk.nvme0n1.in‚ 23.31640625 network.sent‚ 2.518072e+06* disk./.usagePercent‚37.614695983475265! proc.memory.rssMB‚ 2844.609375 disk.sda2.out‚0 disk.rbd1.out‚ 5.5390625 disk.dm-0.in‚ 23.31640625! disk.nvme0c0n1.in‚ 23.31640625 disk.sda2.in‚0 disk.rbd3.in‚0% disk.nvme0n1.out‚27314.9248046875' proc.memory.availableMB‚ 972.8203125 disk.rbd1.in‚0 disk.sda.out‚ 75.1640625" disk.dm-0.out‚27314.9248046875 disk.drbd2165.out‚0 network.recv‚608597 disk.sda.in‚0›Ì^)žº ‚(:“ ÿÅèÒÑÌö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚268.527' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚53.70539999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ñedÙê» ‚(:ß ÿÅèÒ¥½’ù) disk./.usagePercent‚37.61513631643416 disk.rbd3.out‚0 disk.sda.out‚ 75.4921875# disk.dm-0.out‚27324.65966796875* proc.memory.percent‚17.359375953674316 disk.rbd1.in‚0 disk.sda2.in‚0 disk.rbd1.out‚ 5.6015625 disk.sda1.out‚0 disk.sda1.in‚0 network.sent‚ 2.523098e+06$ disk./.usageGB‚560.4927139282227& disk.nvme0n1.out‚27324.65966796875 disk.sda2.out‚0 cpu‚25.610109497296612 disk.sda3.out‚ 75.4921875 network.recv‚612270( disk.nvme0c0n1.out‚27324.65966796875 disk.rbd0.out‚0$ memory_percent‚94.05655860900879# proc.memory.rssMB‚ 2844.16015625 disk.sda.in‚0 disk.dm-0.in‚ 23.61328125 proc.cpu.threads‚17 disk.drbd2165.in‚0 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.sda3.in‚0 disk.drbd2165.out‚0 disk.rbd3.in‚0! disk.nvme0c0n1.in‚ 23.61328125 disk.nvme0n1.in‚ 23.61328125 disk.dm-1.out‚ 75.4921875' proc.memory.availableMB‚ 973.7734375fÒ™Ÿ¼ ‚(:” ŽÆèÒÓŒôö gpu.0.gpu‚89.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚295.399' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚59.079800000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0×ÝPŠã½ ‚(:Ø ŽÆèÒç™Ôø) disk./.usagePercent‚37.61536057903416 disk.rbd1.out‚ 5.6015625 disk.sda.out‚ 75.7109375* proc.memory.percent‚17.362117767333984 disk.dm-1.in‚0 disk.rbd0.out‚0 disk.dm-0.in‚ 23.921875 disk.drbd2165.in‚0 disk.sda2.in‚0 disk.rbd1.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda3.in‚0 disk.drbd2165.out‚0 disk.sda2.out‚0( proc.memory.availableMB‚ 973.01953125! proc.memory.rssMB‚ 2844.609375 proc.cpu.threads‚17 network.sent‚ 2.528085e+06 network.recv‚615611 disk.rbd3.in‚0$ memory_percent‚94.06116008758545$ disk./.usageGB‚560.4960556030273 disk.nvme0c0n1.in‚ 23.921875 disk.nvme0n1.in‚ 23.921875& disk.nvme0n1.out‚27336.89697265625 disk.dm-1.out‚ 75.7109375 disk.sda3.out‚ 75.7109375( disk.nvme0c0n1.out‚27336.89697265625 disk.rbd0.in‚0 disk.sda.in‚0 cpu‚25.608760969141755# disk.dm-0.out‚27336.89697265625 disk.rbd3.out‚0éþ唾 ‚(:‰ ÆèÒåþÙö gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚40.0 gpu.0.powerWatts‚250.216' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚50.0432 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Bæ¿ ‚(:Û ÆèÒÌø disk.drbd2165.in‚0' disk.nvme0c0n1.out‚27346.4599609375 disk.nvme0n1.in‚ 55.0703125* proc.memory.percent‚17.362117767333984 network.recv‚619441 disk.drbd2165.out‚0 disk.nvme0c0n1.in‚ 55.0703125 disk.rbd1.out‚ 5.7265625$ disk./.usageGB‚560.5026473999023 disk.sda1.out‚0 disk.rbd3.out‚0( proc.memory.availableMB‚ 973.24609375 disk.sda.out‚ 76.13671875 disk.rbd0.in‚0 disk.dm-0.in‚ 55.0703125" disk.dm-0.out‚27346.4599609375$ memory_percent‚94.05977725982666 disk.dm-1.in‚0 disk.rbd0.out‚0% disk.nvme0n1.out‚27346.4599609375 cpu‚25.607313204208857 proc.cpu.threads‚17 disk.dm-1.out‚ 76.13671875 disk.sda1.in‚0 disk.rbd3.in‚0 disk.sda2.out‚0 disk.rbd1.in‚0 network.sent‚ 2.533118e+06 disk.sda.in‚0 disk.sda3.in‚0 disk.sda3.out‚ 76.13671875 disk.sda2.in‚0! proc.memory.rssMB‚ 2844.609375) disk./.usagePercent‚37.61580296005332Šôñþ2À - _wandbruntime‚2321  _runtime‚2321ãÅBÇÁ Á _step‚111 & lr-AdamW/pg1‚0.0006874231651168817 & lr-AdamW/pg2‚0.0006874231651168817 trainer/global_step‚1399 % _timestamp‚1.7842921369041903e+09 _runtime‚2321.121321512'dË Å _step‚111 & lr-AdamW/pg1‚0.0006874231651168817 & lr-AdamW/pg2‚0.0006874231651168817 trainer/global_step‚1399 % _timestamp‚1.7842921369041903e+09 _runtime‚2321.121321512oþƒh´2à - _wandbruntime‚2321  _runtime‚2321}q5¾í'Ä çO A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚266 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚110513397756928 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚400772442529792 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚11888783 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚11888789 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚7554414 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3591158 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚110513800384512 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚12094287 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚402087483107840 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚402086111544832 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1311421340832 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚12104260 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3962995 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚11888783 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1315014404608 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚108817426944000 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1696338904064 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚401911496017314 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1313669015040 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚12094287 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚400598704648192 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚400772468703232 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚402086111544832 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚11888789 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1312765941154 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚23983070 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚402087483107840 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚261 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1315014404608 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚400772468703232 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚108817058852864 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3591153 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚12104260 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚400772442529792 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚23993049 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 % _timestamp‚1.7842921369052098e+09 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚402087483107840 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚402086111544832 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1311421340832 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚12104260 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3962995 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚11888783 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1315014404608 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚108817426944000 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1696338904064 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚401911496017314 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1313669015040 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚12094287 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚400598704648192 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚400772468703232 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚402086111544832 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚11888789 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1312765941154 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚23983070 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚402087483107840 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚261 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1315014404608 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚400772468703232 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚108817058852864 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3591153 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚12104260 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚400772442529792 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚23993049 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.curre™÷Ú¾nt‚5 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 % _timestamp‚1.7842921369052098e+09 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 IDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚12112752 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚11896995 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚248 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚1399 _runtime‚2322.508187537 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚402366408880640 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚11896989 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚7559706 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚12112752 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚402190301261848 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚253 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7559453 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1315939447296 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚108894545260544 _step‚113 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1313689407512 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 % _timestamp‚1.7842921382907603e+09 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚23999754 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚401050469433344 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚24009747 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚400876611854336 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1312344807078 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚401050443259904 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1697571245056 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚402188931233446 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3966005 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1697536715776 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚402366408880640 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚12102765 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3593696 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚401050443259904 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3965757 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚110592081976320 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚12102765 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚402365037310464 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚110592484596736 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚401050469433344?Ãðôû&È õM KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚12112752 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚11896995 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚248 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚1399 _runtime‚2322.508187537 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚402366408880640 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚11896989 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚7559706 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚12112752 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚402190301261848 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚253 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7559453 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1315939447296 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚108894545260544 _step‚113 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1313689407512 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 % _timestamp‚1.7842921382907603e+09 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚23999754 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚401050469433344 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚24009747 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚400876611854336 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1312344807078 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚401050443259904 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1697571245056 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚402188931233446 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3966005 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1697536715776 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚402366408880640 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚12102765 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3593696 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚401050443259904 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3965757 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚110592081976320 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚12102765 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚402365037310464 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚110592484596736 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚401050469433344 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚400876586426368 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚23999754 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1315939447296 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0q÷ªîæ2É - _wandbruntime‚2322  _runtime‚2322©ü@PÊ € _runtime‚2322.594467033 0loss/FRt1/assignment_loss‚2.4911886501895957 2loss/SRqqt1/assignment_loss‚0.9190145915421826 /loss/FRt1/detection_loss‚0.6204705127226491 0loss/FRt2/assignment_loss‚1.4809964404556422 0loss/FBt2/assignment_loss‚0.0733580968324404 1loss/SRqqt2/detection_loss‚0.7771536163532462 /loss/FBt1/detection_loss‚0.5893267113403948 .loss/FBt2/detection_loss‚0.892765622021574 % _timestamp‚1.7842921383774989e+09 epoch‚3 trainer/global_step‚1399 _step‚114 2loss/SRqqt2/assignment_loss‚0.8469439328234131 1loss/FBt1/assignment_loss‚0.07084734798857895 /loss/FRt2/detection_loss‚1.8056629750399555 1loss/SRqqt1/detection_loss‚1.0350237010472167 &loss/total_loss‚11.602752198356889餄 Ë „ 2loss/SRqqt2/assignment_loss‚0.8469439328234131 1loss/FBt1/assignment_loss‚0.07084734798857895 /loss/FRt2/detection_loss‚1.8056629750399555 1loss/SRqqt1/detection_loss‚1.0350237010472167 &loss/total_loss‚11.602752198356889 _runtime‚2322.594467033 0loss/FRt1/assignment_loss‚2.4911886501895957 2loss/SRqqt1/assignment_loss‚0.9190145915421826 /loss/FRt1/detection_loss‚0.6204705127226491 0loss/FRt2/assignment_loss‚1.4809964404556422 0loss/FBt2/assignment_loss‚0.0733580968324404 1loss/SRqqt2/detection_loss‚0.7771536163532462 /loss/FBt1/detection_loss‚0.5893267113403948 .loss/FBt2/detection_loss‚0.892765622021574 % _timestamp‚1.7842921383774989e+09 epoch‚3 trainer/global_step‚1399 _step‚114räèñ<”Ì ‚(:‰ ¬ÆèÒÃà¢÷ gpu.0.gpu‚90.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚266.646' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.3292 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ß(æÍ ‚(:Û ¬ÆèÒ®€Êø disk.rbd1.out‚ 5.7265625 disk.rbd3.in‚0! proc.memory.rssMB‚ 2844.609375* proc.memory.percent‚17.362117767333984 cpu‚25.605882911540792 proc.cpu.threads‚17 disk.dm-1.in‚0 disk.sda3.out‚ 76.33984375! disk.nvme0c0n1.in‚ 55.37890625$ memory_percent‚94.06242370605469 disk.sda.in‚0" disk.dm-0.out‚27358.2451171875 network.sent‚ 2.538103e+06 disk.sda1.out‚0 disk.sda3.in‚0 disk.sda2.out‚0$ proc.memory.availableMB‚972.8125$ disk./.usageGB‚560.5096664428711 disk.rbd0.out‚0 disk.drbd2165.out‚0 disk.nvme0n1.in‚ 55.37890625 disk.rbd1.in‚0 disk.dm-0.in‚ 55.37890625 disk.drbd2165.in‚0' disk.nvme0c0n1.out‚27358.2451171875 disk.rbd0.in‚0 disk.sda1.in‚0 disk.sda2.in‚0 network.recv‚622790* disk./.usagePercent‚37.616274013916325 disk.sda.out‚ 76.33984375 disk.dm-1.out‚ 76.33984375 disk.rbd3.out‚0% disk.nvme0n1.out‚27358.2451171875~Î1ŸÎ ‚(:” »ÆèÒ™ßòö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚38.0 gpu.0.powerWatts‚110.286' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.057199999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0«n…@æÏ ‚(:Û »ÆèÒ½Çüù disk.rbd0.in‚0 proc.cpu.threads‚17 disk.rbd1.in‚0$ memory_percent‚94.06239986419678 cpu‚25.604439349676074) disk./.usagePercent‚37.61676810845741# disk.dm-0.out‚27368.30126953125 disk.sda1.in‚0( proc.memory.availableMB‚ 972.81640625 disk.sda3.out‚ 76.703125! disk.nvme0c0n1.in‚ 55.67578125( disk.nvme0c0n1.out‚27368.30126953125 disk.sda.in‚0 network.recv‚627536 disk.sda3.in‚0 disk.drbd2165.in‚0& disk.nvme0n1.out‚27368.30126953125 disk.sda.out‚ 76.703125 disk.dm-1.in‚0 disk.rbd0.out‚0 disk.sda2.out‚0 disk.rbd1.out‚ 5.7890625 disk.dm-1.out‚ 76.703125 disk.drbd2165.out‚0 disk.rbd3.in‚0 disk.nvme0n1.in‚ 55.67578125 disk.sda2.in‚0* proc.memory.percent‚17.362117767333984 network.sent‚ 2.601152e+06 disk.dm-0.in‚ 55.67578125 disk.sda1.out‚0 disk.rbd3.out‚0! proc.memory.rssMB‚ 2844.609375$ disk./.usageGB‚560.5170288085938‰v¦’Ð ‚(:‡ ÊÆèÒ‚àÛö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚276.07' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.214 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0l6? æÑ ‚(:Û ÊÆèÒø¥›ù network.sent‚ 2.606273e+06 disk.sda3.out‚ 76.82421875$ disk.nvme0n1.out‚27374.357421875 disk.sda2.in‚0( proc.memory.availableMB‚ 972.81640625$ disk./.usageGB‚560.5201988220215 disk.dm-0.in‚ 55.67578125 disk.drbd2165.out‚0 disk.rbd3.out‚0& disk.nvme0c0n1.out‚27374.357421875 disk.sda2.out‚0) disk./.usagePercent‚37.61698085071836 disk.sda3.in‚0 disk.drbd2165.in‚0 disk.rbd3.in‚0! proc.memory.rssMB‚ 2844.609375 proc.cpu.threads‚17 disk.dm-1.in‚0 disk.dm-1.out‚ 76.82421875 disk.rbd0.in‚0 disk.sda1.out‚0 disk.sda.in‚0 disk.sda.out‚ 76.82421875! disk.nvme0c0n1.in‚ 55.67578125 network.recv‚630999 disk.rbd0.out‚0 disk.sda1.in‚0 disk.nvme0n1.in‚ 55.67578125* proc.memory.percent‚17.362117767333984 cpu‚25.603061265538713 disk.rbd1.out‚ 5.7890625$ memory_percent‚94.06239986419678! disk.dm-0.out‚27374.357421875 disk.rbd1.in‚0ÓáÜè”Ò ‚(:‰ ÙÆèÒ¼ú÷ gpu.0.gpu‚93.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚287.327' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚57.4654 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0M™…æÓ ‚(:Û ÙÆèÒñõ€ù( disk.nvme0c0n1.out‚27383.78662109375 disk.dm-1.in‚0 disk.dm-0.in‚ 393.484375! proc.memory.rssMB‚ 2844.609375$ disk./.usageGB‚560.5239295959473 disk.rbd1.out‚ 5.8515625 disk.dm-1.out‚ 77.20703125$ memory_percent‚94.06099319458008& proc.memory.availableMB‚ 973.046875* proc.memory.percent‚17.362117767333984 proc.cpu.threads‚17 disk.rbd3.out‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd1.in‚0 disk.rbd0.in‚0 disk.drbd2165.in‚0 disk.rbd3.in‚0 disk.nvme0n1.in‚ 393.484375& disk.nvme0n1.out‚27383.78662109375 disk.sda1.out‚0 network.recv‚634470# disk.dm-0.out‚27383.78662109375 disk.sda1.in‚0 disk.sda3.in‚0 disk.sda3.out‚ 77.20703125 disk.nvme0c0n1.in‚ 393.484375 disk.sda.in‚0 disk.sda.out‚ 77.20703125 cpu‚25.60189755043556 network.sent‚ 2.611262e+06) disk./.usagePercent‚37.61723122608685 disk.drbd2165.out‚0 disk.rbd0.out‚0ŒW g”Ô ‚(:‰ èÆèÒÖ݆÷ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚261.766' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.3532 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 Ñ2ñÕ ‚(:æ èÆèÒåÕªù disk.rbd1.in‚0$ disk./.usageGB‚560.5304069519043 disk.sda.out‚ 77.89453125 disk.rbd0.out‚0# disk.dm-0.out‚27395.35205078125* proc.memory.percent‚17.362117767333984! disk.nvme0n1.in‚ 1404.09765625 disk.sda2.out‚0) disk./.usagePercent‚37.61766592687999 disk.sda.in‚0 disk.dm-0.in‚ 1404.09765625 disk.rbd3.out‚0( disk.nvme0c0n1.out‚27395.35205078125 proc.cpu.threads‚17 disk.sda1.out‚0 disk.sda3.in‚0& disk.nvme0n1.out‚27395.35205078125! proc.memory.rssMB‚ 2844.609375 cpu‚25.600425135633053 disk.sda2.in‚0 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.sda1.in‚0 disk.sda3.out‚ 77.89453125$ memory_percent‚94.06249523162842 disk.rbd1.out‚ 5.8515625 network.sent‚ 2.61628e+06 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.rbd3.in‚0# disk.nvme0c0n1.in‚ 1404.09765625 network.recv‚637705 disk.dm-1.out‚ 77.89453125( proc.memory.availableMB‚ 972.80078125ÁÊdÝ2Ö - _wandbruntime‚2399  _runtime‚2399]/ì‰Ç× Á & lr-AdamW/pg1‚0.0006869701781694674 & lr-AdamW/pg2‚0.0006869701781694674 trainer/global_step‚1449 % _timestamp‚1.7842922155184968e+09 _runtime‚2399.735646977 _step‚115ÉÜŸŠËØ Å _runtime‚2399.735646977 _step‚115 & lr-AdamW/pg1‚0.0006869701781694674 & lr-AdamW/pg2‚0.0006869701781694674 trainer/global_step‚1449 % _timestamp‚1.7842922155184968e+09s$féf2Ù - _wandbruntime‚2399  _runtime‚2399àE ÷Ú çO \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36633600 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1756222649344 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚12530012 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 % _timestamp‚1.7842922155196617e+09 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1359984945664 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 trainer/global_step‚1449 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4101129 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1359003054262 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚414672505208832 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚7819432 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚415851822039222 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 G>DeviceStatsMonitor.on_train»Mfö_batch_start/active.large_pool.peak‚1627 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚414672505208832 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1756259282944 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1361330335232 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚416033835544064 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚12299089 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3718303 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚414492818984960 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚414492793556992 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚12299083 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3718308 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1357658453940 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚24819122 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚269 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 _runtime‚2399.737050767 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚12530012 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚7819701 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚24829101 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚114448006610944 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚116 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚414672479035392 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚12299089 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚416032463981056 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚415850452010932 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚264 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚416032463981056 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚12299083 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚12520039 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4101393 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚12520039 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404724736 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚112691747328000 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚112691379236864 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚7819701 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚24829101 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚114448006610944 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚116 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚414672479035392 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚12299089 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚416032463981056 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚415850452010932 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚264 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚416032463981056 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚12299083 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚12520039 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4101393 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚12520039 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404724736 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚112691747328000 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚112691379236864 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚414672505208832 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1756259282944 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1361330335232 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚416033835544064 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚12299089tèN˜”Ü ‚(:‰ ÷ÆèÒ啯÷ gpu.0.gpu‚90.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚318.461' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚63.6922 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÒßœòÝ ‚(:ç ÷ÆèÒ”–»ù( disk.nvme0c0n1.out‚27405.68505859375# disk.dm-0.out‚27405.68505859375! proc.memory.rssMB‚ 2844.609375 disk.sda2.out‚0 disk.sda.in‚0 disk.sda1.in‚0 disk.sda3.in‚0 disk.sda2.in‚0 disk.drbd2165.out‚0# disk.nvme0c0n1.in‚ 1680.70703125 disk.rbd1.in‚0 disk.rbd0.in‚0* proc.memory.percent‚17.362117767333984 network.recv‚640882 disk.rbd1.out‚ 5.9140625 disk.rbd0.out‚0 disk.dm-0.in‚ 1680.70703125 disk.sda3.out‚ 78.34765625 proc.cpu.threads‚17 network.sent‚ 2.621258e+06 disk.rbd3.out‚0& disk.nvme0n1.out‚27405.68505859375$ disk./.usageGB‚560.5372657775879 disk.drbd2165.in‚0 cpu‚25.599181661605982 disk.rbd3.in‚0! disk.nvme0n1.in‚ 1680.70703125 disk.sda.out‚ 78.34765625 disk.dm-1.out‚ 78.34765625$ memory_percent‚94.06247138977051' proc.memory.availableMB‚ 972.8046875* disk./.usagePercent‚37.618126228426554 disk.dm-1.in‚0 disk.sda1.out‚0;àÿä2Þ - _wandbruntime‚2401  _runtime‚2401O²»ö&ß ðM G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚24835806 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚12307295 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚12538504 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1358581920186 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚12307295 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚114526679132672 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 _step‚117 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1360909981184 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1362255377920 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1362255377920 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404717568 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7824695 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚416312761316864 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1757408770560 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚416130627283756 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚12538504 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚414950479765504 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚416312761316864 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚114526274415104 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚112768865644544 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚255 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3720846 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36626432 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1757445396992 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚260 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1360909981184 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚12307289 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4103849 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚414950505938944 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚24845799 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚12528517 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚414770700763136 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 trainer/global_step‚1449 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚12528517 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚24835806 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968*%W: à ôM N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚24835806 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚24835806 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚12307295 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚12538504 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1358581920186 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚12307295 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚114526679132672 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 _step‚117 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1360909981184 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1362255377920 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 YIDeviceStatsMonitor.on_train_batch_end/reser;ö RÀved_bytes.large_pool.allocated‚ 47351595008 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1362255377920 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404717568 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7824695 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚416312761316864 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1757408770560 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚416130627283756 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚12538504 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚414950479765504 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚416312761316864 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚114526274415104 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚112768865644544 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚255 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3720846 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36626432 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1757445396992 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚260 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1360909981184 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚12307289 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4103849 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚414950505938944 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚24845799 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚12528517 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚414770700763136 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 trainer/global_step‚1449 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚12528517 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820u.Ã-2á - _wandbruntime‚2401  _runtime‚2401;f!¸â  .loss/FBt2/detection_loss‚1.691884060540507 &loss/total_loss‚11.989504355406115 trainer/global_step‚1449 _runtime‚2401.209440901 0loss/FRt2/assignment_loss‚1.2815101051777251 1loss/FBt1/assignment_loss‚0.03549188817518184 0loss/SRqqt2/detection_loss‚0.818811637326834 % _timestamp‚1.7842922169924986e+09 0loss/FRt1/assignment_loss‚2.5345024214571805 2loss/FBt2/assignment_loss‚0.052550053864968554 /loss/FRt1/detection_loss‚0.5730552124356926 epoch‚3 _step‚118 2loss/SRqqt2/assignment_loss‚0.8059884758353625 /loss/FBt1/detection_loss‚0.4024585043587121 2loss/SRqqt1/assignment_loss‚0.8950105812395749 /loss/FRt2/detection_loss‚1.7772482357291064 1loss/SRqqt1/detection_loss‚1.1209931792652705Q¦˜ ã … 0loss/FRt2/assignment_loss‚1.2815101051777251 1loss/FBt1/assignment_loss‚0.03549188817518184 0loss/SRqqt2/detection_loss‚0.818811637326834 % _timestamp‚1.7842922169924986e+09 0loss/FRt1/assignment_loss‚2.5345024214571805 2loss/FBt2/assignment_loss‚0.052550053864968554 /loss/FRt1/detection_loss‚0.5730552124356926 epoch‚3 _step‚118 2loss/SRqqt2/assignment_loss‚0.8059884758353625 /loss/FBt1/detection_loss‚0.4024585043587121 2loss/SRqqt1/assignment_loss‚0.8950105812395749 /loss/FRt2/detection_loss‚1.7772482357291064 1loss/SRqqt1/detection_loss‚1.1209931792652705 .loss/FBt2/detection_loss‚1.691884060540507 &loss/total_loss‚11.989504355406115 trainer/global_step‚1449 _runtime‚2401.209440901v‰Ì/ä ‚(:’ †ÇèÒÃë¸÷ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚283.03' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚56.60599999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ï†Àãå ‚(:Ø †ÇèÒ‹¸™ù% disk.nvme0n1.out‚27419.0966796875$ disk./.usageGB‚560.5437088012695 disk.sda.in‚0 disk.drbd2165.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 5.9140625! proc.memory.rssMB‚ 2844.609375 disk.sda1.out‚0 disk.dm-0.in‚ 1681.00390625 disk.sda3.in‚0' proc.memory.availableMB‚ 972.8359375 disk.dm-1.out‚78.53125" disk.dm-0.out‚27419.0966796875 disk.sda3.out‚78.53125 cpu‚25.59775714053991! disk.nvme0n1.in‚ 1681.00390625 disk.sda2.in‚0 network.sent‚ 2.6736e+06 network.recv‚645017 disk.sda1.in‚0 disk.drbd2165.in‚0 disk.rbd3.in‚0# disk.nvme0c0n1.in‚ 1681.00390625 disk.dm-1.in‚0' disk.nvme0c0n1.out‚27419.0966796875 disk.sda2.out‚0$ memory_percent‚94.06228065490723* proc.memory.percent‚17.362117767333984* disk./.usagePercent‚37.618558625151884 disk.sda.out‚78.53125 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.rbd3.out‚0ùÁï'•æ ‚(:Š •ÇèÒËæí÷ gpu.0.gpu‚100.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚339.701' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚67.9402 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ™à’ìç ‚(:á •ÇèÒ¹¿·ù proc.cpu.threads‚17 disk.rbd0.in‚0 disk.dm-0.in‚ 1681.00390625 disk.rbd3.in‚0( disk.nvme0c0n1.out‚27423.88037109375 disk.rbd1.in‚0 disk.drbd2165.in‚0# disk.nvme0c0n1.in‚ 1681.00390625 disk.sda2.in‚0 disk.sda.out‚ 78.8046875' proc.memory.availableMB‚ 973.0390625# disk.dm-0.out‚27423.88037109375 disk.sda2.out‚0 disk.sda.in‚0) disk./.usagePercent‚37.61877674357106$ disk./.usageGB‚560.5469589233398 disk.dm-1.in‚0 disk.sda1.in‚0 cpu‚25.59654314844221 network.recv‚649351 disk.sda3.out‚ 78.8046875 disk.rbd3.out‚0 disk.rbd1.out‚ 5.9765625! proc.memory.rssMB‚ 2844.609375 disk.rbd0.out‚0 disk.sda1.out‚0 disk.drbd2165.out‚0& disk.nvme0n1.out‚27423.88037109375# memory_percent‚94.0610408782959* proc.memory.percent‚17.362117767333984 network.sent‚ 2.726145e+06 disk.dm-1.out‚ 78.8046875 disk.sda3.in‚0! disk.nvme0n1.in‚ 1681.00390625æËœŸè ‚(:” ¤ÇèÒ¾²ûö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚263.954' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚52.790800000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ü¬’‘âé ‚(:× ¤ÇèҨĴù disk.dm-1.in‚0# disk.dm-0.out‚27436.94970703125* proc.memory.percent‚17.362117767333984 disk.sda.out‚ 79.1640625 disk.dm-1.out‚ 79.1640625 disk.dm-0.in‚ 1681.3125 disk.drbd2165.out‚0$ memory_percent‚94.06330585479736! proc.memory.rssMB‚ 2844.609375 network.sent‚ 2.731123e+06) disk./.usagePercent‚37.61921784455255 disk.rbd3.out‚0 disk.rbd1.in‚0 disk.sda.in‚0 disk.rbd0.out‚0 disk.rbd3.in‚0 disk.sda2.in‚0 disk.nvme0n1.in‚ 1681.3125$ disk./.usageGB‚560.5535316467285 disk.rbd0.in‚0 disk.sda1.in‚0 disk.sda3.in‚0 disk.nvme0c0n1.in‚ 1681.3125 disk.sda2.out‚0( proc.memory.availableMB‚ 972.66796875 cpu‚25.59524831774602 proc.cpu.threads‚17( disk.nvme0c0n1.out‚27436.94970703125& disk.nvme0n1.out‚27436.94970703125 disk.sda3.out‚ 79.1640625 disk.drbd2165.in‚0 disk.rbd1.out‚ 5.9765625 disk.sda1.out‚0 network.recv‚652517ôíV”ê ‚(:‰ ³ÇèÒ¥ÙÃö gpu.0.gpu‚94.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚335.642' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚67.1284 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Eçy’Üë ‚(:Ñ ³ÇèÒ“œÚù disk.sda.in‚0" disk.dm-0.out‚27447.5556640625 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.rbd3.in‚0% disk.nvme0n1.out‚27447.5556640625# memory_percent‚94.0619707107544( proc.memory.availableMB‚ 972.88671875 network.recv‚656185 disk.sda.out‚ 79.56640625 disk.dm-1.in‚0 disk.nvme0c0n1.in‚2354.25! proc.memory.rssMB‚ 2844.609375 network.sent‚ 2.738312e+06 disk.rbd1.out‚ 6.0390625 disk.rbd3.out‚0 disk.sda2.in‚0 proc.cpu.threads‚17 disk.dm-1.out‚ 79.56640625 disk.rbd0.in‚0 disk.sda2.out‚0 cpu‚25.593961235207107 disk.rbd0.out‚0 disk.dm-0.in‚2354.25) disk./.usagePercent‚37.61947359607926$ disk./.usageGB‚560.5573425292969 disk.rbd1.in‚0 disk.sda3.in‚0* proc.memory.percent‚17.362117767333984 disk.sda1.in‚0 disk.drbd2165.out‚0 disk.sda3.out‚ 79.56640625' disk.nvme0c0n1.out‚27447.5556640625 disk.nvme0n1.in‚2354.25 Ûõ·”ì ‚(:‰ ÂÇèÒÍú‚÷ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚270.414' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.0828 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚06zäjíí ‚(:â ÂÇèÒ¡í¢ù disk.sda1.out‚0 disk.sda3.in‚0 disk.rbd3.out‚0 disk.sda2.in‚0 disk.rbd1.out‚ 6.0390625 disk.rbd3.in‚0 cpu‚25.592606081204586 disk.sda1.in‚0 proc.cpu.threads‚17) disk./.usagePercent‚37.61967891412172$ disk./.usageGB‚560.5604019165039 disk.dm-1.in‚0 disk.rbd0.out‚0! disk.nvme0n1.in‚ 2354.55859375 disk.sda3.out‚ 79.6796875 disk.drbd2165.out‚0 disk.sda.in‚0 disk.sda.out‚ 79.6796875 disk.rbd0.in‚0# disk.dm-0.out‚27460.45166015625( disk.nvme0c0n1.out‚27460.45166015625& disk.nvme0n1.out‚27460.45166015625 disk.sda2.out‚0 disk.rbd1.in‚0 disk.dm-0.in‚ 2354.55859375 disk.drbd2165.in‚0# disk.nvme0c0n1.in‚ 2354.55859375$ memory_percent‚94.06347274780273& proc.memory.availableMB‚ 972.640625! proc.memory.rssMB‚ 2844.609375* proc.memory.percent‚17.362117767333984 network.sent‚ 2.741234e+06 network.recv‚659834 disk.dm-1.out‚ 79.67968752Lj12î - _wandbruntime‚2478  _runtime‚2478Á KAÆï À $ _timestamp‚1.784292294287012e+09 _runtime‚2478.504177811 _step‚119 & lr-AdamW/pg1‚0.0006864952864778379 & lr-AdamW/pg2‚0.0006864952864778379 trainer/global_step‚1499Ïê\ Êð Ä & lr-AdamW/pg1‚0.0006864952864778379 & lr-AdamW/pg2‚0.0006864952864778379 trainer/global_step‚1499 $ _timestamp‚1.784292294287012e+09 _runtime‚2478.504177811 _step‚119wª·º2ñ - _wandbruntime‚2478  _runtime‚2478êqƒFí'ò çO A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚429790778032840 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚118381491264512 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚429978816417280 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚25655174 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1406300876288 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚8083184 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1815791643648 _step‚120 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚25665153 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚12945791 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚428572541714432 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4237985 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚429792148061130 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚428386882465792 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚429980187980288 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1406300876288 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚12955764 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚118381893892096 % _timestamp‚1.7842922942881792e+09 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚8083443 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚259 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚12945791 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚254 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1815826180096 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚429978816417280 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1407646265856 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚428572515540992 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚12709389 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚12709383 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3845458 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4237731 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1405240167370 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚12955764 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872—åedó ëO F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚116566067712000 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3845453 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚12709383 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚428572515540992 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚12955764 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚118381893892096 % _timestamp‚1.7842922942881792e+09 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚8083443 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚259 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚12945791 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚254 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1815826180096 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚429978816417280 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1407646265856 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚428572515540992 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚12709389 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚12709383 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3845458 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4237731 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1405240167370 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚12955764 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚429790778032840 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 JADeviceStatsMonitor.on_train_batch_start/active.ä‰Õsmall_pool.current‚9973 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚118381491264512 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚429978816417280 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚25655174 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1406300876288 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚8083184 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1815791643648 _step‚120 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚25665153 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚12945791 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚428572541714432 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4237985 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚429792148061130 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚428386882465792 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚429980187980288 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1406300876288 IDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚428850542444544 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚428664789671936 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4240705 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚430069583277262 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1408571308544 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3848001 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚430070953305664 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _runtime‚ 2479.89140327 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 400523264 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 $ _timestamp‚1.784292295673901e+09 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚32432128 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 trainer/global_step‚1499 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚428850516271104 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1407225911808 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚118460185822208 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚118460586345472 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 _step‚121 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3847996 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚25681851 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8088460 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0E&-ù&ö óM KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚25681851 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8088460 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1406163633728 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚12717589 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚25671858 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚430257742182912 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚428850542444544 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚12717595 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1817032225792 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚428850542444544 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚428664789671936 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4240705 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚430069583277262 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1408571308544 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3848001 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚430070953305664 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _runtime‚ 2479.89140327 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 400523264 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 $ _timestamp‚1.784292295673901e+09 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚32432128 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 trainer/global_step‚1499 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚428850516271104 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1407225911808 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚118460185822208 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚118460586345472 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 _step‚121 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3847996 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2y’Rè•2÷ - _wandbruntime‚2479  _runtime‚2479f,„w ø ƒ epoch‚3 0loss/FRt1/assignment_loss‚2.4503434866666796 0loss/FRt2/assignment_loss‚1.3033579290601454 &loss/total_loss‚11.449903060688781 _runtime‚2479.977804856 _step‚122 1loss/SRqqt1/detection_loss‚1.2164518177357255 2loss/SRqqt2/assignment_loss‚0.7502793457097141 0loss/FBt1/assignment_loss‚0.0664561034732326 2loss/FBt2/assignment_loss‚0.051246202280525566 /loss/FRt2/detection_loss‚1.6098319300155717 1loss/SRqqt2/detection_loss‚0.7294101874447532 2loss/SRqqt1/assignment_loss‚0.9528763001898721 /loss/FRt1/detection_loss‚0.5656067755072749 0loss/FBt1/detection_loss‚0.45914224490243943 /loss/FBt2/detection_loss‚1.2949007377028465 trainer/global_step‚1499 % _timestamp‚1.7842922957608159e+09¶ã.G ù ‡ 1loss/SRqqt1/detection_loss‚1.2164518177357255 2loss/SRqqt2/assignment_loss‚0.7502793457097141 0loss/FBt1/assignment_loss‚0.0664561034732326 2loss/FBt2/assignment_loss‚0.051246202280525566 /loss/FRt2/detection_loss‚1.6098319300155717 1loss/SRqqt2/detection_loss‚0.7294101874447532 2loss/SRqqt1/assignment_loss‚0.9528763001898721 /loss/FRt1/detection_loss‚0.5656067755072749 0loss/FBt1/detection_loss‚0.45914224490243943 /loss/FBt2/detection_loss‚1.2949007377028465 trainer/global_step‚1499 % _timestamp‚1.7842922957608159e+09 epoch‚3 0loss/FRt1/assignment_loss‚2.4503434866666796 0loss/FRt2/assignment_loss‚1.3033579290601454 &loss/total_loss‚11.449903060688781 _runtime‚2479.977804856 _step‚122zHÊæ•ú ‚(:Š ÑÇèÒüÚÉ÷ gpu.0.gpu‚100.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚327.969' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚65.5938 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ê´éû ‚(:Þ ÑÇèÒ„„øø disk.sda.out‚ 79.87890625 disk.sda3.in‚0 disk.dm-0.in‚ 2354.85546875$ disk./.usageGB‚560.5699043273926 disk.rbd1.in‚0 disk.rbd1.out‚ 6.1015625' disk.nvme0c0n1.out‚27470.4267578125 disk.sda2.out‚0 disk.drbd2165.out‚0! disk.nvme0n1.in‚ 2354.85546875 cpu‚25.591463552879343 network.sent‚ 2.74603e+06) disk./.usagePercent‚37.62031662888951 disk.sda.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 79.87890625 disk.sda1.out‚0 disk.drbd2165.in‚0% disk.nvme0n1.out‚27470.4267578125" disk.dm-0.out‚27470.4267578125 disk.sda1.in‚0 disk.rbd3.in‚0# disk.nvme0c0n1.in‚ 2354.85546875 disk.sda2.in‚0$ memory_percent‚94.06356811523438# proc.memory.availableMB‚972.625! proc.memory.rssMB‚ 2844.609375 disk.rbd3.out‚0* proc.memory.percent‚17.362117767333984 proc.cpu.threads‚17 network.recv‚662681 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda3.out‚ 79.87890625ˆøM¾”ü ‚(:‰ àÇèÒõƒÁø gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚270.729' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.1458 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Í*‘¥îý ‚(:ã àÇèÒÍó¹ú disk.sda.out‚ 80.2109375 disk.dm-1.out‚ 80.2109375 disk.drbd2165.out‚0 network.sent‚ 2.808386e+06 proc.cpu.threads‚17 disk.drbd2165.in‚0# disk.nvme0c0n1.in‚ 2354.86328125& disk.nvme0n1.out‚27477.16162109375 disk.sda2.in‚0 disk.sda3.in‚0 disk.rbd3.in‚0 disk.rbd1.out‚ 6.1015625! proc.memory.rssMB‚ 2844.609375 disk.rbd3.out‚0! disk.nvme0n1.in‚ 2354.86328125 network.recv‚666894 disk.rbd0.in‚0 disk.dm-0.in‚ 2354.86328125# disk.dm-0.out‚27477.16162109375 cpu‚25.590097583484038* disk./.usagePercent‚37.620047052955954 disk.dm-1.in‚0 disk.sda2.out‚0# memory_percent‚94.0615177154541 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda.in‚0' proc.memory.availableMB‚ 972.9609375 disk.rbd0.out‚0 disk.sda3.out‚ 80.2109375( disk.nvme0c0n1.out‚27477.16162109375 disk.rbd1.in‚0* proc.memory.percent‚17.362117767333984$ disk./.usageGB‚560.5658874511719Íâᤞþ ‚(:“ ïÇèÒ¼þ¢÷ gpu.0.gpu‚90.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚283.847' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚56.76939999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0å&Cüßÿ ‚(:Ô ïÇèÒ¿®ù! proc.memory.rssMB‚ 2844.609375 network.recv‚670462 disk.sda3.in‚0 disk.drbd2165.in‚0 disk.rbd1.out‚ 6.1640625 disk.rbd0.out‚0 disk.sda1.out‚0 disk.rbd3.out‚0 network.sent‚ 2.813368e+06$ disk./.usageGB‚560.5771331787109 disk.rbd3.in‚0 disk.sda.in‚0 disk.dm-1.out‚ 80.578125 disk.rbd0.in‚0 disk.nvme0n1.in‚ 2388.84375% disk.nvme0n1.out‚27488.6455078125( proc.memory.availableMB‚ 972.94921875 proc.cpu.threads‚17 disk.sda.out‚ 80.578125" disk.dm-0.out‚27488.6455078125 disk.sda1.in‚0 disk.sda3.out‚ 80.578125' disk.nvme0c0n1.out‚27488.6455078125 disk.rbd1.in‚0 cpu‚25.588999580621643( disk./.usagePercent‚37.6208017631669 disk.drbd2165.out‚0 disk.nvme0c0n1.in‚ 2388.84375$ memory_percent‚94.06158924102783* proc.memory.percent‚17.362117767333984 disk.dm-1.in‚0 disk.dm-0.in‚ 2388.84375 disk.sda2.in‚0 disk.sda2.out‚0½ÛÈž€ ‚(:“ þÇèÒ…äÝö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚288.679' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚57.73579999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ôJµî ‚(:ã þÇèÒ¬Îßø$ disk./.usageGB‚560.5806083679199 disk.dm-1.in‚0! disk.nvme0n1.in‚ 2459.33203125! proc.memory.rssMB‚ 2844.609375) disk./.usagePercent‚37.62103498603059 disk.sda3.in‚0 disk.drbd2165.out‚0# disk.nvme0c0n1.in‚ 2459.33203125( disk.nvme0c0n1.out‚27502.04833984375$ memory_percent‚94.06454563140869 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.rbd1.out‚ 6.1640625 network.sent‚ 2.818353e+06 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda3.out‚ 80.7890625 disk.sda.in‚0 disk.sda.out‚ 80.7890625 disk.dm-1.out‚ 80.7890625# disk.dm-0.out‚27502.04833984375 disk.rbd1.in‚0( proc.memory.availableMB‚ 972.46484375& disk.nvme0n1.out‚27502.04833984375* proc.memory.percent‚17.362117767333984 proc.cpu.threads‚17 disk.dm-0.in‚ 2459.33203125 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 cpu‚25.58780616076836 network.recv‚673630RËøÍŸ‚ ‚(:” ÈèÒóô÷ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚283.559' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚56.711800000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 nˆ6êƒ ‚(:ß ÈèÒäÜÅù disk.dm-1.out‚ 81.0390625 disk.rbd0.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda3.in‚0 disk.sda3.out‚ 81.0390625! disk.nvme0c0n1.in‚ 2459.640625 disk.rbd3.in‚0 disk.nvme0n1.in‚ 2459.640625 disk.sda2.out‚0& disk.nvme0n1.out‚27513.41748046875* proc.memory.percent‚17.362117767333984 disk.dm-1.in‚0 disk.drbd2165.in‚0$ memory_percent‚94.06259059906006( proc.memory.availableMB‚ 972.78515625 disk.sda.in‚0 disk.sda.out‚ 81.0390625 disk.rbd0.out‚0 disk.dm-0.in‚ 2459.640625( disk.nvme0c0n1.out‚27513.41748046875 disk.sda2.in‚0 cpu‚25.586695201544867 network.recv‚677292* disk./.usagePercent‚37.621475574997014$ disk./.usageGB‚560.5871734619141 disk.rbd1.in‚0 disk.rbd1.out‚ 6.2265625 disk.rbd3.out‚0! proc.memory.rssMB‚ 2844.609375 network.sent‚ 2.823333e+06# disk.dm-0.out‚27513.41748046875 disk.drbd2165.out‚0 proc.cpu.threads‚17&Ú2„ - _wandbruntime‚2557  _runtime‚2557N~Íz… Á _runtime‚2557.304877142 _step‚123 & lr-AdamW/pg1‚0.0006859985205433991 & lr-AdamW/pg2‚0.00068599852‘é_ M05433991 trainer/global_step‚1549 % _timestamp‚1.7842923730869107e+098¹¿Ë† Å & lr-AdamW/pg1‚0.0006859985205433991 & lr-AdamW/pg2‚0.0006859985205433991 trainer/global_step‚1549 % _timestamp‚1.7842923730869107e+09 _runtime‚2557.304877142 _step‚123{ou¿2‡ - _wandbruntime‚2557  _runtime‚2557.ɬdí'ˆ çO trainer/global_step‚1549 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚442472578220032 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚442280996802560 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1451477280478 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚8348650 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1450132680156 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚442472552046592 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3972603 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚120440020004864 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚443925168853504 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1452616806912 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1453962196480 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚442280971374592 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚2557.305414306 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚442472578220032 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1452616806912 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚122316253912064 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚241 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚120440388096000 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1453962196480 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚13119683 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 _step‚124 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚13371543 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1876266346496 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚32439296 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 % _timestamp‚1.7842923730878336e+09 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚13119683 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚122316654442496 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 400530432 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚443731104054748 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚13119689 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚13381516 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4376042 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚13371543 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚8348404 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚443926540416512 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚13381516 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3972608 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚443732474083038 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚443925168853504 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚26501205 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚26491226 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚442472552046592 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 473515950082hý~ñ'‰ ëO TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 _step‚124 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚13371543 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1876266346496 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚32439296 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 % _timestamp‚1.7842923730878336e+09 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚13119683 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚122316654442496 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 400530432 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚443731104054748 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚13119689 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚13381516 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4376042 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚13371543 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚8348404 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚443926540416512 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚13381516 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3972608 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚443732474083038 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚443925168853504 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚26501205 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚26491226 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚442472552046592 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 trainer/global_step‚1549 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚442472578220032 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚442280996802560 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1451477280478 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚8348650 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1450132680156 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚442472552046592 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3972603 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚120440020004864 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚443925168853504 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1452616806912 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1453962196480 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚442280971374592 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚2557.305414306 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚442472578220032 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1452616806912 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚122316253912064 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚241 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚120440388096000 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1453962196480 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚13119683 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736|]ìÓ2Š - _wandbruntime‚2558  _runtime‚25586ì¯ö&‹ ðM UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚13380021 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 400523264 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚444011279327572 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚444009909299170 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚442750552776704 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚442558878580736 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚442558853152768 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚13127889 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚444205466189312 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1454887239168 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚26507910 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚13390008 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚122395344741376 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚442750578950144 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1452400746836 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8353679 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚232 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 $ _timestamp‚1.784292374474427e+09 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚13390008 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚26517903 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚444205466189312 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚122394944218112 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚32432128 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚13380021 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1451056146402 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚444204094619136 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚13127889 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1877470237696 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚26507910 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3975151 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3975146 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4378765 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚442750552776704 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚13127895 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4378533 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 trainer/global_step‚1549 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚13127895 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1877437805568 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175­€¿xŒ ôM E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚444205466189312 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚122394944218112 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚32432128 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚13380021 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1451056146402 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚444204094619136 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚13127889 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1877470237696 D>DÞ!­‚eviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚26507910 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3975151 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3975146 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4378765 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚442750552776704 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚13127895 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4378533 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 trainer/global_step‚1549 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚13127895 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1877437805568 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚13380021 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 400523264 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚444011279327572 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚444009909299170 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚442750552776704 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚442558878580736 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚442558853152768 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚13127889 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚444205466189312 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1454887239168 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚26507910 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚13390008 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚122395344741376 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚442750578950144 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1452400746836 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8353679 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚232 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 $ _timestamp‚1.784292374474427e+09 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚13390008 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚26517903}¦-¹2 - _wandbruntime‚2558  _runtime‚2558˜ Ž € 2loss/SRqqt1/assignment_loss‚0.8669061945536389 2loss/SRqqt2/assignment_loss‚0.7268007092823766 /loss/FRt1/detection_loss‚0.6104258168113387 0loss/SRqqt1/detection_loss‚1.205213729893932 % _timestamp‚1.7842923745617898e+09 1loss/FBt1/assignment_loss‚0.01579308051406015 1loss/SRqqt2/detection_loss‚0.7420814532673147 /loss/FRt1/assignment_loss‚2.382266994025193 /loss/FBt1/detection_loss‚0.4144932383057126 &loss/total_loss‚11.078133371678426 epoch‚3 trainer/global_step‚1549 _step‚126 0loss/FRt2/assignment_loss‚1.2020120299075876 1loss/FBt2/assignment_loss‚0.08725330158075784 /loss/FRt2/detection_loss‚1.7814819129836528 /loss/FBt2/detection_loss‚1.0434049105528613 _runtime‚2558.778801046›Àß'  „ _step‚126 0loss/FRt2/assignment_loss‚1.2020120299075876 1loss/FBt2/assignment_loss‚0.08725330158075784 /loss/FRt2/detection_loss‚1.7814819129836528 /loss/FBt2/detection_loss‚1.0434049105528613 _runtime‚2558.778801046 2loss/SRqqt1/assignment_loss‚0.8669061945536389 2loss/SRqqt2/assignment_loss‚0.7268007092823766 /loss/FRt1/detection_loss‚0.6104258168113387 0loss/SRqqt1/detection_loss‚1.205213729893932 % _timestamp‚1.7842923745617898e+09 1loss/FBt1/assignment_loss‚0.01579308051406015 1loss/SRqqt2/detection_loss‚0.7420814532673147 /loss/FRt1/assignment_loss‚2.382266994025193 /loss/FBt1/detection_loss‚0.4144932383057126 &loss/total_loss‚11.078133371678426 epoch‚3 trainer/global_step‚1549~géž ‚(:“ œÈèÒ¨Žæö gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚299.167' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚59.83339999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0#0Éã‘ ‚(:Ø œÈèÒÄÅù disk.rbd3.out‚0) disk./.usagePercent‚37.62198502999016 disk.dm-0.in‚ 2459.9375 disk.sda1.out‚0 disk.sda3.in‚0* proc.memory.percent‚17.362117767333984 proc.cpu.threads‚17 network.sent‚ 2.828475e+06 disk.sda1.in‚0 disk.rbd1.out‚ 6.2265625( proc.memory.availableMB‚ 972.48828125$ disk./.usageGB‚560.5947647094727 disk.sda.out‚ 81.1953125 disk.dm-1.out‚ 81.1953125( disk.nvme0c0n1.out‚27526.08447265625& disk.nvme0n1.out‚27526.08447265625! proc.memory.rssMB‚ 2844.609375 disk.rbd1.in‚0 network.recv‚680474 disk.rbd0.in‚0# disk.dm-0.out‚27526.08447265625 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.nvme0n1.in‚ 2459.9375$ memory_percent‚94.06440258026123 disk.dm-1.in‚0 disk.sda3.out‚ 81.1953125 disk.nvme0c0n1.in‚ 2459.9375 cpu‚25.585332305336717 disk.sda2.out‚0 disk.rbd0.out‚0 disk.rbd3.in‚0 disk.sda2.in‚0 disk.sda.in‚0ú¬Û ’’ ‚(:‡ «ÈèÒÆâöö gpu.0.gpu‚99.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚163.08' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚32.616 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0(žî÷è“ ‚(:Ý «ÈèÒ­Øø disk.sda3.in‚0* proc.memory.percent‚17.362117767333984 disk.rbd0.out‚0 disk.drbd2165.in‚0 disk.drbd2165.out‚0) disk./.usagePercent‚37.62243406720521" disk.dm-0.out‚27536.5791015625 disk.rbd3.in‚0 disk.rbd3.out‚0 network.recv‚684920$ disk./.usageGB‚560.6014556884766 disk.dm-0.in‚ 2460.24609375 disk.sda2.out‚0 proc.cpu.threads‚17 disk.sda.in‚0 disk.rbd0.in‚0 disk.sda3.out‚ 81.7265625# disk.nvme0c0n1.in‚ 2460.24609375! disk.nvme0n1.in‚ 2460.24609375! proc.memory.rssMB‚ 2844.609375% disk.nvme0n1.out‚27536.5791015625 disk.rbd1.in‚0 disk.rbd1.out‚ 6.3515625% proc.memory.availableMB‚ 972.46875 network.sent‚ 2.891539e+06 disk.dm-1.in‚0 disk.dm-1.out‚ 81.7265625 disk.sda1.in‚0' disk.nvme0c0n1.out‚27536.5791015625 disk.sda2.in‚0 cpu‚25.58427286995418 disk.sda.out‚ 81.7265625 disk.sda1.out‚0$ memory_percent‚94.06452178955078t‹Pž” ‚(:“ ºÈèÒ×Ùûö gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚288.679' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚57.73579999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0x Fë• ‚(:à ºÈèÒ¸úù network.recv‚688155$ disk./.usageGB‚560.6049919128418 disk.sda3.out‚ 81.9140625 disk.drbd2165.in‚0" disk.nvme0c0n1.in‚ 2550.0859375 disk.dm-0.in‚ 2550.0859375 disk.rbd3.in‚0( disk.nvme0c0n1.out‚27542.66650390625 cpu‚25.58301950471006& disk.nvme0n1.out‚27542.66650390625! proc.memory.rssMB‚ 2844.609375 disk.dm-1.in‚0 disk.rbd0.out‚0# disk.dm-0.out‚27542.66650390625 proc.cpu.threads‚17 network.sent‚ 2.896523e+06 disk.sda.out‚ 81.9140625 disk.sda1.out‚0 disk.nvme0n1.in‚ 2550.0859375( proc.memory.availableMB‚ 971.98046875 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd1.in‚0 disk.rbd0.in‚0$ memory_percent‚94.06750202178955* proc.memory.percent‚17.362117767333984) disk./.usagePercent‚37.62267138618945 disk.sda.in‚0 disk.dm-1.out‚ 81.9140625 disk.sda1.in‚0 disk.sda3.in‚0 disk.drbd2165.out‚0 disk.rbd1.out‚ 6.3515625 disk.rbd3.out‚0u=¦8Ÿ– ‚(:” ÉÈèÒîÌŽ÷ gpu.0.gpu‚91.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚289.503' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚57.900600000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0YÔ Lä— ‚(:Ù ÉÈèÒ·Ê™ù" disk.dm-0.out‚27552.5830078125 disk.sda2.in‚0 disk.rbd1.out‚ 6.4140625 disk.rbd0.in‚0 disk.sda1.out‚0 disk.sda2.out‚0 disk.rbd1.in‚0 cpu‚25.582068366130038 network.sent‚ 2.901543e+06 disk.sda.out‚82.28125 disk.nvme0n1.in‚ 2550.3828125 network.recv‚691798 disk.rbd0.out‚0 disk.sda3.in‚0 disk.rbd3.out‚0" disk.nvme0c0n1.in‚ 2550.3828125( proc.memory.availableMB‚ 971.98828125 disk.sda.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚82.28125 disk.sda3.out‚82.28125% disk.nvme0n1.out‚27552.5830078125* proc.memory.percent‚17.362117767333984$ disk./.usageGB‚560.6082878112793 disk.drbd2165.in‚0$ memory_percent‚94.06745433807373 proc.cpu.threads‚17* disk./.usagePercent‚37.622892576699044 disk.dm-0.in‚ 2550.3828125' disk.nvme0c0n1.out‚27552.5830078125 disk.sda1.in‚0 disk.drbd2165.out‚0 disk.rbd3.in‚0! proc.memory.rssMB‚ 2844.609375ÄYôh˜ ‚(:’ ØÈèÒ£¡ÿö gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚302.16' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚60.43200000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0`ªÏÔæ™ ‚(:Û ØÈèÒÄ„„ù# disk.nvme0c0n1.in‚ 2550.69140625 disk.sda.in‚0 cpu‚25.581040349432353 disk.sda2.out‚0 disk.sda.out‚82.34375 disk.dm-1.in‚0 disk.sda1.in‚0$ memory_percent‚94.06535625457764 network.recv‚694961 disk.rbd3.out‚0 disk.sda2.in‚0 disk.rbd0.out‚0$ disk./.usageGB‚560.6148490905762' disk.nvme0c0n1.out‚27565.6201171875! disk.nvme0n1.in‚ 2550.69140625 disk.rbd1.in‚0 disk.dm-1.out‚82.34375 disk.dm-0.in‚ 2550.69140625( proc.memory.availableMB‚ 972.33203125* proc.memory.percent‚17.362117767333984 network.sent‚ 2.906636e+06 disk.sda3.in‚0 disk.sda3.out‚82.34375 disk.rbd3.in‚0% disk.nvme0n1.out‚27565.6201171875 disk.rbd0.in‚0" disk.dm-0.out‚27565.6201171875! proc.memory.rssMB‚ 2844.609375 proc.cpu.threads‚17) disk./.usagePercent‚37.62333290965793 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.rbd1.out‚ 6.4140625iûÕÉ2š - _wandbruntime‚2635  _runtime‚2635ÙLF Ç› Á _runtime‚2635.888042909 _step‚127 & lr-AdamW/pg1‚0.0006854799122724989 & lr-AdamW/pg2‚0.0006854799122724989 trainer/global_step‚1599 % _timestamp‚1.7842924516708105e+09,ìV­Ëœ Å _runtime‚2635.888042909 _step‚127 & lr-AdamW/pg1‚0.0006854799122724989 & lr-AdamW/pg2‚0.0006854799122724989 trainer/global_step‚1599 % _timestamp‚1.7842924516708105e+09’:¿Ó2 - _wandbruntime‚2635  _runtime‚2635ÁôÔçé'ž ãO R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚457871521289728 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1498932737536 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚13807268 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚8613544 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚27337257 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚456372614725632 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚8613291 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1936345834496 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1936311298048 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 _runtime‚2635.892957346 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚13529989 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚13807268 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚13529983 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚13797295 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1497714393586 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚1599 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4099758 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚456372588552192 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚457872892852736 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚457871521289728 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚124314708480000 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4099753 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚27327278 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚457872892852736 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚456175060283392 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1500278127104 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4513538 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚126250651686912 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚457672800104946 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚456372588552192 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 _step‚128 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ! _timestamp‚1.784292451672e+09È[¸úGŸ èO KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 _step‚128 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ! _timestamp‚1.784292451672e+09 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚457871521289728 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1498932737536 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚13807268 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚8613544 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚27337257 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚456372614725632 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚8613291 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1936345834496 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1936311298048 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 _runtime‚2635.892957346 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚13529989 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚13807268 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚13529983 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚13797295 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1497714393586 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚1599 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4099758 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚456372588552192 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚457872892852736 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚457871521289728 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚124314708480000 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4099753 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚27327278 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚457872892852736 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚456175060283392 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1500278127104 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4513538 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚126250651686912 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚457672800104946 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚456372588552192 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352€v»‚û2  - _wandbruntime‚2637  _runtime‚2637Íoã÷&¡ ñM NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚13815760 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚456452967489536 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚13538195 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4102301 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚13538189 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚458151818625536 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8618577 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚124391826796544 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚456650589282304 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1499857773056 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1937515474944 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚13815760 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚13805773 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚458150447055360 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚232 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚2637.275869565 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚27353955 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1497293259510 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚13805773 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4102296 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚456650615455744 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1501203169792 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚237 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚456452942061568 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚457950235321078 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4516513 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚27353955 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚458151818625536 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚13538195 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1937550004224 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 trainer/global_step‚1599 % _timestamp‚1.7842924530580418e+09 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚126329342271488 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚13538189 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚457951605349480 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280Òi_}ü&¢ öM MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚458151818625536 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8618577 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚124391826796544 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚456650589282304 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1499857773056 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1937515474944 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚13815760 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚13805773 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚458150447055360 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚232 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚2637.275869565 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚27353955 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1497293259510 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚13805773 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4102296 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚456650615455744 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1501203169792 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚237 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚456452942061568 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚457950235321078 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4516513 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚27353955 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚458151818625536 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚13538195 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1937550004224 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 trainer/global_step‚1599 % _timestamp‚1.7842924530580418e+09 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚126329342271488 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚13538189 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚457951605349480 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚13815760 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚456452967489536 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚13538195 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4102301 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚13538189 B&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.3333333432674408 *jet/accuracy_2_of_4‚0.6775283813476562 :"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.28734859824180603 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.6955782771110535 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9788175225257874 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.2622273862361908 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.40107980370521545 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.23866350948810577 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.1859927624464035 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9514145851135254 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.3621954917907715 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.4863812029361725 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3499999940395355 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.5831475257873535 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 & Purity/2FRt1SRqqt2FBt/FRt_purity‚0 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.23076923191547394 jet/accuracy_2_of_5‚0 particle/accuracy_5_of_5‚0 epoch‚3 _step‚131 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.18700911104679108 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.021835101768374443 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.7435485124588013 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.4719010293483734 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.24774746596813202 particle/accuracy_1_of_1‚1 @&Purity/*FRþØÒ1t2SRqqt1FBt/event_proportion‚0.0009998140158131719 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 particle/accuracy_3_of_5‚1 :"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.23601286113262177 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.24864378571510315 +jet/accuracy_1_of_1‚0.18385407328605652 + Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9375 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.17624714970588684 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9887477159500122 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.45110371708869934 6 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.987500011920929 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9484919309616089 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9951985478401184 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9786732196807861 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.1447240114212036 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9888947606086731 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.23848290741443634 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9394936561584473 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9801027178764343 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.3657623529434204 :"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.43819698691368103 'particle/f_score‚0.5870311856269836 particle/accuracy_1_of_5‚1 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9795422554016113 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.01787838339805603 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.8287036418914795 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8499428629875183 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.3871569037437439 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.2530638873577118 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9900619983673096 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.015252982266247272 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.14105719327926636 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.4634474217891693 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.2747552990913391 8"Purity/1FRt0SRqqt0FBt/event_purity‚0.179247185587883 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.1502465158700943 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.6861130595207214 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.4415929913520813 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.06895604729652405 :"Purity/0FRt*SRqqt*FBt/event_purity‚0.33186230063438416 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.8499972820281982 6 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.179247185587883 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.2685185372829437 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.2309812754392624 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.3333333432674408 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6411573886871338 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.42824074625968933 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 /particle/accuracy_4_of_4‚0.9317430853843689 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.04333333298563957 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.1145833358168602 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5282539129257202 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9913737177848816 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.38580888509750366 6 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.157444566488266 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.8846153616905212 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9792823791503906 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.1337818205356598 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9783982038497925 particle/accuracy_4_of_5‚0 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.4575469195842743 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.18844068050384521 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9504629969596863 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 jet/accuracy_4_of_5‚0 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.2685185372829437 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 7validation_average_jet_accuracy‚0.23269952833652496 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.23961426317691803 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9453202486038208 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.39060384035110474 ( Purity/2FRt1SRqqt2FBt/FBt_purity‚0.5 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9794203639030457 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.6955782771110535 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.2620716691017151 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9736841917037964 ,jet/accuracy_4_of_4‚0.007130124606192112 particle/accuracy_2_of_3‚1 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.24635279178619385 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.4433732330799103 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 particle/accuracy_2_of_4‚1 -"Purity/2FRt*SRqqt2FBt/event_purity‚0.3125 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.40794527530670166 :"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.30265432596206665 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.01196297723799944 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 :"Purity/0FRt1SRqqt0FBt/event_purity‚0.22702352702617645 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.41830793023109436 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.02927616611123085 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 particle/accuracy_1_of_4‚1 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.2318439781665802 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.07031639665365219 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.47647616267204285 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.3884093463420868 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.5831475257873535 , Purity/2FRt*SRqqt2FBt/FRt_purity‚0.59375 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 jet/accuracy_1_of_5‚1 particle/accuracy_2_of_5‚1 % _timestamp‚1.7842924795662014e+09 7 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.2443147897720337 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9510226249694824 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.3571428656578064 jet/accuracy_3_of_5‚0 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9781649708747864 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5841896533966064 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9020451307296753 :"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.48514047265052795 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 /particle/accuracy_2_of_2‚0.9998962879180908 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.4869791865348816 /"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.390625 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9965986609458923 *loss/val_total_loss‚10.682893164140413 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.47036275267601013 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 8"Purity/1FRt*SRqqt*FBt/event_purity‚0.139821395277977 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9839758276939392 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.6851851940155029 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.2196783721446991 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8300291895866394 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.38633963465690613 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.7777777910232544 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.37268516421318054 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9900619983673096 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.23837478458881378 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.6851851940155029 +particle/sensitivity‚0.5671337842941284 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.3138323426246643 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.5158352255821228 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.24227099120616913 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 :"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.30170223116874695 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 particle/accuracy_1_of_3‚1 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5492375493049622 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.1322994977235794 :"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.24542205035686493 :"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.27613598108291626 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 6 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.699999988079071 :"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.22702352702617645 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.15478822588920593 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.15955324470996857 *jet/accuracy_1_of_4‚0.9481395483016968 +jet/accuracy_3_of_4‚0.06538523733615875 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9739929437637329 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.4168381989002228 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7051398158073425 /particle/accuracy_3_of_4‚0.9940924048423767 _runtime‚2663.784313461 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.4653779864311218 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 :"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.24377791583538055 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 -particle/accuracy_3_of_3‚0.99858158826828 +particle/specificity‚0.9001676440238953 trainer/global_step‚1599 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 6 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.693801999092102 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.38494306802749634 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.4699999690055847 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 8"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.624819278717041 7 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.2414495199918747 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.034212350845336914 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.17282035946846008 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.23854060471057892 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 jet/accuracy_5_of_5‚0 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5630526542663574 5 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.99090975522995 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 +jet/accuracy_2_of_2‚0.09600824117660522|Wçñ0>¬ ª| +jet/accuracy_1_of_1‚0.18385407328605652 + Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9375 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.17624714970588684 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9887477159500122 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.45110371708869934 6 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.987500011920929 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9484919309616089 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9951985478401184 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9786732196807861 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.1447240114212036 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9888947606086731 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.23848290741443634 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9394936561584473 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9801027178764343 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.3657623529434204 :"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.43819698691368103 'particle/f_score‚0.5870311856269836 particle/accuracy_1_of_5‚1 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9795422554016113 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.01787838339805603 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.8287036418914795 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8499428629875183 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.3871569037437439 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.2530638873577118 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9900619983673096 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.015252982266247272 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.14105719327926636 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.4634474217891693 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.2747552990913391 8"Purity/1FRt0SRqqt0FBt/event_purity‚0.179247185587883 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.1502465158700943 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.6861130595207214 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.4415929913520813 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.06895604729652405 :"Purity/0FRt*SRqqt*FBt/event_purity‚0.33186230063438416 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.8499972820281982 6 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.179247185587883 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.2685185372829437 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.2309812754392624 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.3333333432674408 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6411573886871338 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.42824074625968933 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 /particle/accuracy_4_of_4‚0.9317430853843689 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.04333333298563957 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.1145833358168602 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5282539129257202 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9913737177848816 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.38580888509750366 6 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.157444566488266 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.8846153616905212 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9792823791503906 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.1337818205356598 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9783982038497925 particle/accuracy_4_of_5‚0 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.4575469195842743 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.18844068050384521 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9504629969596863 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 jet/accuracy_4_of_5‚0 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.2685185372829437 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 7validation_average_jet_accuracy‚0.23269952833652496 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.23961426317691803 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9453202486038208 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.39060384035110474 ( Purity/2FRt1SRqqt2FBt/FBt_purity‚0.5 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9794203639030457 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.6955782771110535 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.2620716691017151 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9736841917037964 ,jet/accuracy_4_of_4‚0.007130124606192112 particle/accuracy_2_of_3‚1 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.24635279178619385 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.4433732330799103 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 particle/accuracy_2_of_4‚1 -"Purity/2FRt*SRqqt2FBt/event_purity‚0.3125 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.40794527530670166 :"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.30265432596206665 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.01196297723799944 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 :"Purity/0FRt1SRqqt0FBt/event_purity‚0.22702352702617645 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.41830793023109436 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.02927616611123085 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 particle/accuracy_1_of_4‚1 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.2318439781665802 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.07031639665365219 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.47647616267204285 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.3884093463420868 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.5831475257873535 , Purity/2FRt*SRqqt2FBt/FRt_purity‚0.59375 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 jet/accuracy_1_of_5‚1 particle/accuracy_2_of_5‚1 % _timestamp‚1.7842924795662014e+09 7 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.2443147897720337 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9510226249694824 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.3571428656578064 jet/accuracy_3_of_5‚0 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9781649708747864 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5841896533966064 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9020451307296753 :"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.48514047265052795 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 /particle/accuracy_2_of_2‚0.9998962879180908 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.4869791865348816 /"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.390625 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9965986609458923 *loss/val_total_loss‚10.682893164140413 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.47036275267601013 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 8"Purity/1FRt*SRqqt*FBt/event_purity‚0.139821395277977 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9839758276939392 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.6851851940155029 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.2196783721446991 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8300291895866394 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.38633963465690613 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.7777777910232544 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.37268516421318054 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9900619983673096 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.23837478458881378 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.6851851940155029 +particle/sensitivity‚0.5671337842941284 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.3138323426246643 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.5158352255821228 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.24227099120616913 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 :"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.30170223116874695 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 particle/accuracy_1_of_3‚1 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5492375493049622 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.1322994977235794 :"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.24542205035686493 :"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.27613598108291626 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 6 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.699999988079071 :"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.22702352702617645 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.15478822588920593 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.15955324470996857 *jet/accuracy_1_of_4‚0.9481395483016968 +jet/accuracy_3_of_4‚0.06538523733615875 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9739929437637329 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.4168381989002228 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7051398158073425 /particle/accuracy_3_of_4‚0.9940924048423767 _runtime‚2663.784313461 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.4653779864311218 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 :"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.24377791583538055 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 -particle/accuracy_3_of_3‚0.99858158826828 +particle/specificity‚0.9001676440238953 trainer/global_step‚1599 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 6 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.693801999092102 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.38494306802749634 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.4699999690055847 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 8"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.624819278717041 7 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.2414495199918747 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.034212350845336914 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.17282035946846008 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.23854060471057892 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 jet/accuracy_5_of_5‚0 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5630526542663574 5 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.99090975522995 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 +jet/accuracy_2_of_2‚0.09600824117660522 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.7942630052566528 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 *jet/accuracy_1_of_3‚0.8437474370002747 (particle/accuracy‚0.8286365270614624 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.6785714030265808 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.23076923191547394 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 :"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.29892268776893616 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.2406311333179474 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 7 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.9933862090110779 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.4312896132469177 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.24774746596813202 +jet/accuracy_2_of_3‚0.37637418508529663 +jet/accuracy_3_of_3‚0.05116964504122734 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.19191138446331024 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9842909574508667 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 *jet/accuracy_1_of_2‚0.5797703862190247 particle/accuracy_1_of_2‚1 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.3333333432674408 *jet/accuracy_2_of_4‚0.6775283813476562 :"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.28734859824180603 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.6955782771110535 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9788175225257874 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.2622273862361908 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.40107980370521545 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.23866350948810577 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.1859927624464035 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9514145851135254 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.3621954917907715 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.4863812029361725 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3499999940395355 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.5831475257873535 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 & Purity/2FRt1SRqqt2FBt/FRt_purity‚0 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.23076923191547394 jet/accuracy_2_of_5‚0 particle/accuracy_5_of_5‚0 epoch‚3 _step‚131 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.18700911104679108 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.021835101768374443 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.7435485124588013 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.4719010293483734 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.24774746596813202 particle/accuracy_1_of_1‚1 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 particle/accuracy_3_of_5‚1 :"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.23601286113262177 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.24864378571510315ƒ\ý4ª”­ ‚(:‰ …ÉèÒ¯ßÜ÷ gpu.0.gpu‚51.0 gpu.0.memory‚15, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚41.0 gpu.0.powerWatts‚345.344' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚69.0688 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¨cêGë® ‚(:à …ÉèÒ¸ŠËø disk.rbd1.in‚0 disk.rbd0.in‚0 disk.sda3.in‚0) disk./.usagePercent‚37.62457224213119! disk.nvme0n1.in‚ 2702.74609375 disk.sda.out‚ 83.265625 disk.rbd0.out‚0 network.recv‚706794 disk.sda2.in‚0 disk.dm-1.out‚ 83.265625 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd3.in‚0 disk.rbd3.out‚0( proc.memory.availableMB‚ 2452.3203125$ disk./.usageGB‚560.6333160400391 disk.rbd1.out‚ 6.6015625 proc.memory.rssMB‚ 2857.84375* proc.memory.percent‚17.442893981933594 cpu‚25.603365585442848 proc.cpu.threads‚17$ memory_percent‚85.03222465515137 disk.sda.in‚0 disk.dm-1.in‚0 disk.sda3.out‚ 83.265625 disk.drbd2165.out‚0 disk.sda2.out‚0# disk.dm-0.out‚28668.21923828125 disk.drbd2165.in‚0 network.sent‚ 2.979813e+06# disk.nvme0c0n1.in‚ 2702.74609375( disk.nvme0c0n1.out‚28668.21923828125 disk.dm-0.in‚ 2702.74609375& disk.nvme0n1.out‚28668.21923828125&|/Ö’¯ ‚(:‡ ”ÉèÒì¾’÷ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚251.57' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚50.314 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ʶÛî° ‚(:ã ”ÉèÒÄÆéø disk.sda1.out‚0) disk./.usagePercent‚37.62489173953392 disk.rbd1.out‚ 6.6015625 disk.dm-0.in‚ 2703.04296875 disk.sda1.in‚0# memory_percent‚85.0395679473877) proc.memory.percent‚17.44866371154785& disk.nvme0n1.out‚29914.19482421875 disk.sda2.in‚0 disk.sda3.in‚0 disk.sda3.out‚ 83.5390625! disk.nvme0n1.in‚ 2703.04296875 disk.rbd1.in‚0 cpu‚25.601957397712457 disk.drbd2165.in‚0 disk.sda.out‚ 83.5390625 disk.dm-1.in‚0 proc.cpu.threads‚17$ disk./.usageGB‚560.6380767822266( disk.nvme0c0n1.out‚29914.19482421875 disk.sda2.out‚0 disk.sda.in‚0 disk.drbd2165.out‚0 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.rbd0.in‚0 disk.rbd0.out‚0# disk.dm-0.out‚29914.19482421875 network.sent‚ 3.037539e+06 network.recv‚711146( proc.memory.availableMB‚ 2451.1171875" proc.memory.rssMB‚ 2858.7890625# disk.nvme0c0n1.in‚ 2703.04296875 disk.dm-1.out‚ 83.5390625±èL5”± ‚(:‰ £ÉèÒÏ“Öö gpu.0.gpu‚86.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚47.0 gpu.0.powerWatts‚333.913' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚66.7826 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ËWŠØ3² ‚(:ä £ÉèÒö¡…ú" proc.memory.rssMB‚ 2858.7890625 disk.sda.out‚ 85.53515625 disk.dm-1.in‚0 disk.rbd0.out‚0 disk.sda2.in‚0 network.sent‚ 3.042522e+06) disk./.usagePercent‚37.62542116311473 disk.sda.in‚0# disk.dm-0.out‚29935.25244140625 disk.drbd2165.out‚0 disk.dm-0.in‚ 2703.3515625 disk.sda3.out‚ 85.53515625 disk.drbd2165.in‚0 disk.rbd3.in‚0$ memory_percent‚85.03966331481934 network.recv‚714638" disk.nvme0c0n1.in‚ 2703.3515625) proc.memory.percent‚17.44866371154785 disk.dm-1.out‚ 85.53515625 disk.rbd0.in‚0 disk.sda1.out‚0 disk.rbd1.in‚0 disk.sda1.in‚0 disk.nvme0n1.in‚ 2703.3515625 disk.sda2.out‚0 disk.rbd1.out‚ 6.6640625 cpu‚25.600912973624528 proc.cpu.threads‚17$ disk./.usag8Ã&c¼eGB‚560.6459655761719( disk.nvme0c0n1.out‚29935.25244140625& disk.nvme0n1.out‚29935.25244140625 disk.sda3.in‚0 disk.rbd3.out‚0( proc.memory.availableMB‚ 2451.1015625þƒZ)â³ ‚(:× ²ÉèÒáÜâø disk.sda.out‚ 86.11328125 disk.drbd2165.in‚0" disk.nvme0c0n1.out‚ 35963.46875) proc.memory.percent‚17.44866371154785 cpu‚25.599837303441607 disk.dm-0.in‚ 3339.08203125 disk.rbd3.out‚0 disk.sda2.in‚0 network.recv‚718051 disk.rbd0.out‚0 disk.sda1.out‚0* disk./.usagePercent‚38.066291737829935 disk.rbd1.in‚0 disk.dm-1.in‚0 disk.sda1.in‚0 disk.sda3.in‚0 disk.rbd3.in‚0# disk.nvme0c0n1.in‚ 3339.08203125 disk.dm-0.out‚ 35963.46875 disk.sda3.out‚ 86.11328125 disk.sda.in‚0 disk.nvme0n1.out‚ 35963.46875) proc.memory.availableMB‚ 2451.01953125" proc.memory.rssMB‚ 2858.7890625 proc.cpu.threads‚17 disk.drbd2165.out‚0 network.sent‚ 3.047545e+06$ disk./.usageGB‚567.2152557373047 disk.rbd1.out‚ 6.6640625 disk.dm-1.out‚ 86.11328125 disk.rbd0.in‚0! disk.nvme0n1.in‚ 3339.08203125 disk.sda2.out‚0$ memory_percent‚85.04016399383545dþž´ ‚(:“ ²ÉèÒËàšø gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚298.74' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚59.748000000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0­eÞŸµ ‚(:” ÁÉèÒ÷óö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚287.087' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚57.417399999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ä„*ç¶ ‚(:Ü ÁÉèÒ¶ý•ù disk.sda.out‚ 86.48046875 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.sda2.in‚0) proc.memory.percent‚17.44866371154785 disk.sda.in‚0 disk.rbd0.out‚0 disk.rbd3.in‚0$ memory_percent‚85.04016399383545" proc.memory.rssMB‚ 2858.7890625$ disk./.usageGB‚567.2217826843262 disk.sda1.in‚0 disk.sda1.out‚0 disk.nvme0n1.in‚ 3355.359375) proc.memory.availableMB‚ 2451.01953125 network.sent‚ 3.052532e+06 network.recv‚721614 disk.dm-0.in‚ 3355.359375 disk.sda3.in‚0& disk.nvme0c0n1.out‚36680.529296875 proc.cpu.threads‚17) disk./.usagePercent‚38.06672976672102 disk.sda3.out‚ 86.48046875 disk.rbd3.out‚0$ disk.nvme0n1.out‚36680.529296875 cpu‚25.598740583703183 disk.dm-1.out‚ 86.48046875 disk.rbd0.in‚0! disk.nvme0c0n1.in‚ 3355.359375 disk.dm-1.in‚0! disk.dm-0.out‚36680.529296875 disk.rbd1.in‚0 disk.sda2.out‚0 disk.rbd1.out‚ 6.7265625àËkNž· ‚(:“ ÐÉèÒ莃÷ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚39.0 gpu.0.powerWatts‚206.82' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚41.364000000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0b„Då¸ ‚(:Ú ÐÉèÒ¦¤æø disk.sda.out‚ 86.9609375 disk.rbd0.in‚0 proc.cpu.threads‚17) disk./.usagePercent‚38.06694071692923# memory_percent‚85.0388765335083 disk.nvme0n1.in‚ 3355.359375 disk.sda2.in‚0 disk.sda2.out‚0) proc.memory.availableMB‚ 2451.23046875) proc.memory.percent‚17.44866371154785 disk.drbd2165.in‚0 disk.rbd3.out‚0' disk.nvme0c0n1.out‚36689.8193359375% disk.nvme0n1.out‚36689.8193359375 disk.dm-0.in‚ 3355.359375 network.sent‚ 3.057516e+06# disk./.usageGB‚567.224925994873 disk.rbd0.out‚0 disk.sda1.in‚0 disk.sda3.in‚0 disk.sda3.out‚ 86.9609375 disk.drbd2165.out‚0! disk.nvme0c0n1.in‚ 3355.359375" disk.dm-0.out‚36689.8193359375 disk.sda1.out‚0 disk.rbd1.out‚ 6.7265625 disk.rbd3.in‚0" proc.memory.rssMB‚ 2858.7890625 cpu‚25.597670903775327 disk.rbd1.in‚0 network.recv‚725020 disk.sda.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 86.9609375ÞVî‘2¹ - _wandbruntime‚2748  _runtime‚2748š´$OǺ Á & lr-AdamW/pg1‚0.0006849394949743772 & lr-AdamW/pg2‚0.0006849394949743772 trainer/global_step‚1649 % _timestamp‚1.7842925642147365e+09 _runtime‚2748.431969254 _step‚132ÇÍU Ì» Æ & lr-AdamW/pg1‚0.0006849394949743772 & lr-AdamW/pg2‚0.0006849394949743772 trainer/global_step‚1649 % _timestamp‚1.7842925642147365e+09 _runtime‚2748.431969254 _step‚132„BÏEN2¼ - _wandbruntime‚2748  _runtime‚2748¾R,Lì'½ æO `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚474100632666112 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1551091282432 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1548436037814 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚8921925 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚28354323 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚14072295 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚474100606492672 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1551091282432 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2003645690880 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚32439296 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1549780638136 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚475653069338112 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚14072295 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚475651697775104 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 trainer/global_step‚1649 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚475651697775104 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚28364302 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚8921681 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚14292007 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚14292007 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚14282034 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚130240786801664 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚473895083552768 _step‚133 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚244 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚475651697775104 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 trainer/global_step‚1649 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚475651697775104 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚28364302 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚8921681 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚14292007 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚14292007 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚14282034 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚130240786801664 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚473895083552768 _step‚133 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚244 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚14080501 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚32432128 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚475723669435438 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1552016317952 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚231 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚474172939902976 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1552016317952 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚130319479926784 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚130319079403520 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8926984 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚475722299407036 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚28381000 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚475931995110912 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚14300499 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚474378633396224 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚28381000 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4238364 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 400523264 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚474172965330944 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚14080495 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚14080495 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚14080501 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚236 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4238369 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚128314259427328 _runtime‚2749.848859663 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚1649 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚14290512 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1553361714688 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚474378607222784 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0ÿ’¡Õü&Á öM E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚14300499 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1549359504060 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚14080501 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚32432128 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚475723669435438 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1552016317952 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚231 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚474172939902976 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1552016317952 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚130319479926784 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚130319079403520 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8926984 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚475722299407036 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚28381000 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚475931995110912 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚14300499 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚474378633396224 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚28381000 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4238364 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 400523264 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚474172965330944 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚14080495 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚14080495 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚14080501 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚236 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4238369 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚128314259427328 _runtime‚2749.848859663 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚1649 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚14290512 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1553361714688 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚474378607222784 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 PDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚489598050211328 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚260 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚9187510 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚14717759 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚255 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚32439296 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚489599421774336 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1596017751244 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚134174842042368 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4824794 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚488000669171712 _runtime‚2829.649226775 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚29190375 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚488000669171712 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚14482595 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1598752602624 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚14707786^õ Ô ìO R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚489598050211328 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚260 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 L@DeviceStatsMonitor.on_train_batch_start/inactivE¸˜Cè%e_split.all.freed‚9187510 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚14717759 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚255 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚32439296 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚489599421774336 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1596017751244 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚134174842042368 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4824794 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚488000669171712 _runtime‚2829.649226775 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚29190375 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚488000669171712 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚14482595 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1598752602624 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚14707786 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2063781078016 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚9187770 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚487789147033600 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4362976 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚487789172461568 trainer/global_step‚1699 % _timestamp‚1.7842926454317427e+09 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1598752602624 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚14482589 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 400530432 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2063748638720 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚29217052 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9192794 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚488278669901824 _runtime‚2831.062977044 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1596941217602 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚138 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚132188579811328 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚14490801 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚14726251 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4827280 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚14716264 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1599677645312 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚29207059 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2064979759104 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚488278669901824 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚14490795 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚489662625428944 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚488067028811776 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1595596617168 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 % _timestamp‚1.7842926468453348e+09 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2064947326976 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚488067054239744 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4365519 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚246 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1599677645312 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚14726251 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚489878347547136 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚29217052 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚488278643728384 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4365514 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚489663995457346 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚489878347547136 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚14716264 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚241 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚488278643728384 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1598332248576 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚9193040 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4827521 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚134253527138304 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚132188947902464 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 trainer/global_step‚1699 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0¶.ãÍü&× öM RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1596941217602 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚138 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚132188579811328 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚14490801 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚14726251 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4827280 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚14716264 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1599677645312 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚29207059 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2064979759104 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚488278669901824 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚14490795 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚489662625428944 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚488067028811776 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1595596617168 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 % _timestamp‚1.7842926468453348e+09 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2064947326976 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚488067054239744 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4365519 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚246 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1599677645312 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚14726251 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚489878347547136 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚29217052 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚488278643728384 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4365514 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚489663995457346 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚489878347547136 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚14716264 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚241 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚488278643728384 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1598332248576 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚9193040 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4827521 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚134253527138304 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚132188947902464 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 trainer/global_step‚1699 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚29217052 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9192794 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚488278669901824 _runtime‚2831.062977044 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 FDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚503544402647552 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚503545774210560 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚503544402647552 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚501900705677312 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚501900679503872 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚9453436 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚503324146206430 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1645068533248 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 _step‚141 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4490121 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚30026427 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 trainer/global_step‚1749 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚271 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚138108915385344 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚501900705677312 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚501683235942400 % _timestamp‚1.7842927268177226e+09 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1645068533248 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚9453165 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2123536134144 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1640910264030 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚14892895 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚15143511 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚15133538 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4963310 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4963044 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚138109318012928 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2123501597696 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚135985413787648 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚501900679503872 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1643723143680 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚30036406 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚266 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚503544402647552 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚503545774210560 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚503544402647552 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚501900705677312 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚501900679503872 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚9453436 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚503324146206430 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1645068533248 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 _step‚141 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4490121 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚30026427 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 trainer/global_step‚1749 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚271 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚138108915385344 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚501900705677312 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚501683235942400 % _timestamp‚1.7842927268177226e+09 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1645068533248 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚9453165 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2123536134144 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1640910264030 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚14892895 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚15143511 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚15133538 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4963310 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4963044 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚138109318012928 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2123501597696 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚135985413787648 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚501900679503872 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1643723143680 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚30036406 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚266 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 IDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1643178330710 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 _runtime‚2912.451578408 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚30043111 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚9458755 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚138187589741568 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 trainer/global_step‚1749 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚502178680233984 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚14901101 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1644648179200 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚501961143148544 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚14901095 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4966086 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 $ _timestamp‚1.784292728234208e+09 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9458492 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1644648179200 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1645993575936 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚503602951450852 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚15142016 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚30053104 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MGDeviceStatsMonitorAø]4.on_train_batch_end/inactive_split.large_pool.current‚5 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚136063268286464 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚503824699983360 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚503823328413184 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1645993575936 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4492664 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4965828 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚136062900195328 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚502178706407424 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2124689546240 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚503604321479254 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚15152003 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚15142016 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚502178706407424 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚14901101 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1643178330710 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 _runtime‚2912.451578408 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚30043111 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚9458755 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚138187589741568 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 trainer/global_step‚1749 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚502178680233984 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚14901101 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1644648179200 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚501961143148544 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚14901095 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4966086 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 $ _timestamp‚1.784292728234208e+09 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9458492 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1644648179200 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1645993575936 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚503602951450852 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚15142016 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚30053104 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚136063268286464 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚503824699983360 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚503823328413184 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1645993575936 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4492664 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4965828 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚136062900195328 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚502178706407424 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2124689546240 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚503604321479254 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚15152003 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚15142016 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚502178706407424 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚14901101 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚517490755083776 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1687147377138 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚517492126646784 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5101830 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚15569263 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚515800716009472 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4617276 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚515800742182912 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚280 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2182913056768 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2182876423168 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚30862479 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚30872458 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚15303189 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚517490755083776 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚15559290 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚517492126646784 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5101830 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚15569263 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚515800716009472 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4617276 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚515800742182912 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚280 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2182913056768 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2182876423168 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚30862479 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚30872458 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚15303189 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚517490755083776 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚15559290 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚517490755083776 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1687147377138‘Ù;Ã2ƒ - _wandbruntime‚2993  _runtime‚2993á‡S÷&„ ñM aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2184088421376 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _step‚146 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚30879163 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚30889156 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚15311395 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 trainer/global_step‚1799 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚139937588670464 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚139937220579328 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9724312 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚516078716739584 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7842928093096807e+09 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1690964109824 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚15311395 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚15567768 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚15311401 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚30889156 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚15311401 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1689415443818 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚517771052419584 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4619814 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2184051794944 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820Ž}A<ü&… öM XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1690964109824 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚15311395 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚15567768 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚15311401 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚30889156 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚15311401 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1689415443818 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚517771052419584 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4619814 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2184051794944 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2184088421376 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _step‚146 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚30879163 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚30889156 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚15311395 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 trainer/global_step‚1799 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚139937588670464 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚139937220579328 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9724312 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚516078716739584 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7842928093096807e+09 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640’E—2† - _wandbruntime‚2993  _runtime‚2993>AU‡ ý /loss/FRt1/detection_loss‚0.5524074983355373 .loss/FRt2/detection_loss‚1.637204126189803 $ _timestamp‚1.784292809395988e+09 _runtime‚2993.612961388 /loss/FRt1/assignment_loss‚2.188769979747859 0loss/FRt2/assignment_loss‚1.0614820559619356 1los\{ì ïs/SRqqt2/detection_loss‚0.7974927707219795 /loss/FBt2/detection_loss‚0.6511536822869227 trainer/global_step‚1799 _step‚147 %loss/total_loss‚10.82638891011948 1loss/SRqqt1/detection_loss‚1.0962544560714071 .loss/FBt1/detection_loss‚1.244392687059713 epoch‚4 2loss/SRqqt1/assignment_loss‚0.7821169414211872 2loss/SRqqt2/assignment_loss‚0.6613241133967926 0loss/FBt1/assignment_loss‚0.0939868940171517 2loss/FBt2/assignment_loss‚0.059803704909192575ƒ… Xˆ ‚ %loss/total_loss‚10.82638891011948 1loss/SRqqt1/detection_loss‚1.0962544560714071 .loss/FBt1/detection_loss‚1.244392687059713 epoch‚4 2loss/SRqqt1/assignment_loss‚0.7821169414211872 2loss/SRqqt2/assignment_loss‚0.6613241133967926 0loss/FBt1/assignment_loss‚0.0939868940171517 2loss/FBt2/assignment_loss‚0.059803704909192575 /loss/FRt1/detection_loss‚0.5524074983355373 .loss/FRt2/detection_loss‚1.637204126189803 $ _timestamp‚1.784292809395988e+09 _runtime‚2993.612961388 /loss/FRt1/assignment_loss‚2.188769979747859 0loss/FRt2/assignment_loss‚1.0614820559619356 1loss/SRqqt2/detection_loss‚0.7974927707219795 /loss/FBt2/detection_loss‚0.6511536822869227 trainer/global_step‚1799 _step‚147“¡Ò”‰ ‚(:‰ ÏËèÒòùéö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚265.067' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.0134 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ì;PòŠ ‚(:ç ÏËèҽ˜ù disk.sda.out‚ 109.87890625 proc.cpu.threads‚17 network.recv‚785257 disk.sda1.in‚0 disk.rbd3.in‚0& disk.nvme0n1.out‚50376.06982421875* proc.memory.percent‚17.448806762695312 disk.rbd1.in‚0) disk./.usagePercent‚37.37982366323925# disk./.usageGB‚556.986385345459 disk.dm-1.out‚ 109.87890625 disk.rbd0.in‚0( disk.nvme0c0n1.out‚50376.06982421875! disk.nvme0n1.in‚ 4624.50390625 disk.sda2.in‚0 disk.sda2.out‚0 network.sent‚ 3.316248e+06 disk.rbd0.out‚0 disk.dm-0.in‚ 4624.50390625$ memory_percent‚90.96496105194092# disk.dm-0.out‚50376.06982421875 disk.drbd2165.in‚0 disk.rbd3.out‚0 proc.memory.rssMB‚ 2858.8125 cpu‚25.58156992858041 disk.sda3.in‚0# disk.nvme0c0n1.in‚ 4624.50390625) proc.memory.availableMB‚ 1480.30078125 disk.dm-1.in‚0 disk.sda1.out‚0 disk.sda3.out‚ 109.87890625 disk.drbd2165.out‚0 disk.rbd1.out‚ 7.2890625 disk.sda.in‚0iµÈž‹ ‚(:“ ÞËèÒœ‰üö gpu.0.gpu‚91.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚284.135' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚56.82699999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0yîŒ ‚(:ã ÞËèÒŒÿµù' disk.nvme0c0n1.out‚50396.2646484375 disk.dm-1.in‚0 disk.sda3.in‚0 disk.rbd1.out‚ 7.3515625$ disk./.usageGB‚556.9880256652832 disk.dm-0.in‚ 4624.50390625 disk.sda3.out‚ 111.7578125( proc.memory.availableMB‚ 1480.5859375* disk./.usagePercent‚37.379933746478976% disk.nvme0n1.out‚50396.2646484375 disk.dm-1.out‚ 111.7578125 disk.rbd0.in‚0 disk.sda1.out‚0 cpu‚25.580669519326243 disk.rbd3.in‚0 disk.sda2.out‚0 disk.sda.out‚ 111.7578125 disk.rbd0.out‚0 disk.sda1.in‚0 disk.drbd2165.in‚0 disk.drbd2165.out‚0 proc.memory.rssMB‚ 2858.8125 disk.sda2.in‚0 disk.rbd1.in‚0 proc.cpu.threads‚17 network.recv‚788811# disk.nvme0c0n1.in‚ 4624.50390625 disk.sda.in‚0" disk.dm-0.out‚50396.2646484375$ memory_percent‚90.96322059631348* proc.memory.percent‚17.448806762695312 network.sent‚ 3.379659e+06 disk.rbd3.out‚0! disk.nvme0n1.in‚ 4624.50390625i¹MÚ” ‚(:‰ íËèÒðŒüö gpu.0.gpu‚59.0 gpu.0.memory‚18, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚38.0 gpu.0.powerWatts‚160.469' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚32.0938 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0½ö*móŽ ‚(:è íËèÒײ‘ù$ memory_percent‚90.96324443817139$ disk./.usageGB‚556.9908714294434 disk.rbd1.in‚0 disk.rbd1.out‚ 7.3515625 disk.dm-0.in‚ 4624.50390625 proc.memory.rssMB‚ 2858.8125 network.sent‚ 3.386212e+06 disk.dm-1.in‚0 network.recv‚793334 disk.rbd3.in‚0 disk.sda2.in‚0 disk.rbd0.in‚0 disk.sda1.out‚0 disk.drbd2165.in‚0) proc.memory.availableMB‚ 1480.58203125 proc.cpu.threads‚17 disk.rbd0.out‚0 disk.sda.out‚ 112.76171875 disk.dm-1.out‚ 112.76171875 disk.drbd2165.out‚0 cpu‚25.57962309411918 disk.sda.in‚0 disk.sda1.in‚0! disk.nvme0n1.in‚ 4624.50390625* proc.memory.percent‚17.448806762695312) disk./.usagePercent‚37.38012472809952 disk.sda2.out‚0# disk.dm-0.out‚50406.56201171875 disk.rbd3.out‚0( disk.nvme0c0n1.out‚50406.56201171875& disk.nvme0n1.out‚50406.56201171875 disk.sda3.in‚0 disk.sda3.out‚ 112.76171875# disk.nvme0c0n1.in‚ 4624.50390625ŽÞ ˆ” ‚(:‰ üËèÒÆ”ªø gpu.0.gpu‚90.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚256.992' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚51.3984 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0dÐ`ªò ‚(:ç üËèÒÙô‡ù disk.drbd2165.in‚0 disk.drbd2165.out‚0 network.recv‚796780 disk.sda1.out‚0 disk.sda3.in‚0) proc.memory.availableMB‚ 1480.32421875( disk.nvme0c0n1.out‚50415.45654296875! disk.nvme0n1.in‚ 4773.80078125 disk.sda3.out‚ 113.34765625$ memory_percent‚90.96481800079346( disk./.usagePercent‚37.3804767384591# disk.nvme0c0n1.in‚ 4773.80078125 proc.memory.rssMB‚ 2858.8125$ disk./.usageGB‚556.9961166381836 disk.sda2.in‚0 disk.sda.out‚ 113.34765625 disk.dm-1.in‚0 disk.rbd3.in‚0 disk.rbd3.out‚0 cpu‚25.57874700155363 proc.cpu.threads‚17 disk.rbd1.out‚ 7.4140625 disk.dm-1.out‚ 113.34765625 disk.rbd0.out‚0& disk.nvme0n1.out‚50415.45654296875* proc.memory.percent‚17.448806762695312 disk.rbd0.in‚0 disk.sda1.in‚0 network.sent‚ 3.389931e+06 disk.sda2.out‚0 disk.rbd1.in‚0 disk.sda.in‚0 disk.dm-0.in‚ 4773.80078125# disk.dm-0.out‚50415.45654296875Ó7â‘ ‚(:… ‹ÌèÒÙõÇö gpu.0.gpu‚9.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚41.0 gpu.0.powerWatts‚338.59' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚67.718 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0A8î’ ‚(:ã ‹ÌèÒĨ¾ø$ disk./.usageGB‚557.0007553100586 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 network.sent‚ 3.394963e+06 disk.rbd3.in‚0# disk.nvme0n1.out‚50424.69140625) proc.memory.availableMB‚ 1480.30859375# proc.memory.rssMB‚ 2858.81640625 cpu‚25.577886631805505# disk.nvme0c0n1.in‚ 4773.80078125 disk.sda1.in‚0 disk.sda3.out‚ 113.62109375 disk.rbd1.in‚0 disk.dm-0.in‚ 4773.80078125 disk.dm-0.out‚50424.69140625 disk.sda1.out‚0! disk.nvme0n1.in‚ 4773.80078125 proc.cpu.threads‚17 disk.sda3.in‚0 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.sda.in‚0 disk.rbd3.out‚0% disk.nvme0c0n1.out‚50424.69140625# memory_percent‚90.9649133682251* proc.memory.percent‚17.448830604553223 disk.sda.out‚ 113.62109375 disk.dm-1.out‚ 113.62109375 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd1.out‚ 7.4140625 network.recv‚800603) disk./.usagePercent‚37.38078804362073¢2“ - _wandbruntime‚3070  _runtime‚3070Ó†êÅ” ¿ % lr-AdamW/pg1‚0.000682560450673391 % lr-AdamW/pg2‚0.000682560450673391 trainer/global_step‚1849 % _timestamp‚1.7842928866660132e+09 _runtime‚3070.883201406 _step‚148öüµÈÊ• Ä % lr-AdamW/pg1‚0.000682560450673391 % lr-AdamW/pg2‚0.000682560450673391 trainer/global_step‚1849 % _timestamp‚1.7842928866660132e+09 _runtime‚3070.883201406 _step‚148”ˆ‚¶}2– - _wandbruntime‚3070  _runtime‚30703—Îuí'— çO WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5240220 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1737700394496 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1737700394496 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404724736 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 trainer/global_step‚1849 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚149 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚15713489 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚286 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚143734054555648 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1734729090568 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1733384490246 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚31708510 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚529700778688512 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚145976440604160 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4744421 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚529471439187968 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5239934 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2242017957376 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1736355004928 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚15713489 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚15995015 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚531204798250246 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚31698531 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36633600 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚15985042 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1736355004928 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚531438479083008 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚143734422646784 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚531437107520000 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚529700752515072 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚531206168278536 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚529700752515072 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7842928866671085e+09 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚529700778688512 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚15713495 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚145976035879424 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚15713495 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚529700778688512 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚145976440604160 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4744421 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚529471439187968 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5239934 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2242017957376 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1736355004928 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚15713489 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚15995015 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚531204798250246 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚31698531 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36633600 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚15985042 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1736355004928 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚531438479083008 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚143734422646784 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚531437107520000 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚529700752515072 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚531206168278536 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚529700752515072 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7842928866671085e+09 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚529700778688512 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚15713495 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚145976035879424 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚15713495 IDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚278 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚15721695 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚16003507 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4746964 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1737280040448 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚31725208 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚146055110906368 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404717568 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 _runtime‚3072.269599854 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚15993520 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚143811909054464 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚15993520 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚31725208 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚529978779418624 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5242958 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1735652556926 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚529978753245184 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 trainer/global_step‚1849 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1738625437184 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚531717404855808 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚9989927 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4746969 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 _step‚150 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚529978779418624 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1734307956492 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1737280040448 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2243201851904 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9989649 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚529749320966144 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚15721701 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚15721695 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0X#ôû&› õM G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚31725208 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚529978779418624 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5242958 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1735652556926 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚529978753245184 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 trainer/global_step‚1849 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1738625437184 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚531717404855808 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚9989927 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4746969 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 _step‚150 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚529978779418624 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1734307956492 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1737280040448 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2243201851904 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9989649 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚529749320966144 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚15721701 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚15721695 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚531717404855808 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚143811540963328 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2243165225472 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚278 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚15721695 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚16003507 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4746964 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1737280040448 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚31725208 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚146055110906368 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404717568 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 _runtime‚3072.269599854 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚15993520 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚143811909054464 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚15993520–Q P=2œ - _wandbruntime‚3072  _runtime‚3072ˆŒï ‚ 0loss/FRt1/assignment_loss‚2.1237664560244562 0loss/FRt2/assignment_loss‚0.9429922209314581 0loss/FBt2/detection_loss‚0.41071751184682265 epoch‚4 _runtime‚3072.355666566 0loss/FBt1/assignment_loss‚0.1100827128292577 /loss/FRt1/detection_loss‚0.6024587777677096 1loss/SRqqt2/detection_loss‚0.6205573687997721 trainer/global_step‚1849 /loss/FRt2/detection_loss‚1.6395011450348527 $ _timestamp‚1.784292888138746e+09 2loss/SRqqt1/assignment_loss‚0.7402527830757176 2loss/SRqqt2/assignment_loss‚0.5539855318507818 2loss/FBt2/assignment_loss‚0.060081902463714594 1loss/SRqqt1/detection_loss‚1.1259946771094511 /loss/FBt1/detection_loss‚1.4550834682637028 &loss/total_loss‚10.385474555997698 _step‚151CvßJ ž ‡ 1loss/SRqqt2/detection_loss‚0.6205573687997721 trainer/global_step‚1849 /loss/FRt2/detection_loss‚1.6395011450348527 $ _timestamp‚1.784292888138746e+09 2loss/SRqqt1/assignment_loss‚0.7402527830757176 2loss/SRqqt2/assignment_loss‚0.5539855318507818 2loss/FBt2/assignment_loss‚0.060081902463714594 1loss/SRqqt1/detection_loss‚1.1259946771094511 /loss/FBt1/detection_loss‚1.4550834682637028 &loss/total_loss‚10.385474555997698 _step‚151 0loss/FRt1/assignment_loss‚2.1237664560244562 0loss/FRt2/assignment_loss‚0.9429922209314581 0loss/FBt2/detection_loss‚0.41071751184682265 epoch‚4 _runtime‚3072.355666566 0loss/FBt1/assignment_loss‚0.1100827128292577 /loss/FRt1/detection_loss‚0.6024587777677096—xãE®”Ÿ ‚(:‰ šÌèÒ…ß‘÷ gpu.0.gpu‚95.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚290.031' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚58.0062 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ê^6üð  ‚(:å šÌèÒÍÒäø disk.sda1.out‚0 disk.rbd1.out‚ 7.4765625 disk.dm-1.out‚ 114.078125# disk.dm-0.out‚50437.40283203125# proc.memory.rssMB‚ 2858.81640625 disk.rbd3.out‚0# disk.nvme0c0n1.in‚ 4773.80078125! disk.nvme0n1.in‚ 4773.80078125) proc.memory.availableMB‚ 1480.30859375 disk.dm-0.in‚ 4773.80078125 disk.sda3.out‚ 114.078125 disk.drbd2165.in‚0( disk.nvme0c0n1.out‚50437.40283203125& disk.nvme0n1.out‚50437.40283203125 disk.sda2.in‚0* proc.memory.percent‚17.448830604553223) disk./.usagePercent‚37.38111803733237 disk.rbd1.in‚0 network.sent‚ 3.399777e+06# disk./.usageGB‚557.005672454834 disk.drbd2165.out‚0# memory_percent‚90.9649133682251 cpu‚25.576859049373432 network.recv‚803084 disk.sda.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda3.in‚0 disk.sda.out‚ 114.078125 disk.dm-1.in‚0 proc.cpu.threads‚17 disk.sda1.in‚0 disk.rbd3.in‚0 disk.sda2.out‚0\†mÝ”¡ ‚(:‰ ©ÌèÒÏ •ø gpu.0.gpu‚93.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚332.413' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚66.4826 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0y³Lç¢ ‚(:Ü ©ÌèÒ¨åþø disk.drbd2165.in‚0! disk.nvme0n1.out‚ 50446.359375 disk.rbd3.out‚0 network.sent‚ 3.464415e+06 disk.dm-1.in‚0 disk.sda2.in‚0$ memory_percent‚90.96572399139404 proc.cpu.threads‚17) disk./.usagePercent‚37.38149872053578 disk.sda.out‚ 114.36328125 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd2165.out‚0 disk.rbd1.in‚0 disk.rbd0.out‚0 disk.dm-0.in‚ 4773.80078125 disk.sda2.out‚0 disk.rbd1.out‚ 7.4765625 network.recv‚808823# disk./.usageGB‚557.011344909668 disk.dm-0.out‚ 50446.359375 disk.sda3.in‚0# disk.nvme0c0n1.out‚ 50446.359375# proc.memory.rssMB‚ 2858.81640625* proc.memory.percent‚17.448830604553223 cpu‚25.57600322006728 disk.dm-1.out‚ 114.36328125 disk.rbd0.in‚0# disk.nvme0c0n1.in‚ 4773.80078125! disk.nvme0n1.in‚ 4773.80078125) proc.memory.availableMB‚ 1480.17578125 disk.sda.in‚0 disk.sda3.out‚ 114.36328125 disk.rbd3.in‚0Ûã˜îŸ£ ‚(:” ¸ÌèÒáÍòö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚289.503' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚57.900600000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0bŒÆê¤ ‚(:ß ¸ÌèÒ”ªìø$ disk./.usageGB‚557.0165138244629 disk.rbd3.in‚0 disk.dm-1.in‚0 disk.sda3.in‚0 disk.rbd3.out‚0 disk.nvme0n1.in‚ 5172.65625 disk.rbd1.in‚0 disk.rbd1.out‚ 7.5390625 disk.sda.in‚0 disk.sda.out‚ 114.72265625$ memory_percent‚90.96574783325195 disk.nvme0c0n1.in‚ 5172.65625 disk.dm-1.out‚ 114.72265625" disk.dm-0.out‚50458.0205078125' proc.memory.availableMB‚ 1480.171875# proc.memory.rssMB‚ 2858.81640625 proc.cpu.threads‚17 network.recv‚812011* disk./.usagePercent‚37.381845610744676 disk.sda3.out‚ 114.72265625 disk.drbd2165.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda1.out‚0 disk.sda2.in‚0' disk.nvme0c0n1.out‚50458.0205078125 network.sent‚ 3.46813e+06 disk.sda1.in‚0 disk.drbd2165.out‚0 disk.sda2.out‚0 disk.dm-0.in‚ 5172.65625* proc.memory.percent‚17.448830604553223% disk.nvme0n1.out‚50458.0205078125 cpu‚25.575001957287338(¢ÌŸ¥ ‚(:” ÇÌèÒöîö gpu.0.gpu‚100.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚345.458' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚69.09160000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0º1ì¦ ‚(:á ÇÌèÒ‘“™ù$ disk./.usageGB‚557.0182113647461 disk.dm-1.in‚0 proc.cpu.threads‚17 network.sent‚ 3.47312e+06( disk.nvme0c0n1.out‚50466.32958984375& disk.nvme0n1.out‚50466.32958984375 disk.sda2.out‚0 disk.rbd0.out‚0 disk.drbd2165.out‚0 disk.rbd1.out‚ 7.5390625 disk.sda.in‚0( proc.memory.availableMB‚ 1480.0859375 cpu‚25.57416083504129) disk./.usagePercent‚37.38195953409742 disk.rbd3.out‚0 disk.rbd1.in‚0 disk.sda.out‚ 114.984375 disk.dm-1.out‚ 114.984375# disk.dm-0.out‚50466.32958984375 disk.drbd2165.in‚0 disk.rbd3.in‚0" disk.nvme0c0n1.in‚ 5352.8359375 disk.sda1.out‚0$ memory_percent‚90.96627235412598* proc.memory.percent‚17.448830604553223 disk.sda3.in‚0 disk.nvme0n1.in‚ 5352.8359375 disk.sda2.in‚0 disk.rbd0.in‚0 disk.dm-0.in‚ 5352.8359375# proc.memory.rssMB‚ 2858.81640625 network.recv‚815341 disk.sda1.in‚0 disk.sda3.out‚ 114.984375à.ž§ ‚(:“ ÖÌèÒªðËö gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚289.84' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚57.967999999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0@ÜIèì¨ ‚(:á ÖÌèÒµ½ù network.sent‚ 3.478253e+06 disk.dm-0.in‚ 5352.8359375 cpu‚25.573249799265902 network.recv‚818524 disk.sda.in‚0 disk.rbd0.out‚0 disk.nvme0n1.in‚ 5352.8359375# disk.nvme0n1.out‚50476.46484375 disk.dm-1.out‚ 115.13671875 disk.rbd0.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd3.out‚0 disk.rbd1.in‚0$ disk./.usageGB‚557.0254364013672 disk.dm-0.out‚50476.46484375* proc.memory.percent‚17.448830604553223 disk.sda3.in‚0% disk.nvme0c0n1.out‚50476.46484375 disk.sda2.out‚0# proc.memory.rssMB‚ 2858.81640625 disk.sda.out‚ 115.13671875" disk.nvme0c0n1.in‚ 5352.8359375) disk./.usagePercent‚37.38244441236727 disk.dm-1.in‚0 disk.sda3.out‚ 115.13671875 disk.drbd2165.out‚0 disk.rbd3.in‚0) proc.memory.availableMB‚ 1480.12890625 proc.cpu.threads‚17 disk.drbd2165.in‚0 disk.sda2.in‚0 disk.rbd1.out‚ 7.6640625$ memory_percent‚90.96601009368896,Æ3s2© - _wandbruntime‚3149  _runtime‚3149K|HíǪ Á % _timestamp‚1.7842929653439374e+09 _runtime‚3149.561983339 _step‚152 & lr-AdamW/pg1‚0.0006819115368242087 & lr-AdamW/pg2‚0.0006819115368242087 trainer/global_step‚1899i²éÛÌ« Æ & lr-AdamW/pg1‚0.0006819115368242087 & lr-AdamW/pg2‚0.0006819115368242087 trainer/global_step‚1899 % _timestamp‚1.7842929653439374e+09 _runtime‚3149.561983339 _step‚152˜°_˜¬2¬ - _wandbruntime‚3149  _runtime‚3149ŸM¯K„­ éO F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚16410794 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚543600815194112 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4871576 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5378646 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚545383459956224 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚543365502668800 _step‚153 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1780966203676 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚543600815194112 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚Î Þk 9979 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚10249952 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚265 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1784016325120 % _timestamp‚1.7842929653446903e+09 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚543600789020672 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1784016325120 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2301069706752 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚545384831519232 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚545145124272154 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚543600789020672 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚147608743030784 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 _runtime‚3149.562097425 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0îÀ¬ô'® îO F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1782670935552 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4871571 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚32544562 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚10250222 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚149909812737536 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚16123789 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚543365528096768 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚16123789 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚32534583 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚16420767 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚16420767 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 trainer/global_step‚1899 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚270 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚543600815194112 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚10249952 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚265 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1784016325120 % _timestamp‚1.7842929653446903e+09 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚543600789020672 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1784016325120 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2301069706752 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚545384831519232 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚545145124272154 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚543600789020672 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚147608743030784 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 _runtime‚3149.562097425 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚16410794 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚543600815194112 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4871576 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5378646 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚545383459956224 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚543365502668800 _step‚153 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1780966203676 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10255272 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚16132001 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2302224404992 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1781889670034 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚147685861347328 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚32561260 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5381158 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚149988085752320 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1780545069600 _runtime‚ 3150.944431 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚16132001 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4874119 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚10255536 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1784941367808 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚147686229438464 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2302258934272 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚545662385721856 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4874114 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚16131995 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚32551267 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚16131995 $ _timestamp‚1.784292966726968e+09 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚545423929516576 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚16419272 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚543878789750784 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚32551267 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0èzAÚг ôM TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚264 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5381417 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚32561260 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5381158 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚149988085752320 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1780545069600 _runtime‚ 3150.944431 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚16132001 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4874119 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚10255536 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1784941367808 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚147686229438464 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2302258934272 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚545662385721856 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4874114 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚16131995 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚32551267 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚16131995 $ _timestamp‚1.784292966726968e+09 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚545423929516576 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚16419272 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚543878789750784 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚32551267 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚543878815924224 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 M=DeviceStatsMon øZ^* itor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10255272 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚16132001 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2302224404992 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1781889670034 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚147685861347328 B ¶ „ _step‚155 1loss/FBt2/assignment_loss‚0.02222903793147286 trainer/global_step‚1899 2loss/SRqqt1/assignment_loss‚0.7327790854055396 2loss/SRqqt2/assignment_loss‚0.5310895229102949 /loss/FRt1/detection_loss‚0.6419610630621271 $ _timestamp‚1.784292966814426e+09 epoch‚4 /loss/FRt1/assignment_loss‚2.162521896761615 0loss/FRt2/assignment_loss‚0.8769768194612024 1loss/FBt1/assignment_loss‚0.12631763255483427 /loss/FRt2/detection_loss‚1.6346229888530899 1loss/SRqqt1/detection_loss‚0.9892913115272919 /loss/FBt1/detection_loss‚0.8170030795804718 /loss/FBt2/detection_loss‚0.5605954052808211 1loss/SRqqt2/detection_loss‚0.7531259727803673 %loss/total_loss‚9.848513816109127 _runtime‚3151.031393194›s\š.Ÿ· ‚(:” ôÌèÒ¹†îö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚274.502' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚54.900400000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0²e;oõ¸ ‚(:ê ôÌèÒŸ¾¤ù disk.dm-1.in‚0" disk.nvme0c0n1.in‚ 5352.8359375* proc.memory.percent‚17.448830604553223 disk.rbd1.out‚ 7.7265625 network.sent‚ 3.535625e+06 disk.dm-1.out‚ 116.14453125 disk.drbd2165.out‚0& disk.nvme0n1.out‚50499.70068359375 disk.dm-0.in‚ 5352.8359375 disk.sda2.in‚0 disk.sda2.out‚0 cpu‚25.571622925730193 network.recv‚826127 disk.rbd0.in‚0 disk.sda1.in‚0 disk.drbd2165.in‚0 disk.rbd3.out‚0( disk.nvme0c0n1.out‚50499.70068359375 disk.sda1.out‚0 disk.nvme0n1.in‚ 5352.8359375# proc.memory.rssMB‚ 2858.81640625# disk./.usageGB‚557.033145904541# disk.dm-0.out‚50499.70068359375 disk.sda3.in‚0 disk.rbd1.in‚0$ memory_percent‚90.97049236297607* disk./.usagePercent‚37.382961803593965 disk.sda.out‚ 116.14453125 disk.rbd0.out‚0) proc.memory.availableMB‚ 1479.39453125 proc.cpu.threads‚17 disk.sda.in‚0 disk.sda3.out‚ 116.14453125 disk.rbd3.in‚0›6vq•¹ ‚(:Š ƒÍèÒûÌÛ÷ gpu.0.gpu‚100.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚171.553' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.3106 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0qäº ‚(:Ù ƒÍèÒ;Ôù network.recv‚830233$ disk./.usageGB‚557.0443305969238 disk.rbd3.out‚0 cpu‚25.570813081902745 proc.cpu.threads‚17" disk.dm-0.out‚50512.4208984375 disk.sda1.out‚0* disk./.usagePercent‚37.383712417684364 disk.rbd0.in‚0 disk.dm-0.in‚ 5832.2265625 disk.drbd2165.in‚0 disk.rbd1.in‚0 disk.sda.in‚0$ memory_percent‚90.97158908843994) proc.memory.availableMB‚ 1479.21484375 network.sent‚ 3.588015e+06' disk.nvme0c0n1.out‚50512.4208984375# proc.memory.rssMB‚ 2858.81640625 disk.sda1.in‚0 disk.sda3.out‚116.375 disk.rbd1.out‚ 7.7265625 disk.dm-1.in‚0 disk.dm-1.out‚116.375 disk.rbd0.out‚0 disk.sda.out‚116.375 disk.sda3.in‚0 disk.rbd3.in‚0" disk.nvme0c0n1.in‚ 5832.2265625 disk.nvme0n1.in‚ 5832.2265625% disk.nvme0n1.out‚50512.4208984375 disk.sda2.in‚0 disk.sda2.out‚0 disk.drbd2165.out‚0* proc.memory.percent‚17.448830604553223€ˆF·Ÿ» ‚(:” ’ÍèÒòãø gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚274.423' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚54.884600000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0°â ]ç¼ ‚(:Ü ’ÍèÒÔìÖù' proc.memory.availableMB‚ 1478.953125# proc.memory.rssMB‚ 2858.81640625 network.sent‚ 3.593008e+06& disk.nvme0n1.out‚50520.59912109375 disk.rbd0.out‚0 network.recv‚833133 disk.drbd2165.out‚0 disk.dm-0.in‚ 5832.2265625$ disk./.usageGB‚557.0487480163574 disk.rbd3.out‚0 disk.sda2.in‚0 disk.sda.in‚0 disk.sda1.in‚0) disk./.usagePercent‚37.38400887440901 disk.drbd2165.in‚0 disk.nvme0n1.in‚ 5832.2265625 disk.rbd1.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚116.5625 cpu‚25.569860995274805( disk.nvme0c0n1.out‚50520.59912109375# disk.dm-0.out‚50520.59912109375$ memory_percent‚90.97318649291992* proc.memory.percent‚17.448830604553223 proc.cpu.threads‚17 disk.sda3.out‚116.5625 disk.sda.out‚116.5625 disk.rbd0.in‚0" disk.nvme0c0n1.in‚ 5832.2265625 disk.sda3.in‚0 disk.rbd3.in‚0 disk.sda2.out‚0 disk.rbd1.out‚ 7.7890625 disk.sda1.out‚0uh´ŸŸ½ ‚(:” ¡ÍèÒ‹ßîö gpu.0.gpu‚91.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚247.268' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚49.453599999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0~>·î¾ ‚(:ã ¡ÍèÒ¾ ø" disk.nvme0c0n1.in‚ 5832.2265625) proc.memory.availableMB‚ 1478.36328125# proc.memory.rssMB‚ 2858.81640625( disk./.usagePercent‚37.3846734699679 disk.sda1.out‚0 disk.sda3.in‚0 disk.sda3.out‚ 117.0390625& disk.nvme0n1.out‚50532.28662109375$ memory_percent‚90.97678661346436 disk.nvme0n1.in‚ 5832.2265625 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd1.out‚ 7.7890625 disk.dm-1.in‚0 disk.dm-1.out‚ 117.0390625 disk.sda1.in‚0 disk.drbd2165.in‚0 disk.rbd3.in‚0 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.rbd3.out‚0* proc.memory.percent‚17.448830604553223 cpu‚25.569019634894563 disk.drbd2165.out‚0 disk.rbd1.in‚0 disk.dm-0.in‚ 5832.2265625# disk.dm-0.out‚50532.28662109375 network.sent‚ 3.5976e+06 network.recv‚836698 disk.sda.in‚0 disk.sda.out‚ 117.0390625( disk.nvme0c0n1.out‚50532.28662109375# disk./.usageGB‚557.058650970459 disk.rbd0.out‚0^¿ ‹”¿ ‚(:‰ °ÍèÒ‘Ç©ø gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚303.048' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚60.6096 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0›ÉùŽòÀ ‚(:ç °ÍèÒù disk.rbd0.in‚0 disk.rbd3.in‚0& disk.nvme0n1.out‚50541.99658203125( proc.memory.availableMB‚ 1478.4609375 disk.sda1.out‚0 disk.nvme0n1.in‚ 5832.2265625 network.recv‚839524# disk.dm-0.out‚50541.99658203125 disk.sda1.in‚0$ disk./.usageGB‚557.0639762878418 disk.sda3.out‚ 117.19140625 disk.drbd2165.in‚0 disk.rbd3.out‚0 disk.rbd1.in‚0 disk.sda.out‚ 117.19140625# memory_percent‚90.9761905670166# proc.memory.rssMB‚ 2858.81640625 disk.sda2.out‚0 disk.sda.in‚0* proc.memory.percent‚17.448830604553223( disk./.usagePercent‚37.3850308564857( disk.nvme0c0n1.out‚50541.99658203125 disk.rbd1.out‚ 7.8515625 proc.cpu.threads‚17 disk.dm-1.out‚ 117.19140625 disk.dm-0.in‚ 5832.2265625 disk.drbd2165.out‚0" disk.nvme0c0n1.in‚ 5832.2265625 disk.sda2.in‚0 disk.dm-1.in‚0 network.sent‚ 3.603556e+06 disk.rbd0.out‚0 disk.sda3.in‚0 cpu‚25.5683007597215277Ô¾2Á - _wandbruntime‚3228  _runtime‚3228B€^ýÆ À $ _timestamp‚1.784293043946749e+09 _runtime‚3228.163902829 _step‚156 & lr-AdamW/pg1‚0.0006812410431378921 & lr-AdamW/pg2‚0.0006812410431378921 trainer/global_step‚1949\¥½Ëà Å trainer/global_step‚1949 $ _timestamp‚1.784293043946749e+09 _runtime‚3228.163902829 _step‚156 & lr-AdamW/pg1‚0.0006812410431378921 & lr-AdamW/pg2‚0.0006812410431378921œ«M$a2Ä - _wandbruntime‚3228  _runtime‚3228úiÂî'Å èO _step‚157 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 _runtime‚ 3228.16519562 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚559331183955456 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4998726 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚557500825526272 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚559331183955456 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚557500851699712 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1825858716462 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚10515061 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚559329812392448 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚16846519 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4998721 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚151482695323648 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚33380614 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2360168455680 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚16534095 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚557500825526272 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚16534095 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚33370635 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚284 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5516056 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚557500851699712 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1830332255744 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995@Xó'Æ íO [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1830332255744 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2360131822080 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚16534089 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚16836546 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚16534089 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1828986866176 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚153842827145728 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚559085450294062 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1827203316784 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚151483063414784 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚559086820322352 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚153843231870464 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚16836546 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚33380614 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2360168455680 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚16534095 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚557500825526272 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚16534095 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚33370635 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚284 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5516056 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚557500851699712 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1830332255744 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 _step‚157 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 _runtime‚ 3228.16519562 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚559331183955456 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4998726 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚557500825526272 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚559331183955456 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚557500851699712 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1825858716462 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚10515061 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚559329812392448 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚16846519 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4998721 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚151482695323648 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚16542295 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1828126783142 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 _KDeviceStatsM“ûOˆ¬$onitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚151560181731328 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 % _timestamp‚1.7842930453325384e+09 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10520040 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚557537473355776 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1826782182708 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1829911901696 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚557778826256384 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404717568 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2361344379392 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1829911901696 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5001264 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚153921894201856 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚16542301 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚33397312 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚16542295 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5519038 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚559365625566886 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1831257298432 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚10520307 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚262 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36626432 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚557778852429824 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚33387319 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚16855011 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚559610109728256 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 trainer/global_step‚1949 _step‚158 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚151560549822464 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚33397312 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1831257298432 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚267 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 _runtime‚3229.550102993 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚557778852429824 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚559364255538484 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2361307752960 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚559608738158080 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5518776 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216ÁÅG/þ&É øM T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚559610109728256 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 trainer/global_step‚1949 _step‚158 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚151560549822464 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚33397312 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1831257298432 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚267 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 _runtime‚3229.550102993 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚557778852429824 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚559364255538484 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2361307752960 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚559608738158080 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5518776 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚16855011 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚16542295 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1828126783142 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚151560181731328 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 % _timestamp‚1.7842930453325384e+09 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10520040 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚557537473355776 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1826782182708 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1829911901696 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚557778826256384 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404717568 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2361344379392 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1829911901696 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5001264 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚153921894201856 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚16542301 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚33397312 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚16542295 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5519038 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚559365625566886 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1831257298432 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚10520307 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚262 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36626432 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚557778852429824 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚33387319 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚16855011 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2žÅ‘G2Ê - _wandbruntime‚3229  _runtime‚3229Âa^Ë € %loss/total_loss‚9.954721914027992 /loss/FRt1/assignment_loss‚2.135125179489261 1loss/FBt2/assignment_loss‚0.05581147260258194 /loss/FBt2/detection_loss‚0.6469899349225064 trainer/global_step‚1949 % _timestamp‚1.7842930454200964e+09 /loss/FRt1/detection_loss‚0.5372423789199552 /loss/FRt2/detection_loss‚1.6913433391368597 _runtime‚3229.637064597 0loss/FRt2/assignment_loss‚0.8883517195756299 1loss/FBt1/assignment_loss‚0.05765227197673095 1loss/SRqqt1/detection_loss‚1.0100154743053293 /loss/FBt1/detection_loss‚0.9285966482013464 epoch‚4 _step‚159 2loss/SRqqt1/assignment_loss‚0.7828460595181934 2loss/SRqqt2/assignment_loss‚0.5414277564330164 1loss/SRqqt2/detection_loss‚0.6793196789465803d– Ì … 1loss/FBt2/assignment_loss‚0.05581147260258194 /loss/FBt2/detection_loss‚0.6469899349225064 trainer/global_step‚1949 % _timestamp‚1.7842930454200964e+09 /loss/FRt1/detection_loss‚0.5372423789199552 /loss/FRt2/detection_loss‚1.6913433391368597 _runtime‚3229.637064597 0loss/FRt2/assignment_loss‚0.8883517195756299 1loss/FBt1/assignment_loss‚0.05765227197673095 1loss/SRqqt1/detection_loss‚1.0100154743053293 /loss/FBt1/detection_loss‚0.9285966482013464 epoch‚4 _step‚159 2loss/SRqqt1/assignment_loss‚0.7828460595181934 2loss/SRqqt2/assignment_loss‚0.5414277564330164 1loss/SRqqt2/detection_loss‚0.6793196789465803 %loss/total_loss‚9.954721914027992 /loss/FRt1/assignment_loss‚2.135125179489261Ÿ!Š8ŸÍ ‚(:” ¿ÍèÒøêÓö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚291.919' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚58.383799999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Vk³ôîÎ ‚(:ã ¿ÍèÒ³ÁÆø disk.sda2.out‚0 disk.rbd0.in‚0 disk.sda1.out‚0 disk.rbd3.out‚0( disk./.usagePercent‚37.3853344814213 disk.rbd1.in‚0 disk.sda.in‚0" disk.nvme0c0n1.in‚ 6150.1015625" disk.dm-0.out‚50554.0595703125 disk.drbd2165.in‚0$ disk./.usageGB‚557.0685005187988' disk.nvme0c0n1.out‚50554.0595703125 disk.dm-1.out‚ 117.44921875 disk.rbd0.out‚0 disk.sda1.in‚0$ memory_percent‚90.97347259521484# proc.memory.rssMB‚ 2858.81640625 network.sent‚ 3.607102e+06 disk.rbd3.in‚0 disk.sda.out‚ 117.44921875 disk.dm-1.in‚0 disk.dm-0.in‚ 6150.1015625* proc.memory.percent‚17.448830604553223 cpu‚25.567307264815536 proc.cpu.threads‚17 disk.rbd1.out‚ 7.8515625 disk.sda3.out‚ 117.44921875& proc.memory.availableMB‚ 1478.90625 network.recv‚842170 disk.sda2.in‚0 disk.drbd2165.out‚0 disk.sda3.in‚0 disk.nvme0n1.in‚ 6150.1015625% disk.nvme0n1.out‚50554.0595703125­™¿õ”Ï ‚(:‰ ÎÍèÒÛ–‘÷ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚185.874' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚37.1748 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‰ÜÚŽîÐ ‚(:ã ÎÍèÒ±àÃø network.sent‚ 3.672058e+06) disk./.usagePercent‚37.38569084390897 disk.sda.in‚0! disk.dm-0.out‚50562.462890625 disk.sda3.in‚0 disk.sda3.out‚ 117.61328125 disk.drbd2165.out‚0 cpu‚25.566618374682356 disk.sda2.out‚0 disk.sda.out‚ 117.61328125 proc.cpu.threads‚17 disk.sda2.in‚0 disk.dm-1.in‚0 disk.rbd0.out‚0 disk.sda1.out‚0 disk.nvme0n1.in‚ 6150.1015625$ disk./.usageGB‚557.0738105773926" disk.nvme0c0n1.in‚ 6150.1015625 disk.rbd3.out‚0 disk.rbd1.in‚0 disk.dm-1.out‚ 117.61328125 disk.dm-0.in‚ 6150.1015625# proc.memory.rssMB‚ 2858.81640625 network.recv‚847958 disk.rbd1.out‚ 7.9140625 disk.sda1.in‚0 disk.rbd3.in‚0 disk.rbd0.in‚0 disk.drbd2165.in‚0& disk.nvme0c0n1.out‚50562.462890625$ memory_percent‚90.97495079040527( proc.memory.availableMB‚ 1478.6640625* proc.memory.percent‚17.448830604553223$ disk.nvme0n1.out‚50562.4628906256·Â%”Ñ ‚(:‰ ÝÍèÒãºèö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚275.567' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.1134 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0%‰ÿíÒ ‚(:â ÝÍèÒ—†Èø network.sent‚ 3.675773e+06 disk.sda2.in‚0 disk.rbd1.in‚0) disk./.usagePercent‚37.38600010101033 disk.sda3.out‚ 117.87890625$ disk./.usageGB‚557.0784187316895 disk.nvme0n1.in‚ 6150.1015625 disk.sda2.out‚0 disk.rbd0.out‚0" memory_percent‚90.975022315979! disk.dm-0.out‚50573.228515625 network.recv‚851508 disk.sda.out‚ 117.87890625 proc.cpu.threads‚17 disk.rbd3.in‚0$ disk.nvme0n1.out‚50573.228515625 disk.rbd1.out‚ 7.9140625 disk.sda.in‚0 disk.dm-1.out‚ 117.87890625) proc.memory.availableMB‚ 1478.65234375# proc.memory.rssMB‚ 2858.81640625 disk.sda3.in‚0 disk.sda1.out‚0 disk.drbd2165.out‚0" disk.nvme0c0n1.in‚ 6150.1015625& disk.nvme0c0n1.out‚50573.228515625 disk.rbd0.in‚0 disk.dm-0.in‚ 6150.1015625* proc.memory.percent‚17.448830604553223 cpu‚25.565707050689227 disk.sda1.in‚0 disk.drbd2165.in‚0 disk.rbd3.out‚0 disk.dm-1.in‚0RUÒ[”Ó ‚(:‰ ìÍèÒ„¾ìö gpu.0.gpu‚45.0 gpu.0.memory‚15, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚342.114' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚68.4228 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0o›ìÔ ‚(:á ìÍèÒÚ°þø disk.sda2.in‚0 disk.rbd1.out‚ 8.0390625 disk.dm-1.out‚ 118.09375 disk.rbd0.in‚0 disk.sda1.in‚0) proc.memory.availableMB‚ 1478.66015625 disk.sda3.in‚0 disk.drbd2165.in‚0 disk.sda2.out‚0) disk./.usagePercent‚37.38630065385553 disk.sda3.out‚ 118.09375 disk.drbd2165.out‚0 disk.sda.in‚0 disk.sda1.out‚0 network.recv‚854530" disk.nvme0c0n1.in‚ 6150.1015625( disk.nvme0c0n1.out‚50582.01318359375 disk.sda.out‚ 118.09375* proc.memory.percent‚17.448830604553223$ disk./.usageGB‚557.0828971862793 disk.rbd0.out‚0# disk.dm-0.out‚50582.01318359375$ memory_percent‚90.97497463226318 cpu‚25.564948862309024 network.sent‚ 3.680759e+06 disk.rbd3.out‚0 disk.rbd1.in‚0 disk.dm-0.in‚ 6150.1015625# proc.memory.rssMB‚ 2858.81640625& disk.nvme0n1.out‚50582.01318359375 disk.dm-1.in‚0 proc.cpu.threads‚17 disk.rbd3.in‚0 disk.nvme0n1.in‚ 6150.1015625™XÄÁ”Õ ‚(:‰ ûÍèҤ݆÷ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚259.116' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚51.8232 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0†ØÖâÖ ‚(:× ûÍèÒŪºø disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-0.out‚50592.31640625 disk.drbd2165.out‚0$ memory_percent‚90.97495079040527( proc.memory.availableMB‚ 1478.6640625) disk./.usagePercent‚37.38668389713429 disk.dm-0.in‚ 6164.203125 disk.sda1.out‚0 disk.rbd3.in‚0 disk.rbd1.out‚ 8.0390625 disk.sda.out‚ 118.578125 disk.dm-1.in‚0 disk.rbd0.out‚0 disk.sda3.out‚ 118.578125% disk.nvme0c0n1.out‚50592.31640625 disk.nvme0n1.in‚ 6164.203125 disk.dm-1.out‚ 118.578125 disk.sda1.in‚0 cpu‚25.564137826874955 network.sent‚ 3.685886e+06 disk.sda.in‚0 disk.rbd0.in‚0 disk.sda3.in‚0 disk.rbd3.out‚0# proc.memory.rssMB‚ 2858.81640625 disk.rbd1.in‚0 disk.drbd2165.in‚0 proc.cpu.threads‚17* proc.memory.percent‚17.448830604553223 network.recv‚858272$ disk./.usageGB‚557.0886077880859! disk.nvme0c0n1.in‚ 6164.203125# disk.nvme0n1.out‚50592.31640625ÊíZ2× - _wandbruntime‚3306  _runtime‚3306üuó¶ÇØ Á % _timestamp‚1.7842931226437294e+09 _runtime‚3306.861997047 _step‚160 & lr-AdamW/pg1‚0.0006805490126789975 & lr-AdamW/pg2‚0.0006805490126789975 trainer/global_step‚1999®^†oÌÙ Æ % _timestamp‚1.7842931226437294e+09 _runtime‚3306.861997047 _step‚160 & lr-AdamW/pg1‚0.0006805490126789975 & lr-AdamW/pg2‚0.0006805490126789975 trainer/global_step‚1999 Hó2Ú - _wandbruntime‚3306  _runtime‚3306Êuéÿ Û èO ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1873440429892 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚17262298 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5125876 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚261 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚17262298 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1876648186368 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚571400862031872 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚17272271 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 $ _timestamp‚1.784293122644516e+09 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚571400888205312 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚266 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2419061534208 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1875302796800 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚573276164828672 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚10780202 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚571153705914368 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚157776077241856 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚34206687 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚16944389 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚16944395 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚16944389 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚157776479869440 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5125871 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2419096070656 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚34216666 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚155357383798784 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚10780468 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5654592 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _runtime‚3306.862079793 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1875302796800 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚571400862031872 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1876648186368 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚573025776315970 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚571153680486400 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚17272271 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚155357015707648 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 _step‚161 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5654331 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚573276164828672 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚571400888205312 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1872095829570 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚16944395 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚571400862031872 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚17272271 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 $ _timestamp‚1.784293122644516e+09 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚571400888205312 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚266 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2419061534208 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1875302796800 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚573276164828672 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚10780202 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚571153705914368 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚157776077241856 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚34206687 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚16944389 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚16944395 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚16944389 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚157776479869440 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5125871 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2419096070656 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚34216666 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚155357383798784 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚10780468 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5654592 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _runtime‚3306.862079793 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1875302796800 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚571400862031872 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1876648186368 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚573025776315970 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚571153680486400 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚17272271 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚155357015707648 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 _step‚161 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5654331 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚573276164828672 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚571400888205312 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1872095829570 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚16944395 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 I8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1874363896250 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚16952601 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚157854743119360 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚34233364 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚34233364 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚157855145739776 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚571431562264576 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚17280763 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5128419 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5128414 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 trainer/global_step‚1999 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 % _timestamp‚1.7842931240280666e+09 _runtime‚3308.245562531 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚573304581560392 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚571678862761984 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚252 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚573555090594304 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚17270776 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1876227832320 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2420275533312 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚17270776 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1877573229056 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1876227832320 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10785486 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚571678888935424 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚34233364 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚157855145739776 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚571431562264576 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚17280763 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5128419 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5128414 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 trainer/global_step‚1999 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 % _timestamp‚1.7842931240280666e+09 _runtime‚3308.245562531 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚573304581560392 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚571678862761984 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚252 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚573555090594304 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚17270776 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1876227832320 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2420275533312 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 D>DeviceStatsMonitor‰Öñ#.on_train_batch_end/segment.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚17270776 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1877573229056 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1876227832320 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10785486 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚571678888935424 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1874363896250 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚16952601 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚157854743119360 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119¢—(Á2à - _wandbruntime‚3308  _runtime‚33083;sá € epoch‚4 2loss/SRqqt2/assignment_loss‚0.5226329240865079 1loss/FBt1/assignment_loss‚0.13008602864510935 /loss/FRt2/detection_loss‚1.6089235103023904 %loss/total_loss‚9.548585605827105 trainer/global_step‚1999 /loss/FBt2/detection_loss‚0.5351498197750563 0loss/FRt1/assignment_loss‚2.0052590959016667 2loss/SRqqt1/assignment_loss‚0.7449948219586319 /loss/FRt1/detection_loss‚0.5582322111446275 1loss/SRqqt2/detection_loss‚0.7151529542658824 /loss/FBt1/detection_loss‚0.8144775545989562 $ _timestamp‚1.784293124115456e+09 _runtime‚3308.332444382 _step‚163 /loss/FRt2/assignment_loss‚0.965266053862386 2loss/FBt2/assignment_loss‚0.041130405470018616 1loss/SRqqt1/detection_loss‚0.9072802258158723c1 â … 0loss/FRt1/assignment_loss‚2.0052590959016667 2loss/SRqqt1/assignment_loss‚0.7449948219586319 /loss/FRt1/detection_loss‚0.5582322111446275 1loss/SRqqt2/detection_loss‚0.7151529542658824 /loss/FBt1/detection_loss‚0.8144775545989562 $ _timestamp‚1.784293124115456e+09 _runtime‚3308.332444382 _step‚163 /loss/FRt2/assignment_loss‚0.965266053862386 2loss/FBt2/assignment_loss‚0.041130405470018616 1loss/SRqqt1/detection_loss‚0.9072802258158723 epoch‚4 2loss/SRqqt2/assignment_loss‚0.5226329240865079 1loss/FBt1/assignment_loss‚0.13008602864510935 /loss/FRt2/detection_loss‚1.6089235103023904 %loss/total_loss‚9.548585605827105 trainer/global_step‚1999 /loss/FBt2/detection_loss‚0.5351498197750563£¾©&žã ‚(:“ ŠÎèÒ¯²ðö gpu.0.gpu‚47.0 gpu.0.memory‚15, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚330.528' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚66.10560000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ì·ªçä ‚(:Ü ŠÎèÒåöŽù! disk.nvme0c0n1.in‚ 6628.515625* proc.memory.percent‚17.448830604553223 disk.sda1.out‚0 disk.sda.out‚ 118.86328125 disk.dm-0.in‚ 6628.515625 disk.drbd2165.in‚0 disk.rbd3.out‚0& disk.nvme0c0n1.out‚50601.998046875$ disk./.usageGB‚557.0940132141113 disk.rbd0.in‚0# memory_percent‚90.9754753112793 proc.cpu.threads‚17 disk.sda1.in‚0 network.sent‚ 3.69097e+06) disk./.usagePercent‚37.38704665981031 disk.rbd0.out‚0! disk.dm-0.out‚50601.998046875 disk.sda3.in‚0# proc.memory.rssMB‚ 2858.81640625 disk.sda2.out‚0 disk.rbd1.out‚ 8.1015625 disk.sda2.in‚0 disk.rbd1.in‚0 cpu‚25.56840719438697 network.recv‚861352 disk.dm-1.in‚0 disk.sda3.out‚ 118.86328125 disk.drbd2165.out‚0 disk.nvme0n1.in‚ 6628.515625$ disk.nvme0n1.out‚50601.998046875' proc.memory.availableMB‚ 1478.578125 disk.rbd3.in‚0 disk.sda.in‚0 disk.dm-1.out‚ 118.86328125LÁ!’å ‚(:‡ ™ÎèÒôÁ¸÷ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚331.497' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚66.2994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0K:êæ ‚(:ß ™ÎèÒÉçôø disk.drbd2165.in‚0! disk.nvme0c0n1.in‚ 6628.515625# disk.dm-0.out‚50614.34130859375 disk.sda1.out‚0 proc.cpu.threads‚17$ disk./.usageGB‚557.0986442565918 disk.sda2.out‚0 disk.rbd1.in‚0) disk./.usagePercent‚37.38735745295688 disk.sda.in‚0# proc.memory.rssMB‚ 2804.80859375 disk.rbd0.in‚0' proc.memory.availableMB‚ 1533.359375 disk.drbd2165.out‚0 disk.rbd3.in‚0 disk.rbd0.out‚0 cpu‚25.579436765966044 disk.dm-1.out‚ 119.109375 disk.sda1.in‚0 network.sent‚ 3.754071e+06 network.recv‚866401 disk.nvme0n1.in‚ 6628.515625& disk.nvme0n1.out‚50614.34130859375 disk.sda2.in‚0$ memory_percent‚90.64111709594727 disk.sda3.out‚ 119.109375 disk.rbd3.out‚0( disk.nvme0c0n1.out‚50614.34130859375 disk.sda.out‚ 119.109375 disk.dm-1.in‚0* proc.memory.percent‚17.119193077087402 disk.sda3.in‚0 disk.rbd1.out‚ 8.1015625 disk.dm-0.in‚ 6628.515625ŒI‹2ç - _wandbruntime‚3334  _runtime‚3334†6B=è ¼z 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.1571812927722931 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.1362694948911667 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5987222194671631 *jet/accuracy_2_of_4‚0.7156152725219727 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.7996599078178406 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 6 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.543749988079071 $ _timestamp‚1.784293150293741e+09 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6226603984832764 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7142857313156128 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 particle/accuracy_3_of_3‚1 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.5574635863304138 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.15245017409324646 8"Purity/1FRt0SRqqt1FBt/event_purity‚0.313802570104599 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9966313242912292 ."Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.40625 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.033418554812669754 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 jet/accuracy_2_of_5‚1 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5577569007873535 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9980361461639404 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7960505485534668 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.17816470563411713 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.03309882804751396 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3229166865348816 8"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.550882875919342 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 )jet/accuracy_1_of_3‚0.900711715221405 particle/accuracy_1_of_3‚1 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7142857313156128 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.17180967330932617 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9960675239562988 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.3865740895271301 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9972801208496094 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.19145743548870087 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.29960599541664124 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7889645099639893 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 *jet/accuracy_1_of_2‚0.7317983508110046 jet/accuracy_1_of_5‚1 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9938672184944153 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6343399882316589 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6047952771186829 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 +particle/specificity‚0.8970661163330078 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.18488705158233643 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.1783018857240677 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7916666865348816 particle/accuracy_2_of_2‚1 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7221844792366028 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2654668986797333 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.24916736781597137 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.23390595614910126 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.46795958280563354 ("Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6240332722663879 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.20831939578056335 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8463944792747498 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.4701395034790039 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.14075863361358643 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9933808445930481 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.5398752689361572 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.27894243597984314 jet/accuracy_3_of_5‚1 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.07407407462596893 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.21372097730636597 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.1918787658214569 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 +jet/accuracy_3_of_4‚0.06014878302812576 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.1869795024394989 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.1538461595773697 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5775270462036133 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9976814985275269 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6198742389678955 8"Purity/0FRt0SRqqt*FBt/event_purity‚0.989503026008606 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 jet/accuracy_4_of_5‚0 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.21522071957588196 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.21171638369560242 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.6549904942512512 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.6875 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.06913406401872635 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.5323513746261597 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7916666865348816 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 8"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.522318422794342 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.7882838249206543 9"Purity/2FRt0SRqqt0FBt/event_purity‚0.0171547569334507 +jet/accuracy_3_of_3‚0.04953882098197937 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 particle/accuracy_4_of_5‚1 particle/accuracy_5_of_5‚0 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8259488344192505 6 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.997619092464447 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.20063325762748718 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 'particle/f_score‚0.6154343485832214 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6040421724319458 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 particle/accuracy_2_of_3‚1 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.29501330852508545 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.19774077832698822 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9543450474739075 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6443070769309998 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5075536966323853 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5487446784973145 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6230413913726807 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.17180967330932617 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 /particle/accuracy_3_of_4‚0.9938491582870483 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.5001577734947205 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 & Purity/1FRt1SRqqt*FBt/FBt_purity‚1 particle/accuracy_3_of_5‚1 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7888889312744141 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.24204999208450317 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5628164410591125 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9747141599655151 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8008692860603333 ("Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.1538461595773697 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6302083730697632 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9929066896438599 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9911805391311646 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 particle/accuracy_1_of_2‚1 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.8986495733261108 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.6589342355728149 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.625 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9821993708610535 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.49426713585853577 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9890351891517639 epoch‚4 _runtime‚3334.511621382 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9629628658294678 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9771724939346313 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.16794168949127197 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5588107109069824 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9909991025924683 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9972801208496094 _step‚164 6 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.993127703666687 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.06780551373958588 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.17667154967784882 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.6831159591674805 ,jet/accuracy_4_of_4‚0.013899499550461769 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9230769276618958 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.4710586965084076 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.16513125598430634 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5011357069015503 8"Purity/0FRt1SRqqt1FBt/event_purity‚0.270557701587677 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.14607222378253937 +jet/accuracy_1_of_1‚0.26612600684165955 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5075536966323853 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6520718932151794 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.028758548200130463 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.18936505913734436 & Purity/1FRt1SRqqt2FBt/FBt_purity‚1 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.1696910709142685 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9992441534996033 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.13091814517974854 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.017209414392709732 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5302785038948059 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5091068148612976 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4889547824859619 7validation_average_jet_accuracy‚0.30915895104408264 particle/accuracy_2_of_5‚1 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.07407407462596893 6 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.201080322265625 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.20362897217273712 8"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.541999340057373 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.6831159591674805 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.19371627271175385 'particle/accuracy‚0.835801899433136 jet/accuracy_5_of_5‚0 particle/accuracy_1_of_5‚1 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.5502080917358398 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.6258276104927063 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.009006227366626263 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.18350791931152344 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.5249199867248535 +jet/accuracy_2_of_2‚0.08843416720628738 trainer/global_step‚1999 )loss/val_total_loss‚9.470254627359617 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.49012255668640137 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.20656196773052216 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.14666667580604553 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.19371627271175385 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.20287033915519714 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 particle/accuracy_2_of_4‚1 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9806038737297058 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 *jet/accuracy_2_of_3‚0.3872841000556946 *jet/accuracy_1_of_4‚0.9660916328430176 particle/accuracy_1_of_4‚1 +particle/sensitivity‚0.6118019223213196 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6706400513648987 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.47453704476356506 particle/accuracy_1_of_1‚1 /particle/accuracy_4_of_4‚0.9685434699058533 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9053624272346497@ç §Ø"é Áz 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 6 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.543749988079071 $ _timestamp‚1.784293150293741e+09 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6226603984832764 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7142857313156128 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 particle/accuracy_3_of_3‚1 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.5574635863304138 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.15245017409324646 8"Purity/1FRt0SRqqt1FBt/event_purity‚0.313802570104599 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9966313242912292 ."Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.40625 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.033418554812669754 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 jet/accuracy_2_of_5‚1 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5577569007873535 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9980361461639404 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7960505485534668 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.17816470563411713 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.03309882804751396 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3229166865348816 8"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.550882875919342 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 )jet/accuracy_1_of_3‚0.900711715221405 particle/accuracy_1_of_3‚1 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7142857313156128 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.17180967330932617 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9960675239562988 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.3865740895271301 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9972801208496094 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.19145743548870087 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.29960599541664124 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7889645099639893 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 *jet/accuracy_1_of_2‚0.7317983508110046 jet/accuracy_1_of_5‚1 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9938672184944153 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6343399882316589 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6047952771186829 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 +particle/specificity‚0.8970661163330078 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.18488705158233643 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.1783018857240677 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7916666865348816 particle/accuracy_2_of_2‚1 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7221844792366028 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2654668986797333 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.24916736781597137 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.23390595614910126 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.46795958280563354 ("Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6240332722663879 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.20831939578056335 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8463944792747498 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.4701395034790039 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.14075863361358643 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9933808445930481 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.5398752689361572 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.27894243597984314 jet/accuracy_3_of_5‚1 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.07407407462596893 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.21372097730636597 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.1918787658214569 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 +jet/accuracy_3_of_4‚0.06014878302812576 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.1869795024394989 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.1538461595773697 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5775270462036133 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9976814985275269 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6198742389678955 8"Purity/0FRt0SRqqt*FBt/event_purity‚0.989503026008606 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 jet/accuracy_4_of_5‚0 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.21522071957588196 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.21171638369560242 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.6549904942512512 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.6875 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.06913406401872635 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.5323513746261597 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7916666865348816 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 8"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.522318422794342 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.7882838249206543 9"Purity/2FRt0SRqqt0FBt/event_purity‚0.0171547569334507 +jet/accuracy_3_of_3‚0.04953882098197937 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 particle/accuracy_4_of_5‚1 particle/accuracy_5_of_5‚0 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8259488344192505 6 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.997619092464447 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.20063325762748718 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 'particle/f_score‚0.6154343485832214 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6040421724319458 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 particle/accuracy_2_of_3‚1 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.29501330852508545 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.19774077832698822 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9543450474739075 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6443070769309998 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5075536966323853 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5487446784973145 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6230413913726807 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.17180967330932617 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 /particle/accuracy_3_of_4‚0.9938491582870483 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.5001577734947205 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 & Purity/1FRt1SRqqt*FBt/FBt_purity‚1 particle/accuracy_3_of_5‚1 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7888889312744141 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.2420499920845f1=o0317 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5628164410591125 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9747141599655151 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8008692860603333 ("Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.1538461595773697 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6302083730697632 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9929066896438599 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9911805391311646 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 particle/accuracy_1_of_2‚1 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.8986495733261108 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.6589342355728149 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.625 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9821993708610535 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.49426713585853577 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9890351891517639 epoch‚4 _runtime‚3334.511621382 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9629628658294678 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9771724939346313 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.16794168949127197 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5588107109069824 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9909991025924683 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9972801208496094 _step‚164 6 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.993127703666687 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.06780551373958588 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.17667154967784882 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.6831159591674805 ,jet/accuracy_4_of_4‚0.013899499550461769 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9230769276618958 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.4710586965084076 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.16513125598430634 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5011357069015503 8"Purity/0FRt1SRqqt1FBt/event_purity‚0.270557701587677 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.14607222378253937 +jet/accuracy_1_of_1‚0.26612600684165955 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5075536966323853 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6520718932151794 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.028758548200130463 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.18936505913734436 & Purity/1FRt1SRqqt2FBt/FBt_purity‚1 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.1696910709142685 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9992441534996033 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.13091814517974854 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.017209414392709732 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5302785038948059 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5091068148612976 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4889547824859619 7validation_average_jet_accuracy‚0.30915895104408264 particle/accuracy_2_of_5‚1 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.07407407462596893 6 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.201080322265625 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.20362897217273712 8"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.541999340057373 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.6831159591674805 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.19371627271175385 'particle/accuracy‚0.835801899433136 jet/accuracy_5_of_5‚0 particle/accuracy_1_of_5‚1 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.5502080917358398 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.6258276104927063 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.009006227366626263 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.18350791931152344 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.5249199867248535 +jet/accuracy_2_of_2‚0.08843416720628738 trainer/global_step‚1999 )loss/val_total_loss‚9.470254627359617 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.49012255668640137 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.20656196773052216 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.14666667580604553 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.19371627271175385 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.20287033915519714 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 particle/accuracy_2_of_4‚1 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9806038737297058 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 *jet/accuracy_2_of_3‚0.3872841000556946 *jet/accuracy_1_of_4‚0.9660916328430176 particle/accuracy_1_of_4‚1 +particle/sensitivity‚0.6118019223213196 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6706400513648987 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.47453704476356506 particle/accuracy_1_of_1‚1 /particle/accuracy_4_of_4‚0.9685434699058533 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9053624272346497 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.1571812927722931 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.1362694948911667 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5987222194671631 *jet/accuracy_2_of_4‚0.7156152725219727 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.7996599078178406 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925¤Sj¦}’ê ‚(:‡ ¨ÎèÒúÂýö gpu.0.gpu‚90.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚41.0 gpu.0.powerWatts‚242.96' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚48.592 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0>a°Ûñë ‚(:æ ¨ÎèÒÏ‚¯ù disk.sda1.out‚0& disk.nvme0n1.out‚51593.57861328125) disk./.usagePercent‚37.38797289506919$ disk./.usageGB‚557.1078147888184 disk.rbd3.out‚0( disk.nvme0c0n1.out‚51593.57861328125 disk.dm-1.in‚0 disk.sda3.in‚0 disk.sda.out‚ 119.63671875 cpu‚25.57900169495821 network.recv‚869748 disk.sda3.out‚ 119.63671875 disk.dm-1.out‚ 119.63671875) proc.memory.availableMB‚ 2812.99609375 disk.nvme0n1.in‚ 6628.515625 disk.rbd1.out‚ 8.1640625 disk.rbd0.out‚0* proc.memory.percent‚17.244267463684082 disk.dm-0.in‚ 6628.515625# disk.dm-0.out‚51593.57861328125 disk.rbd3.in‚0 disk.rbd0.in‚0$ memory_percent‚82.83083438873291# proc.memory.rssMB‚ 2825.30078125 network.sent‚ 3.759049e+06! disk.nvme0c0n1.in‚ 6628.515625 disk.drbd2165.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd1.in‚0 disk.sda.in‚0 disk.sda1.in‚0 proc.cpu.threads‚17 disk.drbd2165.out‚0(Q¿r”ì ‚(:‰ ·ÎèÒ›þ‡ø gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚167.439' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.4878 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚06Æ têí ‚(:ß ·ÎèÒ¨ÿ ù! disk.nvme0c0n1.in‚ 6628.515625' proc.memory.availableMB‚ 2812.734375 proc.cpu.threads‚17 disk.sda1.in‚0 disk.rbd3.out‚0$ disk./.usageGB‚557.1133117675781 disk.sda1.out‚0 disk.sda2.out‚0 disk.sda.in‚0" disk.dm-0.out‚51603.2744140625 disk.drbd2165.in‚0' disk.nvme0c0n1.out‚51603.2744140625$ memory_percent‚82.83243179321289" proc.memory.rssMB‚ 2825.4609375( proc.memory.percent‚17.2452449798584 disk.dm-1.out‚ 119.84765625 network.recv‚874405 disk.sda2.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 8.1640625 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.dm-0.in‚ 6628.515625 disk.rbd3.in‚0 cpu‚25.578209792441843 disk.dm-1.in‚0) disk./.usagePercent‚37.38834180192603 disk.drbd2165.out‚0 disk.nvme0n1.in‚ 6628.515625% disk.nvme0n1.out‚51603.2744140625 disk.sda.out‚ 119.84765625 network.sent‚ 3.816307e+06 disk.sda3.in‚0 disk.sda3.out‚ 119.84765625Ê ‰É”î ‚(:‰ ÆÎèÒÃׄø gpu.0.gpu‚90.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚288.968' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚57.7936 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ¨kéï ‚(:Þ ÆÎèÒî´–ù disk.dm-1.out‚ 120.40625& disk.nvme0n1.out‚52581.68115234375 disk.rbd1.in‚0 disk.rbd1.out‚ 8.2265625 disk.sda1.out‚0 disk.sda3.out‚ 120.40625 disk.drbd2165.out‚0" proc.memory.rssMB‚ 2825.4609375 network.recv‚877646 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda2.out‚0( disk./.usagePercent‚37.3884762058815$ disk./.usageGB‚557.1153144836426 disk.sda.out‚ 120.40625 disk.dm-0.in‚ 6740.08203125 disk.rbd3.out‚0 disk.sda2.in‚0 disk.sda.in‚0 disk.dm-1.in‚0( proc.memory.availableMB‚ 2812.2421875 cpu‚25.57727715720354 network.sent‚ 3.821337e+06# disk.dm-0.out‚52581.68115234375 disk.sda1.in‚0# disk.nvme0c0n1.in‚ 6740.08203125( disk.nvme0c0n1.out‚52581.68115234375 proc.cpu.threads‚17! disk.nvme0n1.in‚ 6740.08203125 disk.drbd2165.in‚0 disk.rbd3.in‚0$ memory_percent‚82.83543586730957 disk.sda3.in‚0( proc.memory.percent‚17.2452449798584?”wù”ð ‚(:‰ ÕÎèÒˆù¨÷ gpu.0.gpu‚94.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚46.0 gpu.0.powerWatts‚323.137' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚64.6274 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0õô¢™òñ ‚(:ç ÕÎèÒùë«ù proc.cpu.threads‚17 disk.sda3.in‚0 disk.drbd2165.out‚0 disk.rbd3.in‚0 disk.dm-1.out‚ 120.85546875 disk.rbd0.in‚0# disk.nvme0c0n1.in‚ 6740.08203125 network.recv‚881307 disk.rbd1.in‚0 disk.rbd1.out‚ 8.2265625 disk.dm-0.in‚ 6740.08203125 disk.drbd2165.in‚0' disk.nvme0c0n1.out‚52593.3349609375 network.sent‚ 3.826322e+06) disk./.usagePercent‚37.38880389552533 disk.sda.in‚0 disk.sda1.out‚0 disk.rbd3.out‚0 cpu‚25.576495536623682 disk.sda2.in‚0" proc.memory.rssMB‚ 2825.4609375( proc.memory.percent‚17.2452449798584 disk.dm-1.in‚0 disk.rbd0.out‚0 disk.sda1.in‚0! disk.nvme0n1.in‚ 6740.08203125 disk.sda2.out‚0$ memory_percent‚82.83393383026123$ disk./.usageGB‚557.1201972961426 disk.sda.out‚ 120.85546875% disk.nvme0n1.out‚52593.3349609375" disk.dm-0.out‚52593.3349609375 disk.sda3.out‚ 120.85546875) proc.memory.availableMB‚ 2812.48828125“‰õ/ò ‚(:… äÎèÒ£±Ô÷ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚293.8' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚58.76 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0©„#jèó ‚(:Ý äÎèÒ´€ù disk.sda1.out‚0 disk.sda3.in‚0 disk.drbd2165.out‚0 disk.rbd3.in‚0 disk.rbd1.out‚ 8.2890625 cpu‚25.57558269351715 disk.rbd0.out‚0 disk.dm-0.in‚ 6740.08203125 disk.drbd2165.in‚0 disk.sda2.in‚0$ memory_percent‚82.83326625823975$ disk./.usageGB‚557.1252784729004 disk.sda.in‚0 disk.sda.out‚ 121.1328125# disk.nvme0c0n1.in‚ 6740.08203125) disk./.usagePercent‚37.38914489756094 disk.dm-1.in‚0 disk.sda1.in‚0 disk.rbd1.in‚0# disk.nvme0n1.out‚52602.73828125 disk.sda2.out‚0 disk.rbd0.in‚0 disk.sda3.out‚ 121.1328125 disk.rbd3.out‚0% disk.nvme0c0n1.out‚52602.73828125 network.sent‚ 3.831348e+06 disk.dm-0.out‚52602.73828125) proc.memory.availableMB‚ 2812.59765625" proc.memory.rssMB‚ 2825.4609375 network.recv‚884469 disk.dm-1.out‚ 121.1328125! disk.nvme0n1.in‚ 6740.08203125( proc.memory.percent‚17.2452449798584 proc.cpu.threads‚17ØÚ52ô - _wandbruntime‚3418  _runtime‚3418v‚Çõ Á & lr-AdamW/pg1‚0.0006798354898953483 & lr-AdamW/pg2‚0.0006798354898953483 trainer/global_step‚2049 % _timestamp‚1.7842932338721552e+09 _runtime‚3418.089314259 _step‚165ðâ²çÌö Æ _runtime‚3418.089314259 _step‚165 & lr-AdamW/pg1‚0.0006798354898953483 & lr-AdamW/pg2‚0.0006798354898953483 trainer/global_step‚2049 % _timestamp‚1.7842932338721552e+09¥ ¯C2÷ - _wandbruntime‚3418  _runtime‚3418CüÄî'ø èO \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚588873678327808 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1924162074120 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚35243711 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚591056341314048 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚589128879972352 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1927461341696 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚589128879972352 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5825847 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚161766494772224 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2486645903360 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1925506674442 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚17486701 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚17757010 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚17757010 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚589128906145792 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1927461341696 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚588873703755776 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚11087556 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚17747037 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 400530432 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1928806731264 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚159279816429568 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚32439296 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚17486695 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5825617 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚590799210430218 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2486678342656 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _runtime‚3418.090532749 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚591057712877056 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5261939 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚591057712877056 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚17486701 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚230 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚17747037 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚589128906145792 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 $ _timestamp‚1.784293233873205e+09 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ¢@³¹ù íO SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1928806731264 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚159279816429568 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚32439296 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚17486695 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5825617 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚590799210430218 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2486678342656 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _runtime‚3418.090532749 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚591057712877056 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5261939 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚591057712877056 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚17486701 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚230 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚17747037 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚589128906145792 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 $ _timestamp‚1.784293233873205e+09 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚588873678327808 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1924162074120 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚35243711 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚591056341314048 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚589128879972352 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1927461341696 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚589128879972352 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5825847 ^JDej°›¿:viceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚161766494772224 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2486645903360 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1925506674442 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚17486701 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚17757010 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚17757010 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚589128906145792 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1927461341696 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚588873703755776 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚11087556 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚17747037 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 400530432 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0¦YPo2ú - _wandbruntime‚3419  _runtime‚3419ÄÖAËù&û óM YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚17494901 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚591078015674752 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚35250416 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚17765502 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚11093090 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 _step‚167 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚17494901 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚589406906875904 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚233 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚35260409 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2487859288064 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1926430140800 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚161844794034176 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚589406906875904 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5264487 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5828603 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚2049 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚32432128 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚3419.479238049 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚159357302837248 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚591076645646350 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚589151560105984 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚589406880702464 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚11092857 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1929731773952 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚17755515 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5264482 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 % _timestamp‚1.7842932352618384e+09 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚17494907OÉ_šþ&ü øM UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5264487 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5828603 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚2049 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚32432128 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚3419.479238049 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚159357302837248 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚591076645646350 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚589151560105984 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚589406880702464 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚11092857 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1929731773952 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚17755515 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5264482 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 % _timestamp‚1.7842932352618384e+09 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚17494907 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚17494901 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚591078015674752 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚35250416 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚17765502 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚11093090 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 _step‚167 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚17494901 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚589406906875904 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚233 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚35260409 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2487859288064 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1926430140800 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚161844794034176 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚589406906875904§¢‘ 2ý - _wandbruntime‚3419  _runtime‚3419ÂOçþ € 0loss/FRt2/assignment_loss‚0.9042922956360208 2loss/SRqqt2/assignment_loss‚0.6095407963430486 1loss/FBt1/assignment_loss‚0.08462450951102705 1loss/FBt2/assignment_loss‚0.04171106833180485 1loss/SRqqt1/detection_loss‚1.0175154070061534 /loss/FRt1/detection_loss‚0.5821382976935103 /loss/FBt1/detection_loss‚0.7009524223520547 1loss/SRqqt2/detection_loss‚0.7196187359032171 trainer/global_step‚2049 _runtime‚3419.563819808 %loss/total_loss‚9.401021084777378 2loss/SRqqt1/assignment_loss‚0.6510448110306224 % _timestamp‚1.7842932353468113e+09 /loss/FRt2/detection_loss‚1.6453656492613238 _step‚168 /loss/FBt2/detection_loss‚0.5027160666876291 epoch‚5 /loss/FRt1/assignment_loss‚1.941501025020966#¿Ÿ¼ ÿ … 1loss/SRqqt2/detection_loss‚0.7196187359032171 trainer/global_step‚2049 _runtime‚3419.563819808 %loss/total_loss‚9.401021084777378 2loss/SRqqt1/assignment_loss‚0.6510448110306224 % _timestamp‚1.7842932353468113e+09 /loss/FRt2/detection_loss‚1.6453656492613238 _step‚168 /loss/FBt2/detection_loss‚0.5027160666876291 epoch‚5 /loss/FRt1/assignment_loss‚1.941501025020966 0loss/FRt2/assignment_loss‚0.9042922956360208 2loss/SRqqt2/assignment_loss‚0.6095407963430486 1loss/FBt1/assignment_loss‚0.08462450951102705 1loss/FBt2/assignment_loss‚0.04171106833180485 1loss/SRqqt1/detection_loss‚1.0175154070061534 /loss/FRt1/detection_loss‚0.5821382976935103 /loss/FBt1/detection_loss‚0.7009524223520547¨)4꜀ ‚(:‘ óÎèÒ³ÑÆö gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚279.255' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚55.85099999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚02vŒõ ‚(:ê óÎèÒòÐíø) disk./.usagePercent‚37.38946721104655 disk.sda.in‚0& disk.nvme0n1.out‚52615.67919921875 disk.sda2.out‚0 disk.dm-1.out‚ 121.41796875 disk.rbd0.out‚0# disk.dm-0.out‚52615.67919921875 disk.sda2.in‚0 disk.sda1.out‚0 disk.rbd3.in‚0) proc.memory.availableMB‚ 2812.86328125 disk.sda1.in‚0 disk.drbd2165.in‚0# disk.nvme0c0n1.in‚ 6740.08203125$ memory_percent‚82.83164501190186" proc.memory.rssMB‚ 2825.4609375 disk.dm-1.in‚0 disk.sda3.out‚ 121.41796875! disk.nvme0n1.in‚ 6740.08203125 disk.rbd1.in‚0( proc.memory.percent‚17.2452449798584 cpu‚25.574820187312483$ disk./.usageGB‚557.1300811767578 disk.sda.out‚ 121.41796875 disk.sda3.in‚0 disk.rbd3.out‚0( disk.nvme0c0n1.out‚52615.67919921875 disk.rbd1.out‚ 8.2890625 network.sent‚ 3.836435e+06 network.recv‚887841 disk.rbd0.in‚0 disk.dm-0.in‚ 6740.08203125 disk.drbd2165.out‚0 proc.cpu.threads‚17=½”‚ ‚(:‰ ‚ÏèÒÙéûö gpu.0.gpu‚90.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚272.007' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.4014 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0©dT¹ñƒ ‚(:æ ‚ÏèÒ‘íÜø disk.sda2.in‚0 disk.dm-1.out‚ 121.93359375 disk.drbd2165.out‚0' disk.nvme0c0n1.out‚52623.5771484375 disk.sda1.out‚0 disk.drbd2165.in‚0 network.recv‚892353 disk.dm-1.in‚0 disk.sda1.in‚0 proc.cpu.threads‚17$ disk./.usageGB‚557.1344261169434 disk.sda.in‚0" proc.memory.rssMB‚ 2825.4609375 disk.rbd0.in‚0) disk./.usagePercent‚37.38975880362805 disk.rbd3.in‚0! disk.nvme0n1.in‚ 6980.27734375( proc.memory.percent‚17.2452449798584 cpu‚25.57391925017321# disk.nvme0c0n1.in‚ 6980.27734375) proc.memory.availableMB‚ 1840.41796875 disk.sda3.out‚ 121.93359375% disk.nvme0n1.out‚52623.5771484375 network.sent‚ 3.899266e+06 disk.rbd1.in‚0 disk.rbd1.out‚ 8.4140625 disk.sda.out‚ 121.93359375" disk.dm-0.out‚52623.5771484375 disk.rbd3.out‚0$ memory_percent‚88.76698017120361 disk.rbd0.out‚0 disk.sda2.out‚0 disk.dm-0.in‚ 6980.27734375 disk.sda3.in‚0dPÍkœ„ ‚(:‘ ‘ÏèÒÅ™Õö gpu.0.gpu‚1.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚332.989' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚66.59779999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ôo£âõ… ‚(:ê ‘ÏèÒõ¶¼ø network.sent‚ 3.904301e+06 disk.sda.out‚ 122.20703125 disk.sda1.out‚0 disk.sda3.in‚0( disk.nvme0c0n1.out‚52635.53271484375 disk.sda2.in‚0" proc.memory.rssMB‚ 2825.4609375$ disk./.usageGB‚557.1375617980957 disk.rbd0.out‚0 disk.dm-0.in‚ 6980.27734375 disk.rbd0.in‚0# disk.dm-0.out‚52635.53271484375 disk.sda1.in‚0! disk.nvme0n1.in‚ 6980.27734375 disk.rbd1.in‚0 disk.sda.in‚0 disk.dm-1.out‚ 122.20703125 disk.rbd3.out‚0 cpu‚25.573172030504058 disk.dm-1.in‚0) disk./.usagePercent‚37.38996924182119 disk.sda2.out‚0 disk.rbd1.out‚ 8.4140625( proc.memory.percent‚17.2452449798584 disk.drbd2165.in‚0 disk.rbd3.in‚0$ memory_percent‚88.76588344573975 proc.cpu.threads‚17 network.recv‚895563 disk.sda3.out‚ 122.20703125 disk.drbd2165.out‚0# disk.nvme0c0n1.in‚ 6980.27734375& disk.nvme0n1.out‚52635.53271484375) proc.memory.availableMB‚ 1840.59765625Ÿ˜à”† ‚(:‰  ÏèÒùéîö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚265.067' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.0134 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0LCGXë‡ ‚(:à  ÏèÒ¤‡þø disk.sda3.out‚ 122.453125 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.rbd3.out‚0 disk.dm-1.in‚0$ memory_percent‚88.76590728759766( disk.nvme0c0n1.out‚52645.08642578125 disk.sda2.in‚0 disk.sda.in‚0& proc.memory.availableMB‚ 1840.59375( proc.memory.percent‚17.2452449798584 disk.sda3.in‚0 disk.sda2.out‚0" proc.memory.rssMB‚ 2825.4609375 disk.rbd3.in‚0& disk.nvme0n1.out‚52645.08642578125 disk.rbd1.in‚0 disk.sda.out‚ 122.453125 disk.rbd0.out‚0 disk.sda1.in‚0 network.recv‚898852! disk.nvme0n1.in‚ 6980.27734375 disk.rbd1.out‚ 8.4765625 disk.rbd0.in‚0 disk.dm-0.in‚ 6980.27734375 network.sent‚ 3.909399e+06) disk./.usagePercent‚37.39049840939447$ disk./.usageGB‚557.1454467773438# disk.dm-0.out‚52645.08642578125 disk.sda1.out‚0# disk.nvme0c0n1.in‚ 6980.27734375 proc.cpu.threads‚17 disk.dm-1.out‚ 122.453125 cpu‚25.57227622998023)¼3Þˆ ‚(:“ ¯ÏèÒŸÐýö gpu.0.gpu‚93.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚47.0 gpu.0.powerWatts‚339.413' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚67.88260000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0-U‚Îí‰ ‚(:â ¯ÏèÒÖÍŸù disk.rbd3.in‚0 disk.rbd3.out‚0% disk.nvme0c0n1.out‚52656.16796875 proc.cpu.threads‚17! disk.nvme0n1.in‚ 6980.27734375 disk.sda2.in‚0) proc.memory.availableMB‚ 1840.58203125 network.recv‚902453 disk.dm-0.out‚52656.16796875" proc.memory.rssMB‚ 2825.4609375* disk./.usagePercent‚37.390259554365024 disk.sda1.out‚0 disk.sda3.out‚ 122.68359375 disk.drbd2165.out‚0( proc.memory.percent‚17.2452449798584 disk.sda2.out‚0 disk.rbd1.in‚0 network.sent‚ 3.914434e+06$ disk./.usageGB‚557.1418876647949 disk.sda.out‚ 122.68359375 disk.rbd0.out‚0 disk.sda3.in‚0 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.dm-0.in‚ 6980.27734375 disk.drbd2165.in‚0# disk.nvme0n1.out‚52656.16796875$ memory_percent‚88.76597881317139 disk.sda.in‚0 disk.dm-1.out‚ 122.68359375 cpu‚25.571534558247787 disk.sda1.in‚0# disk.nvme0c0n1.in‚ 6980.27734375 disk.rbd1.out‚ 8.4765625âFžŠ ‚(:“ ¾ÏèÒßýòö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚291.631' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚58.32619999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0òñ1Då‹ ‚(:Ú ¾ÏèÒ©Ê‘ù disk.rbd0.out‚0( proc.memory.availableMB‚ 1840.5859375# disk.dm-0.out‚52664.88037109375 disk.rbd3.in‚0! disk.nvme0n1.in‚ 7061.78515625& disk.nvme0n1.out‚52664.88037109375 disk.rbd1.out‚ 8.5390625$ memory_percent‚88.76595497131348 disk.dm-0.in‚ 7061.78515625 disk.sda3.in‚0 disk.drbd2165.out‚0 disk.sda2.in‚0 disk.sda2.out‚0( proc.memory.percent‚17.2452449798584$ disk./.usageGB‚557.1473159790039) disk./.usagePercent‚37.39062385308625 disk.rbd1.in‚0 proc.cpu.threads‚17 disk.drbd2165.in‚0# disk.nvme0c0n1.in‚ 7061.78515625( disk.nvme0c0n1.out‚52664.88037109375 disk.sda.out‚122.875 disk.dm-1.in‚0" proc.memory.rssMB‚ 2825.4609375 disk.dm-1.out‚122.875 disk.rbd0.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda.in‚0 cpu‚25.570733871609576 network.sent‚ 3.919427e+06 network.recv‚905870 disk.sda3.out‚122.875 disk.rbd3.out‚0šÏžJ2Œ - _wandbruntime‚3496  _runtime‚3496ÆÅHÇ Á % _timestamp‚1.7842933127757516e+09 _runtime‚3496.993962761 _step‚169 & lr-AdamW/pg1‚0.0006791005206151794 & lr-AdamW/pg2‚0.0006791005206151794 trainer/global_step‚2099^ú3^“Ž Æ _runtime‚3496.993962761 _step‚169 & lr-AdamW/pg1‚0.0006791005206151794 & lr-AdamW/pg2‚0.0006791005206151794 trainer/globaZ.©W9l_step‚2099 % _timestamp‚1.7842933127757516e+09©î¸2 - _wandbruntime‚3496  _runtime‚3496èËXFî' èO GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1970399187228 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚605004065313280 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5964863 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚604738166423836 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚36079763 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚11354214 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚604739536452126 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚602767767236608 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1973777272320 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚11353952 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚18172789 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1975122661888 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1971743787550 % _timestamp‚1.7842933127765307e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1973777272320 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 _step‚170 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚163153768722432 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚605004065313280 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚18172789 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1975122661888 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚603028916477952 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚18182762 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚165701148217344 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚17896995 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚36069784 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5965120 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚257 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚165700745589760 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚605002693750272 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5389089 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5389094 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚603028942651392 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2547011403776 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 _runtime‚ 3496.99406858 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚17897001 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚17897001 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚603028942651392 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚163154136813568 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 trainer/global_step‚2099 I.Yó'‘ íO MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚605004065313280 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚18172789 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1975122661888 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚603028916477952 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚18182762 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚165701148217344 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚17896995 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚36069784 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5965120 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚257 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚165700745589760 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚605002693750272 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5389089 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5389094 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚603028942651392 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2547011403776 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 _runtime‚ 3496.99406858 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚17897001 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚17897001 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚603028942651392 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚163154136813568 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 trainer/global_step‚2099 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1970399187228 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚605004065313280 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5964863 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚604738166423836 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚36079763 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚11354214 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚604739536452126 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚602767767236608 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1973777272320 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚11353952 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚18172789 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1975122661888 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1971743787550 % _timestamp‚1.7842933127765307e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1973777272320 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 _step‚170 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚163153768722432 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0ªqö´2’ - _wandbruntime‚3498  _runtime‚3498d–Ìù&“ óM ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚603045674442752 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 _runtime‚3498.380136304 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚18181267 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚36096461 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚11359456 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚605281619515904 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚36096461 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2548211051520 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚603306943381504 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5391637 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5391632 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5967819 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1974702307840 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚17905207 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 _step‚171 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚36086468 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚253 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚603045649014784 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚18191254 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚605282991086080 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚603306943381504 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚165779834272768 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚603306917208064 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚17905201 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚18181267 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1972667253908 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1971322653474 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1976047704576 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚248 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1974702307840 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚165779431652352 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚603306917208064 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1976047704576 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚18191254 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 % _timestamp‚1.7842933141626174e+09 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚11359203 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚163231623221248 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚605018341696660 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚253 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚603045649014784 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚18191254 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚605282991086080 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚603306943381504 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_b ô…=¦ytes.all.allocated‚165779834272768 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚603306917208064 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚17905201 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚18181267 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1972667253908 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1971322653474 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1976047704576 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚248 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1974702307840 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚165779431652352 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚603306917208064 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1976047704576 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚18191254 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 % _timestamp‚1.7842933141626174e+09 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚11359203 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚163231623221248 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚605018341696660 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚36096461 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚11359456 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚605281619515904 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚36096461 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2548211051520 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚603306943381504 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5391637 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5391632 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5967819 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1974702307840«Š7Þ2• - _wandbruntime‚3498  _runtime‚3498YИ– € 1loss/SRqqt2/detection_loss‚0.6768067431331584 .loss/FBt1/detection_loss‚1.113228634931147 % _timestamp‚1.7842933142492566e+09 0loss/FRt1/assignment_loss‚1.9513852351751082 2loss/SRqqt2/assignment_loss‚0.5627872597944501 _runtime‚3498.466245546 %loss/total_loss‚9.453108064252909 1loss/FBt1/assignment_loss‚0.03501159935453955 1loss/FBt2/assignment_loss‚0.01560627287661408 /loss/FRt1/detection_loss‚0.5642093766674829 /loss/FRt2/detection_loss‚1.5954727122174275 1loss/SRqqt1/detection_loss‚0.9325092662283574 _step‚172 0loss/FRt2/assignment_loss‚0.9858395301777384 /loss/FBt2/detection_loss‚0.4621589627243917 epoch‚5 trainer/global_step‚2099 2loss/SRqqt1/assignment_loss‚0.5580924709724924&Í3! — … 0loss/FRt1/assignment_loss‚1.9513852351751082 2loss/SRqqt2/assignment_loss‚0.5627872597944501 _runtime‚3498.466245546 %loss/total_loss‚9.453108064252909 1loss/FBt1/assignment_loss‚0.03501159935453955 1loss/FBt2/assignment_loss‚0.01560627287661408 /loss/FRt1/detection_loss‚0.5642093766674829 /loss/FRt2/detection_loss‚1.5954727122174275 1loss/SRqqt1/detection_loss‚0.9325092662283574 _step‚172 0loss/FRt2/assignment_loss‚0.9858395301777384 /loss/FBt2/detection_loss‚0.4621589627243917 epoch‚5 trainer/global_step‚2099 2loss/SRqqt1/assignment_loss‚0.5580924709724924 1loss/SRqqt2/detection_loss‚0.6768067431331584 .loss/FBt1/detection_loss‚1.113228634931147 % _timestamp‚1.7842933142492566e+09¬o‡ •˜ ‚(:Š ÍÏèÒ­Ú³÷ gpu.0.gpu‚100.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚47.0 gpu.0.powerWatts‚339.166' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚67.8332 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ýš‘9î™ ‚(:ã ÍÏèÒâøóø% disk.nvme0n1.out‚52676.4189453125" proc.memory.rssMB‚ 2825.4609375 disk.rbd0.in‚0( proc.memory.availableMB‚ 1840.3046875( proc.memory.percent‚17.2452449798584* disk./.usagePercent‚37.390981239604045 disk.dm-0.in‚ 7061.78515625 disk.sda2.in‚0 disk.drbd2165.out‚0 proc.cpu.threads‚17 network.recv‚909514$ disk./.usageGB‚557.1526412963867 disk.sda1.in‚0 disk.sda3.in‚0 disk.rbd1.in‚0 disk.dm-1.in‚0 disk.rbd3.out‚0 disk.rbd1.out‚ 8.5390625$ memory_percent‚88.76767158508301 cpu‚25.56994183700252 disk.sda1.out‚0 disk.sda2.out‚0 disk.dm-1.out‚ 123.4296875 disk.rbd0.out‚0 disk.sda3.out‚ 123.4296875 disk.drbd2165.in‚0 disk.rbd3.in‚0' disk.nvme0c0n1.out‚52676.4189453125! disk.nvme0n1.in‚ 7061.78515625 network.sent‚ 3.924425e+06 disk.sda.in‚0 disk.sda.out‚ 123.4296875" disk.dm-0.out‚52676.4189453125# disk.nvme0c0n1.in‚ 7061.78515625õÇÞžš ‚(:“ ÜÏèÒÜΨ÷ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚278.811' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚55.76219999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0—²Æì› ‚(:á ÜÏèÒŽŸéø$ disk./.usageGB‚557.1585922241211 disk.dm-1.out‚ 124.1953125 disk.sda3.in‚0! disk.nvme0n1.in‚ 7061.78515625 network.sent‚ 3.98739e+06 network.recv‚913996 disk.sda1.out‚0# disk.nvme0c0n1.in‚ 7061.78515625' disk.nvme0c0n1.out‚52685.4208984375$ memory_percent‚88.76581192016602' proc.memory.availableMB‚ 1840.609375( proc.memory.percent‚17.2452449798584 disk.dm-1.in‚0 disk.sda2.out‚0 cpu‚25.569083642702086 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.drbd2165.out‚0% disk.nvme0n1.out‚52685.4208984375 disk.rbd1.out‚ 8.6015625) disk./.usagePercent‚37.39138061135746" disk.dm-0.out‚52685.4208984375 disk.drbd2165.in‚0 disk.sda2.in‚0 disk.sda.in‚0 disk.sda.out‚ 124.1953125 disk.dm-0.in‚ 7061.78515625 disk.sda1.in‚0 disk.sda3.out‚ 124.1953125 disk.rbd3.out‚0 disk.rbd1.in‚0" proc.memory.rssMB‚ 2825.4609375 disk.rbd3.in‚0ç$ãïžœ ‚(:“ ëÏèÒ°ùîö gpu.0.gpu‚79.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚332.989' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚66.59779999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÌÖ½ê ‚(:ß ëÏèÒùˆù disk.rbd1.out‚ 8.6015625 disk.sda.out‚ 124.546875 disk.rbd0.in‚0 disk.drbd2165.out‚0 disk.sda2.in‚0 disk.dm-0.in‚ 7061.78515625' disk.nvme0c0n1.out‚52694.5712890625" proc.memory.rssMB‚ 2825.4609375* disk./.usagePercent‚37.392163738404705 disk.sda3.in‚0 disk.sda3.out‚ 124.546875! disk.nvme0n1.in‚ 7061.78515625# memory_percent‚88.7669563293457 disk.sda2.out‚0 disk.rbd0.out‚0" disk.dm-0.out‚52694.5712890625 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.rbd1.in‚0 disk.dm-1.in‚0 disk.rbd3.out‚0# disk.nvme0c0n1.in‚ 7061.78515625 proc.cpu.threads‚17 disk.dm-1.out‚ 124.546875 disk.sda1.in‚0$ disk./.usageGB‚557.1702613830566 disk.sda.in‚0 disk.rbd3.in‚0% disk.nvme0n1.out‚52694.5712890625' proc.memory.availableMB‚ 1840.421875( proc.memory.percent‚17.2452449798584 network.sent‚ 3.992385e+06 network.recv‚917568 cpu‚25.568438287396727H ”ž ‚(:‰ úÏèÒüÖóø gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚299.456' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚59.8912 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0°iäŸ ‚(:Ù úÏèÒìÓû disk.sda3.out‚ 124.8515625 disk.sda2.out‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 124.8515625 disk.drbd2165.in‚0 disk.drbd2165.out‚0% disk.nvme0c0n1.out‚52703.80859375) proc.memory.availableMB‚ 1840.41796875 network.recv‚920832 disk.dm-0.out‚52703.80859375 disk.sda1.in‚0 disk.rbd3.out‚0$ disk./.usageGB‚557.1756973266602" disk.nvme0c0n1.in‚ 7104.2421875# disk.nvme0n1.out‚52703.80859375$ memory_percent‚88.76698017120361' disk./.usagePercent‚37.392528549141 disk.sda3.in‚0 disk.rbd1.out‚ 8.6640625 cpu‚25.567561404137287 network.sent‚ 3.997421e+06 disk.rbd3.in‚0 disk.sda2.in‚0 disk.rbd0.out‚0 proc.cpu.threads‚17 disk.sda1.out‚0 disk.sda.in‚0 disk.sda.out‚ 124.8515625 disk.rbd0.in‚0" proc.memory.rssMB‚ 2825.4609375( proc.memory.percent‚17.2452449798584 disk.dm-0.in‚ 7104.2421875 disk.nvme0n1.in‚ 7104.2421875 disk.rbd1.in‚0.h󈓠 ‚(:ˆ ‰ÐèÒŸ´´÷ gpu.0.gpu‚99.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚294.815' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚58.963 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 »ZÑí¡ ‚(:â ‰ÐèÒ‡¶Éø( proc.memory.availableMB‚ 1840.9453125) disk./.usagePercent‚37.39288311957591 disk.dm-1.in‚0 disk.rbd0.out‚0 disk.drbd2165.out‚0$ memory_percent‚88.76376152038574" proc.memory.rssMB‚ 2825.4609375# disk./.usageGB‚557.180980682373 disk.dm-0.in‚ 7230.0859375 disk.sda3.out‚ 125.34765625 disk.sda2.in‚0 disk.sda2.out‚0 network.sent‚ 4.000958e+06 disk.sda.out‚ 125.34765625 disk.sda1.in‚0 disk.drbd2165.in‚0% disk.nvme0n1.out‚52715.1416015625 disk.sda.in‚0 disk.sda1.out‚0' disk.nvme0c0n1.out‚52715.1416015625 proc.cpu.threads‚17" disk.dm-0.out‚52715.1416015625 disk.rbd1.in‚0 disk.rbd1.out‚ 8.6640625 cpu‚25.566900254584752 disk.dm-1.out‚ 125.34765625 disk.rbd3.in‚0 network.recv‚923619 disk.rbd3.out‚0" disk.nvme0c0n1.in‚ 7230.0859375( proc.memory.percent‚17.2452449798584 disk.rbd0.in‚0 disk.sda3.in‚0 disk.nvme0n1.in‚ 7230.0859375nÁCù2¢ - _wandbruntime‚3575  _runtime‚3575-NjÄ£ ¾ & lr-AdamW/pg1‚0.0006783441520441945 & lr-AdamW/pg2‚0.0006783441520441945 trainer/global_step‚2149 $ _timestamp‚1.784293391334261e+09 _runtime‚ 3575.5514521 _step‚173ë۬ɤ à & lr-AdamW/pg1‚0.0006783441520441945 & lr-AdamW/pg2‚0.0006783441520441945 trainer/global_step‚2149 $ _timestamp‚1.784293391334261e+09 _runtime‚ 3575.5514521 _step‚173­•)ã2¥ - _wandbruntime‚3575  _runtime‚3575$ã»*î'¦ èO QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚11619140 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚616928979156992 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚256 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚169635255824384 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6102896 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5516239 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚18307295 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚18307301 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚18307295 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚618949046186496 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚618679862474034 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 $ _timestamp‚1.784293391335365e+09 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚18307301 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚169634853196800 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚18608514 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5516244 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚618950417749504 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚618949046186496 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚261 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚18608514 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚11618879 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚167028457197568 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2020093202944 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 _runtime‚3575.556073224 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0´8]§ íO VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚18608514 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚11618879 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚167028457197568 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2020093202944 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 _runtime‚3575.556073224 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚11619140 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚616928979156992 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚256 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚169635255824384 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6102896 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5516239 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚18307295 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚18307301 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚18307295 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚618949046186496 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚618679862474034 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 $ _timestamp‚1.784293391335365e+09 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚18307301 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚169634853196800 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚18608514 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5516244 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚618950417749504 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚618949046186496 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚261 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572®gèÅ«2¨ - _wandbruntime‚3576  _runtime‚3576‰Å÷Ýù&© óM =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2021018238464 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚257 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚18607019 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚619227971952128 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚18617006 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2607960729600 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚618958667718568 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚169713536243712 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚616939763351552 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚617206953713664 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6105647 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2607995258880 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚36932513 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚167105575514112 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2022363635200 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚617206979887104 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚18315501 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚36932513 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚18315507 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚617206979887104 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚11624434 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚169713938864128 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚619229343522304 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚262 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚18315507 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5518782 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚616939737923584 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚617206953713664 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2022363635200 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 trainer/global_step‚2149 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚36922520 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚619229343522304 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚11624172 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6105390 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5518787 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚36922520 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚167105943605248 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚18617006 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 _runtime‚3576.937388037 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2021018238464ñ ÈÀþ&ª øM WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2017559766582 % _timestamp‚1.7842933927199852e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2022363635200 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚617206979887104 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚18315501 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚36932513 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚18315507 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚617206979887104 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚11624434 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚169713938864128 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚619229343522304 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚262 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚18315507 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5518782 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚616939737923584 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚617206953713664 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2022363635200 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 trainer/global_step‚2149 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚36922520 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚619229343522304 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚11624172 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6105390 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5518787 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚36922520 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚167105943605248 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚18617006 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 _runtime‚3576.937388037 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2021018238464 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2021018238464 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚257 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚18607019 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚619227971952128 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚18617006 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2607960729600 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚618958667718568 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚169713536243712 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚616939763351552 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚617206953713664 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6105647 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2607995258880 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚36932513 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚167105575514112 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 EDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚37741888 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6240405 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚630828989489152 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚254 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚173569027317760 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚632895398622720 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _runtime‚3654.888129469 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚19034266 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚170902409490432 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2666650266624 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚630829015662592 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2062873413444 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚632895398622720 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6240156 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚630555945054208 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2066409133568 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5643394 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚632618818467652 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚18717595 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 _step‚178 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚173569427848192çÚüûô'½ îO `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚630555970482176 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚18717595 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚630828989489152 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2067754523136 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚18717601 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚37741888 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6240405 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚630828989489152 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚254 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚173569027317760 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚632895398622720 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _runtime‚3654.888129469 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚19034266 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚170902409490432 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2666650266624 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚630829015662592 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2062873413444 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚632895398622720 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6240156 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚630555945054208 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2066409133568 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5643394 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚632618818467652 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚18717595 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 _step‚178 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚173569427848192 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚11883799 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚249 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚170902777581568 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 trainer/global_step‚2199 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚37758572 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2067334169088 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚632897623712074 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5645932 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚11888844 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚234 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚173648113674240 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 " _timestamp‚1.7842934720597e+09 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2667817252864 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2065141480124 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚37768565 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚19032771 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2063796879690 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚3656.277241904 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚19032771 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6242912 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚11889083 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2667849684992 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚37768565 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2067334169088 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚239 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5645937 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 400523264 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚631107016392704 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 _step‚179 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚18725807 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚18725801 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 trainer/global_step‚2199 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚631106990219264 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚18725801 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚19042758 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚170979895898112 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2068679565824 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚630833826832384ZÁ~^À õM ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2068679565824 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚630833826832384 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚631106990219264 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2068679565824 GoÝÅl#:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚37758572 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6243146 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚630833852260352 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚37758572 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2067334169088 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚632897623712074 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5645932 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚11888844 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚234 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚173648113674240 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 " _timestamp‚1.7842934720597e+09 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2667817252864 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2065141480124 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚37768565 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚19032771 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2063796879690 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚3656.277241904 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚19032771 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6242912 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚11889083 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2667849684992 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚37768565 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2067334169088 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚239 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5645937 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 400523264 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚631107016392704 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 _step‚179 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚18725807 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚18725801 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 trainer/global_step‚2199 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚631106990219264 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚18725801 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚19042758 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚170979895898112 BÈçÍ ‚(:Ü ŸÑèÒÚì—ù disk.rbd1.out‚ 8.9765625$ memory_percent‚88.76936435699463 disk.dm-1.in‚0 disk.rbd3.in‚0( disk.nvme0c0n1.out‚52843.76904296875* disk./.usagePercent‚37.396813859256646# disk.nvme0c0n1.in‚ 7413.96484375 disk.sda2.out‚0 cpu‚25.571891181205505 disk.sda.in‚0 network.sent‚ 4.175305e+06$ disk./.usageGB‚557.2395515441895 disk.rbd0.out‚0# disk.dm-0.out‚52843.76904296875& disk.nvme0n1.out‚52843.76904296875 network.recv‚975171( proc.memory.percent‚17.2452449798584 disk.drbd2165.in‚0) proc.memory.availableMB‚ 1840.02734375" proc.memory.rssMB‚ 2825.4609375 disk.sda.out‚135.375 disk.sda3.in‚0 disk.sda3.out‚135.375 disk.rbd3.out‚0! disk.nvme0n1.in‚ 7413.96484375 disk.rbd1.in‚0 disk.dm-0.in‚ 7413.96484375 disk.sda1.in‚0 disk.sda1.out‚0 proc.cpu.threads‚17 disk.dm-1.out‚135.375 disk.rbd0.in‚0 disk.drbd2165.out‚0 disk.sda2.in‚0ò¤2Î - _wandbruntime‚3734  _runtime‚3734}ƒùžÅÏ ¿ & lr-AdamW/pg1‚0.0006767674127216521 & lr-AdamW/pg2‚0.0006767674127216521 trainer/global_step‚2249 % _timestamp‚1.7842935501877415e+09 _runtime‚ 3734.4049163 _step‚181š°_ÁÊÐ Ä _step‚181 & lr-AdamW/pg1‚0.0006767674127216521 & lr-AdamW/pg2‚0.0006767674127216521 trainer/global_step‚2249 % _timestamp‚1.7842935501877415e+09 _runtime‚ 3734.4049163µjx&‡2Ñ - _wandbruntime‚3734  _runtime‚3734-7W%ï'Ò éO E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚174777097965568 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚19127901 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2114070453760 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚644450033963008 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚12148911 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6378367 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚19127895 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2109110526552 _runtime‚3734.406147424 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚646560514517850 trainer/global_step‚2249 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚177503493414912 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚238 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2112725064192 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚644729025994752 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚233 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2112725064192 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚38587919 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5770539 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6378134 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 400530432 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚646843122621952 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2114070453760 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚12148673 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚38577940 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚644729025994752 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚32439296 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 % _timestamp‚1.7842935501888022e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚19450045 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2110455126874 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚19450045 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚644729052168192 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚177503893945344 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2726763540480 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚19460018 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚644729052168192 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚646841751058944 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚19460018 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5770544 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2726795979776 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚646559144489560 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚19127895 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚19127901 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚644450059390976 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚174776729874432 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0òä4RÓ îO TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚182 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚644729052168192 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚646841751058944 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚19460018 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5770544 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2726795979776 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚646559144489560 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚19127895 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚19127901 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚644450059390976 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚174776729874432 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚174777097965568 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚19127901 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2114070453760 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚644450033963008 UFÞXð—¢DeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚12148911 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6378367 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚19127895 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2109110526552 _runtime‚3734.406147424 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚646560514517850 trainer/global_step‚2249 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚177503493414912 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚238 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2112725064192 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚644729025994752 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚233 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2112725064192 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚38587919 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5770539 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6378134 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 400530432 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚646843122621952 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2114070453760 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚12148673 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚38577940 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚644729025994752 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚32439296 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 % _timestamp‚1.7842935501888022e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚19450045 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2110455126874 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚19450045 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚644729052168192 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚177503893945344 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2726763540480 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚19460018 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 IDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚38594624 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚645007026724864 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5773087 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 _runtime‚3735.790652573 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚38604617 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚19468510 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2113650099712 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚647120676824576 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚19458523 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚12154196 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2110033992798 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚19136107 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚647122048394752 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚645007052898304 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2728001607680 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚19468510 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚644727915741184 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 _step‚183 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6381109 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12153968 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2727969175552 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚644727941169152 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 400523264 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚38604617 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚19136107 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚19136101 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚38594624 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2114995496448 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚645007026724864 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 trainer/global_step‚2249 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚19136101 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚38594624 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚645007026724864 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5773087 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 _runtime‚3735.790652573 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚38604617 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚19468510 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2113650099712 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚647120676824576 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚19458523 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚12154196 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2110033992798 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚19136107 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚647122048394752 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚645007052898304 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2728001607680 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚19468510 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚644727915741184 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 _step‚183 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6381109 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12153968 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2727969175552 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚644727941169152 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 400523264 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚38604617 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚19136107 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚19136101 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚38594624 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2114995496448 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚645007026724864 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 trainer/global_step‚2249 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚19136101 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚658629062500352 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚181438415413248 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚232 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚660788103495168 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚658629088673792 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2159040994816 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2159040994816 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚32439296 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 trainer/global_step‚2299 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚19885770 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2160386384384 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2786997063680 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2160386384384 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 % _timestamp‚1.7842936291768453e+09 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚19538201 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚19538201 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚19875797 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚19538195 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚178651050258432 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚19875797 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5897694 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2155347639660 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚19538195 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6516344 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 400530432 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚658344122871808 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚39413992 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚660500840539758 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚658344148299776 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚19885770 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚12414270 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2160386384384 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 % _timestamp‚1.7842936291768453e+09 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚19538201 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚19538201 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚19875797 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚19538195 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚178651050258432 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚19875797 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5897694 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2155347639660 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚19538195 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6516344 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 400530432 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚658344122871808 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚39413992 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚660500840539758 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚658344148299776 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚19885770 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚12414270 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚658629062500352 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚181438415413248 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚232 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚660788103495168 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚658629088673792 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2159040994816 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2159040994816 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚32439296 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 trainer/global_step‚2299 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚19885770 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154º¬ÒïŠ2ì - _wandbruntime‚3814  _runtime‚3814<åK— í òM HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2156271105906 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚658907089403904 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2159966030336 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚661068400830976 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚226 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚32432128 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚660779645784292 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚658622030077952 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚658622004649984 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚658907063230464 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2159966030336 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚19884275 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚658907063230464 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2161311427072 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2157615706340 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 trainer/global_step‚2299 _runtime‚3814.785675851 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚19546401 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚39440669 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚19546401 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚19894262 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2788165426176 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚12419569 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6519332 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12419338 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 $ _timestamp‚1.784293630568224e+09 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5900232 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚19894262 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2161311427072 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚660778275755890 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚19546407 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6R¬Äóý&î ÷M K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚178728536666112 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 $ _timestamp‚1.784293630568224e+09 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5900232 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚19894262 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2161311427072 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚660778275755890 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚19546407 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2156271105906 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚658907089403904 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2159966030336 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚661068400830976 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚226 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚32432128 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚660779645784292 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚658622030077952 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚658622004649984 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚658907063230464 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2159966030336 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚19884275 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚658907063230464 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2161311427072 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2157615706340 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 trainer/global_step‚2299 _runtime‚3814.785675851 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚19546401 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚39440669 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚19546401 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚19894262 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2788165426176 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚12419569 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6519332 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12419338 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 EDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2205356925440 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚185372949976064 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2847211242496 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚674734455931392 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚185372549445632 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚672529099005952 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6654158 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚32439296 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2206702315008 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6653925 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚182525738733568 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚233 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 400530432 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 trainer/global_step‚2349 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚12678764 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚674441166561666 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6024844 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚182525370642432 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚672238237208576 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚672529125179392 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚12679002 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚40250044 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚19948495 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2205356925440 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚20301549 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚674735827494400 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚19948495 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚672529125179392 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2847178803200 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚674735827494400 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚672529099005952 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚674439796533376 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚672238211780608 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚20311522 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚674735827494400 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚19948495 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚672529125179392 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2847178803200 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚674735827494400 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚672529099005952 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚674439796533376 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚672238211780608 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚20311522 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2205356925440 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚185372949976064 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2847211242496 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚674734455931392 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚185372549445632 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚672529099005952 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6654158 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚32439296 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2206702315008 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6653925 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚182525738733568 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚233 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 400530432 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 trainer/global_step‚2349 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚12678764 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚674441166561666 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6024844 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚182525370642432 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚672238237208576¾DÈâô2‚ - _wandbruntime‚3893  _runtime‚3893E¬ÐQù&ƒ óM REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚20310027 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚230 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2203852819448 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚40266728 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 400523264 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚185451237482496 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2206281960960 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2206281960960 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2848380432384 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 _step‚191 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚40276721 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2202508219014 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚20320014 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2207627357696 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2207627357696 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚32432128 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚672516118986752 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚19956701 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚12684277 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚40276721 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚672807125909504 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚19956701 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚19956707 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚19956707 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚182603225141248 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚672516093558784 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚672807099736064 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚675013381697024 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚674719971806200 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚675014753267200 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12684047 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6027387 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚672807125909504 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 % _timestamp‚1.7842937096247847e+09 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚20310027 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚225 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚675014753267200 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚185451638005760 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6656890 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚40266728 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2848412864512 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 _runtime‚3893.842179943 trainer/global_step‚2349 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚672807099736064 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12684047 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6027387 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚672807125909504 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 % _timestamp‚1.7842937096247847e+09 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚20310027 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚225 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚675014753267200 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚185451638005760 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6656890 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚40266728 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2848412864512 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 _runtime‚3893.842179943 trainer/global_step‚2349 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚672807099736064 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚40266728 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 400523264 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚185451237482496 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2206281960960 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2206281960960 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2848380432384 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 _step‚191 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚40276721 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2202508219014 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚20320014 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2207627357696 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2207627357696 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚32432128 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚672516118986752 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚19956701 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚12684277 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚40276721 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚672807125909504 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚19956701 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚19956707 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚19956707 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚182603225141248 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚672516093558784 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚672807099736064 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚675013381697024 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚674719971806200 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚675014753267200 P1Ÿˆ ‚(:” ÓÒèÒ Õñö gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚281.431' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚56.286199999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚01¥ôçñ‰ ‚(:æ ÓÒèÒµÇ×ø disk.sda2.in‚0 disk.dm-1.out‚ 154.6484375 disk.rbd0.out‚0 disk.sda1.in‚0 disk.rbd3.in‚0) proc.memory.availableMB‚ 1840.14453125 network.recv‚ 1.019073e+06( proc.memory.percent‚17.2452449798584 proc.cpu.threads‚17* disk./.usagePercent‚37.400698517581205# disk.nvme0c0n1.in‚ 7775.77734375! disk.nvme0n1.in‚ 7775.77734375 disk.dm-0.in‚ 7775.77734375 disk.drbd2165.out‚0 disk.rbd3.out‚0 disk.sda2.out‚0 disk.sda.out‚ 154.6484375" proc.memory.rssMB‚ 2825.4609375& disk.nvme0c0n1.out‚53115.818359375! disk.dm-0.out‚53115.818359375 disk.drbd2165.in‚0 cpu‚25.56974658784755 disk.rbd1.in‚0 disk.dm-1.in‚0$ memory_percent‚88.76864910125732# disk./.usageGB‚557.297435760498$ disk.nvme0n1.out‚53115.818359375 disk.rbd1.out‚ 9.4140625 disk.sda.in‚0 disk.sda3.in‚0 network.sent‚ 4.386933e+06 disk.rbd0.in‚0 disk.sda1.out‚0 disk.sda3.out‚ 154.6484375ºŸTi”Š ‚(:‰ âÒèÒŽ©÷ gpu.0.gpu‚99.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚337.922' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚67.5844 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‚q`ûÏ‹ ‚(:ë âÒèÒˆÕéù disk.sda2.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 155.3828125 disk.rbd0.in‚0 disk.sda3.out‚ 155.3828125 disk.rbd3.out‚0 disk.rbd1.in‚0$ disk./.usageGB‚557.3022499084473 disk.dm-0.in‚ 7775.77734375 disk.sda1.out‚0( proc.memory.percent‚17.2452449798584 network.recv‚ 1.023495e+06# disk.dm-0.out‚53126.61474609375 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.sda.in‚0 diMóÐò'sk.sda.out‚ 155.3828125 network.sent‚ 4.449831e+06 disk.sda1.in‚0( disk.nvme0c0n1.out‚53126.61474609375# memory_percent‚88.7688159942627 disk.sda3.in‚0! disk.nvme0n1.in‚ 7775.77734375& disk.nvme0n1.out‚53126.61474609375 disk.rbd0.out‚0" proc.memory.rssMB‚ 2825.4609375) disk./.usagePercent‚37.40102159908942# disk.nvme0c0n1.in‚ 7775.77734375 disk.sda2.out‚0 proc.cpu.threads‚17 disk.rbd3.in‚0 disk.rbd1.out‚ 9.4765625( proc.memory.availableMB‚ 1840.1171875 cpu‚25.569148503113052ÓæFÊŸŒ ‚(:” ñÒèÒƲöö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚273.894' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚54.778800000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Jw¡`í ‚(:â ñÒèÒ¶ÿ—ù disk.rbd3.in‚0! disk.nvme0c0n1.in‚ 7852.765625& disk.nvme0c0n1.out‚53141.099609375 disk.dm-0.in‚ 7852.765625 disk.dm-1.out‚ 156.16015625 network.sent‚ 4.454824e+06 disk.rbd0.in‚0! disk.dm-0.out‚53141.099609375$ disk.nvme0n1.out‚53141.099609375 disk.rbd1.out‚ 9.4765625 disk.sda1.out‚0 disk.drbd2165.out‚0 disk.rbd3.out‚0 disk.sda.in‚0( proc.memory.percent‚17.2452449798584) disk./.usagePercent‚37.40128247076681 disk.drbd2165.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd1.in‚0 disk.dm-1.in‚0$ memory_percent‚88.76833915710449" proc.memory.rssMB‚ 2825.4609375 disk.sda1.in‚0 disk.sda3.in‚0 disk.sda.out‚ 156.16015625( proc.memory.availableMB‚ 1840.1953125 proc.cpu.threads‚17$ disk./.usageGB‚557.3061370849609 disk.sda3.out‚ 156.16015625 cpu‚25.56838838066561 network.recv‚ 1.027039e+06 disk.rbd0.out‚0 disk.nvme0n1.in‚ 7852.765625ä1¢Q”Ž ‚(:‰ €ÓèÒ‹àö gpu.0.gpu‚90.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚290.031' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚58.0062 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 T"ºò ‚(:ç €ÓèÒí×÷ø! disk.nvme0c0n1.in‚ 7852.765625 disk.nvme0n1.in‚ 7852.765625 disk.rbd0.in‚0 disk.sda1.out‚0 disk.sda3.out‚ 156.33203125 disk.drbd2165.in‚0 disk.sda.in‚0" proc.memory.rssMB‚ 2825.4609375 network.sent‚ 4.45987e+06" disk.dm-0.out‚53149.2744140625) proc.memory.availableMB‚ 1840.43359375 disk.sda.out‚ 156.33203125 disk.rbd3.in‚0* disk./.usagePercent‚37.401708723311316$ disk./.usageGB‚557.3124885559082 disk.dm-0.in‚ 7852.765625 disk.drbd2165.out‚0' disk.nvme0c0n1.out‚53149.2744140625 disk.sda2.in‚0 network.recv‚ 1.030397e+06 disk.sda3.in‚0% disk.nvme0n1.out‚53149.2744140625 disk.sda2.out‚0 disk.rbd1.in‚0 disk.dm-1.out‚ 156.33203125( proc.memory.percent‚17.2452449798584 cpu‚25.567818317860176 disk.rbd3.out‚0 disk.dm-1.in‚0$ memory_percent‚88.76688480377197 proc.cpu.threads‚17 disk.rbd1.out‚ 9.5390625 disk.rbd0.out‚0 disk.sda1.in‚0P¦ˆÙ” ‚(:‰ ÓèÒˆÃö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚40.0 gpu.0.powerWatts‚184.439' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚36.8878 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0uì›eé‘ ‚(:Þ Óèҟϯù! disk.dm-0.out‚53162.330078125 disk.rbd3.out‚0 disk.nvme0n1.in‚ 7852.765625 disk.dm-1.out‚ 156.6953125 disk.sda3.out‚ 156.6953125 disk.sda2.in‚0 disk.rbd1.out‚ 9.5390625( proc.memory.percent‚17.2452449798584 network.recv‚ 1.033957e+06* disk./.usagePercent‚37.402071229979796 disk.rbd0.in‚0 disk.dm-0.in‚ 7852.765625 disk.sda1.out‚0 disk.sda2.out‚0 disk.rbd1.in‚0 disk.sda.in‚0 disk.drbd2165.in‚0! disk.nvme0c0n1.in‚ 7852.765625 disk.sda1.in‚0 disk.drbd2165.out‚0 disk.rbd3.in‚0& disk.nvme0c0n1.out‚53162.330078125 disk.sda.out‚ 156.6953125# memory_percent‚88.7685775756836 cpu‚25.566992276658127 proc.cpu.threads‚17 network.sent‚ 4.464855e+06 disk.sda3.in‚0& proc.memory.availableMB‚ 1840.15625" proc.memory.rssMB‚ 2825.4609375$ disk./.usageGB‚557.3178901672363 disk.dm-1.in‚0$ disk.nvme0n1.out‚53162.330078125 disk.rbd0.out‚0€'[2’ - _wandbruntime‚3970  _runtime‚3970ó×\ôÆ“ À & lr-AdamW/pg1‚0.0006742430680600047 & lr-AdamW/pg2‚0.0006742430680600047 trainer/global_step‚2399 $ _timestamp‚1.784293786742402e+09 _runtime‚3970.959518072 _step‚193§¸FË” Å & lr-AdamW/pg2‚0.0006742430680600047 trainer/global_step‚2399 $ _timestamp‚1.784293786742402e+09 _runtime‚3970.959518072 _step‚193 & lr-AdamW/pg1‚0.0006742430680600047Á{æ12• - _wandbruntime‚3970  _runtime‚3970÷PÜÓï'– éO TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2251672856064 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 % _timestamp‚1.7842937867434404e+09 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚688682179930624 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚686429135511552 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚20737274 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2253018245632 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚20737274 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 trainer/global_step‚2399 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6791634 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2907028595712 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2249166466198 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚189307122249728 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚189306719622144 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚186399691026432 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6151989 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2907063132160 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚20358801 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6151994 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6791888 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚20358795 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2247821865876 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2253018245632 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚686132300689408 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚688680808367616 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097t ýô'— îO WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2249166466198 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚189307122249728 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚189306719622144 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚186399691026432 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6151989 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2907063132160 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚20358801 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6151994 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6791888 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚20358795 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2247821865876 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2253018245632 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚686132300689408 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚688680808367616 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2251672856064 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 % _timestamp‚1.7842937867434404e+09 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚688682179930624 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚686429135511552 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚20737274 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2253018245632 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚20737274 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 trainer/global_step‚2399 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6791634 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2907028595712ÂY½A2˜ - _wandbruntime‚3972  _runtime‚3972‘Zf`×™ óM NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚20745766 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚686707162415104 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12948907 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚20367001 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚686707162415104 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚20735779 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚12949145 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚688660297828108 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚41112773 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚20735779 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚41112773 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚688959734133248 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2252597891584 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚20367007 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6154532 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2252597891584 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚686410182467584 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 % _timestamp‚1.7842937881302664e+09 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚688961105703424 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚686707136241664 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_p4wÃ" ool.current‚ 1345396736 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚41102780 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2908252572672 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6794608 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚688658927799706 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2250089932556 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2908252572672 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6794608 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚688658927799706 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2250089932556 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12948907 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚20367001 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚686707162415104 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚20735779 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚12949145 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚688660297828108 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚41112773 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚20735779 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚41112773 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚688959734133248 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2252597891584 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚20367007 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6154532 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2252597891584 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚686410182467584 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 % _timestamp‚1.7842937881302664e+09 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚688961105703424 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚686707136241664 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736Ã-r$ 2› - _wandbruntime‚3972  _runtime‚3972¹ñ¯>œ ý trainer/global_step‚2399 % _timestamp‚1.7842937882172153e+09 /loss/FRt2/assignment_loss‚0.868110037679009 /loss/FBt1/detection_loss‚0.9810890986965123 _runtime‚3972.434143148 0loss/FRt1/assignment_loss‚1.9196054769703916 1loss/FBt2/assignment_loss‚0.06599677099613557 1loss/SRqqt2/detection_loss‚0.6748754126861929 .loss/FBt2/detection_loss‚0.551879682096859 2loss/SRqqt1/assignment_loss‚0.6768318218670359 1loss/FBt1/assignment_loss‚0.15367539338032915 %loss/total_loss‚9.411897360323483 epoch‚5 _step‚196 1loss/SRqqt2/assignment_loss‚0.562992622780715 /loss/FRt1/detection_loss‚0.5606615978758314 .loss/FRt2/detection_loss‚1.523778974357769 1loss/SRqqt1/detection_loss‚0.8724004709367025vþÏ ‚ .loss/FRt2/detection_loss‚1.523778974357769 1loss/SRqqt1/detection_loss‚0.8724004709367025 trainer/global_step‚2399 % _timestamp‚1.7842937882172153e+09 /loss/FRt2/assignment_loss‚0.868110037679009 /loss/FBt1/detection_loss‚0.9810890986965123 _runtime‚3972.434143148 0loss/FRt1/assignment_loss‚1.9196054769703916 1loss/FBt2/assignment_loss‚0.06599677099613557 1loss/SRqqt2/detection_loss‚0.6748754126861929 .loss/FBt2/detection_loss‚0.551879682096859 2loss/SRqqt1/assignment_loss‚0.6768318218670359 1loss/FBt1/assignment_loss‚0.15367539338032915 %loss/total_loss‚9.411897360323483 epoch‚5 _step‚196 1loss/SRqqt2/assignment_loss‚0.562992622780715 /loss/FRt1/detection_loss‚0.5606615978758314ÄÌà‹;Ÿž ‚(:” žÓèÒ Áäö gpu.0.gpu‚90.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚39.0 gpu.0.powerWatts‚110.286' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.057199999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0I$uçŸ ‚(:Ü žÓèÒܧù) disk./.usagePercent‚37.40240608783459 disk.rbd3.out‚0 disk.rbd1.in‚0 network.recv‚ 1.037135e+06 disk.drbd2165.in‚0) proc.memory.availableMB‚ 1804.91796875 proc.memory.rssMB‚ 2860.46875 disk.sda3.out‚ 156.93359375 network.sent‚ 4.469988e+06 disk.dm-0.out‚ 53171.3359375 disk.sda1.in‚0 disk.sda.out‚ 156.93359375$ memory_percent‚88.98365497589111 disk.sda1.out‚0! disk.nvme0c0n1.in‚ 7852.765625 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda3.in‚0 disk.rbd3.in‚0 cpu‚25.56792974270976 disk.sda.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 156.93359375 proc.cpu.threads‚17 disk.drbd2165.out‚0$ disk./.usageGB‚557.3228797912598$ disk.nvme0c0n1.out‚ 53171.3359375 disk.nvme0n1.in‚ 7852.765625 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd1.out‚ 9.6015625) proc.memory.percent‚17.45891571044922 disk.dm-0.in‚ 7852.765625" disk.nvme0n1.out‚ 53171.3359375ŽˆGóŸ  ‚(:” ­ÓèÒÛŠ÷ gpu.0.gpu‚100.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚329.321' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚65.86420000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 w%‘ò¡ ‚(:ç ­ÓèÒ–ìÞø disk.dm-1.out‚ 157.234375 disk.drbd2165.in‚0 disk.rbd3.out‚0) disk./.usagePercent‚37.40277473868389 disk.drbd2165.out‚0# disk.nvme0c0n1.in‚ 7965.44140625$ memory_percent‚88.64307403564453 disk.sda2.in‚0 disk.rbd1.in‚0 disk.sda.in‚0 disk.sda.out‚ 157.234375 disk.rbd0.out‚0 disk.rbd3.in‚0& proc.memory.availableMB‚ 1860.71875# proc.memory.rssMB‚ 2804.80859375% disk.nvme0n1.out‚53182.0908203125 network.recv‚ 1.042232e+06 disk.dm-0.in‚ 7965.44140625" disk.dm-0.out‚53182.0908203125 disk.sda1.in‚0 disk.sda3.in‚0 disk.rbd1.out‚ 9.6015625 disk.rbd0.in‚0 disk.sda2.out‚0! disk.nvme0n1.in‚ 7965.44140625$ disk./.usageGB‚557.3283729553223 disk.sda1.out‚0' disk.nvme0c0n1.out‚53182.0908203125* proc.memory.percent‚17.119193077087402 cpu‚25.576404562476053 network.sent‚ 4.533014e+06 disk.dm-1.in‚0 disk.sda3.out‚ 157.234375 proc.cpu.threads‚17ø:¦­2¢ - _wandbruntime‚3998  _runtime‚3998Tøq¯ ;£ ìz 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6467307209968567 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.5483965873718262 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.19805635511875153 *jet/accuracy_1_of_1‚0.3037162721157074 particle/accuracy_2_of_2‚1 & Purity/*FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6391205787658691 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8153817057609558 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9976646900177002 ("Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 ,jet/accuracy_3_of_4‚0.059448353946208954 +particle/sensitivity‚0.5928080677986145 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.6180079579353333 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.08891116082668304 'particle/f_score‚0.6162589192390442 6 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.198543518781662 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5804041028022766 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.13139194250106812 7 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.0886625349521637 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8169377446174622 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.7894404530525208 6 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.809278130531311 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.2130935937166214 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8009259104728699 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 jet/accuracy_3_of_5‚1 jet/accuracy_5_of_5‚0 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.1582644134759903 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.21211616694927216 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8144935369491577 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.7772601246833801 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5796316862106323 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.5739283561706543 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.5624106526374817 +jet/accuracy_3_of_3‚0.04813210293650627 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.11538461595773697 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3420366644859314 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7461261749267578 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.22109083831310272 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.017084073275327682 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 particle/accuracy_2_of_4‚1 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9722222089767456 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9356862902641296 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 jet/accuracy_1_of_5‚1 particle/accuracy_1_of_5‚1 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.0555555559694767 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.08941248804330826 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7127313017845154 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 particle/accuracy_1_of_2‚1 epoch‚5 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.26580798625946045 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6990941762924194 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.11538461595773697 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.625 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6544374227523804 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 particle/accuracy_5_of_5‚0 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2855181097984314 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9954468011856079 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9974743723869324 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6288418173789978 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6883915662765503 & Purity/1FRt1SRqqt2FBt/FBt_purity‚1 particle/accuracy_2_of_3‚1 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 /particle/accuracy_3_of_4‚0.9977324604988098 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9885455369949341 /particle/accuracy_4_of_4‚0.9302822351455688 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5687147974967957 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.16215123236179352 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.20078164339065552 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8298941850662231 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9952632188796997 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9976948499679565 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9824140667915344 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.18374758958816528 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6633416414260864 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7011898756027222 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.43518519401550293 & Purity/1FRt1SRqqt*FBt/FBt_purity‚1 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 trainer/global_step‚2399 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2202146202325821 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.21283231675624847 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4609375 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9912070035934448 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5732225179672241 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 8"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.651309609413147 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 *jet/accuracy_1_of_4‚0.9600944519042969 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6285129189491272 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.1447288691997528 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.12128058075904846 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9245488047599792 particle/accuracy_3_of_5‚1 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6887072324752808 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5508333444595337 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.047678131610155106 particle/accuracy_1_of_1‚1 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.1851593255996704 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.2083091288805008 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.14683181047439575 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.08839398622512817 (particle/accuracy‚0.8414597511291504 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5796316862106323 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6994287371635437 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.08891116082668304 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.8571428656578064 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.5959269404411316 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5479825735092163 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 /particle/accuracy_3_of_3‚0.9994933605194092 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 jet/accuracy_2_of_5‚1 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6309079527854919 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7325411438941956 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.2760416865348816 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.18528330326080322 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 particle/accuracy_1_of_4‚1 6 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.996219277381897 6 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.998227596282959 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.24419648945331573 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.48090630769729614 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2080150544643402 *jet/accuracy_2_of_4‚0.7112976908683777 jet/accuracy_4_of_5‚0 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9918577075004578 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.08947145938873291 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8144935369491577 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6172100901603699 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.2700808048248291 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.12495362013578415 ,jet/accuracy_4_of_4‚0.013899499550461769 _runtime‚3998.976243807 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.2083091288805008 *jet/accuracy_1_of_2‚0.7620998620986938 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.2807594835758209 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7382891774177551 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 particle/accuracy_1_of_3‚1 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7142857313156128 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 7validation_average_jet_accuracy‚0.34213995933532715 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5740896463394165 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9944615960121155 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.036264341324567795 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6949048042297363 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.6181246638298035 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.5456812977790833 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.025140589103102684 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9900867938995361 6 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.942307710647583 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6314674019813538 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8747033476829529 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9974743723869324 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9814814329147339 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9814814329147339 ("Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0 6 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.997619092464447 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 % _timestamp‚1.7842938147580342e+09 (loss/val_total_loss‚8.93193004592259 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.6218135356903076 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 ( Purity/2FRt1SRqqt1FBt/FRt_purity‚0.5 , Purity/0FRt2SRqqt*FBt/FBt_purity‚0.96875 , Purity/0FRt2SRqqt1FBt/FBt_purity‚0.96875 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9992441534996033 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.2795983850955963 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.1608899086713791 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03249777853488922 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.6550002694129944 *jet/accuracy_2_of_2‚0.1039314717054367 +particle/specificity‚0.9094588756561279 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6145833134651184 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 _step‚197 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.20671360194683075 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.0555555559694767 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.03215876966714859 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.8125 particle/accuracy_4_of_5‚1 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9963642358779907 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.25479742884635925 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9984177350997925 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9960318207740784 & Purity/0FRt1SRqqt1FBt/FBt_purity‚1 particle/accuracy_2_of_5‚1 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.26289302110671997 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.11913314461708069 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7343257665634155 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.33500149846076965 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2019629031419754 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.7772601246833801 8 Purity/1FRt2SRqqt1FBt/FRt_puriy¹Ïety‚0.10000000149011612 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8009259104728699 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.46710172295570374 )jet/accuracy_1_of_3‚0.896331250667572 +jet/accuracy_2_of_3‚0.44467979669570923 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.18638375401496887 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5957235097885132 9"Purity/2FRt*SRqqt1FBt/event_purity‚0.1316666603088379 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.22511716187000275gM›mw=¤ ñz ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 *jet/accuracy_1_of_4‚0.9600944519042969 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6285129189491272 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.1447288691997528 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.12128058075904846 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9245488047599792 particle/accuracy_3_of_5‚1 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6887072324752808 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5508333444595337 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.047678131610155106 particle/accuracy_1_of_1‚1 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.1851593255996704 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.2083091288805008 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.14683181047439575 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.08839398622512817 (particle/accuracy‚0.8414597511291504 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5796316862106323 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6994287371635437 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.08891116082668304 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.8571428656578064 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.5959269404411316 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5479825735092163 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 /particle/accuracy_3_of_3‚0.9994933605194092 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 jet/accuracy_2_of_5‚1 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6309079527854919 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7325411438941956 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.2760416865348816 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.18528330326080322 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 particle/accuracy_1_of_4‚1 6 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.996219277381897 6 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.998227596282959 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.24419648945331573 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.48090630769729614 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2080150544643402 *jet/accuracy_2_of_4‚0.7112976908683777 jet/accuracy_4_of_5‚0 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9918577075004578 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.08947145938873291 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8144935369491577 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6172100901603699 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.2700808048248291 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.12495362013578415 ,jet/accuracy_4_of_4‚0.013899499550461769 _runtime‚3998.976243807 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.2083091288805008 *jet/accuracy_1_of_2‚0.7620998620986938 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.2807594835758209 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7382891774177551 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 particle/accuracy_1_of_3‚1 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7142857313156128 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 7validation_average_jet_accuracy‚0.34213995933532715 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5740896463394165 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9944615960121155 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.036264341324567795 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6949048042297363 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.6181246638298035 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.5456812977790833 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.025140589103102684 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9900867938995361 6 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.942307710647583 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6314674019813538 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8747033476829529 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9974743723869324 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9814814329147339 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9814814329147339 ("Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0 6 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.997619092464447 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 % _timestamp‚1.7842938147580342e+09 (loss/val_total_loss‚8.93193004592259 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.6218135356903076 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 ( Purity/2FRt1SRqqt1FBt/FRt_purity‚0.5 , Purity/0FRt2SRqqt*FBt/FBt_purity‚0.96875 , Purity/0FRt2SRqqt1FBt/FBt_purity‚0.96875 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9992441534996033 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.2795983850955963 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.1608899086713791 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03249777853488922 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.6550002694129944 *jet/accuracy_2_of_2‚0.1039314717054367 +particle/specificity‚0.9094588756561279 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6145833134651184 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 _step‚197 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.20671360194683075 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.0555555559694767 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.03215876966714859 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.8125 particle/accuracy_4_of_5‚1 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9963642358779907 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.25479742884635925 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9984177350997925 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9960318207740784 & Purity/0FRt1SRqqt1FBt/FBt_purity‚1 particle/accuracy_2_of_5‚1 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.26289302110671997 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.11913314461708069 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7343257665634155 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.33500149846076965 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2019629031419754 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.7772601246833801 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8009259104728699 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.46710172295570374 )jet/accuracy_1_of_3‚0.896331250667572 +jet/accuracy_2_of_3‚0.44467979669570923 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.18638375401496887 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5957235097885132 9"Purity/2FRt*SRqqt1FBt/event_purity‚0.1316666603088379 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.22511716187000275 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6467307209968567 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.5483965873718262 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.19805635511875153 *jet/accuracy_1_of_1‚0.3037162721157074 particle/accuracy_2_of_2‚1 & Purity/*FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6391205787658691 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8153817057609558 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9976646900177002 ("Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 ,jet/accuracy_3_of_4‚0.059448353946208954 +particle/sensitivity‚0.5928080677986145 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.6180079579353333 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.08891116082668304 'particle/f_score‚0.6162589192390442 6 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.198543518781662 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5804041028022766 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.13139194250106812 7 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.0886625349521637 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8169377446174622 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.7894404530525208 6 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.809278130531311 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.2130935937166214 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8009259104728699 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 jet/accuracy_3_of_5‚1 jet/accuracy_5_of_5‚0 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.1582644134759903 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.21211616694927216 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8144935369491577 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.7772601246833801 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5796316862106323 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.5739283561706543 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.5624106526374817 +jet/accuracy_3_of_3‚0.04813210293650627 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.11538461595773697 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3420366644859314 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7461261749267578 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.22109083831310272 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.017084073275327682 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 particle/accuracy_2_of_4‚1 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9722222089767456 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9356862902641296 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 jet/accuracy_1_of_5‚1 particle/accuracy_1_of_5‚1 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.0555555559694767 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.08941248804330826 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7127313017845154 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 particle/accuracy_1_of_2‚1 epoch‚5 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.26580798625946045 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6990941762924194 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.11538461595773697 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.625 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6544374227523804 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 particle/accuracy_5_of_5‚0 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2855181097984314 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9954468011856079 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9974743723869324 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6288418173789978 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6883915662765503 & Purity/1FRt1SRqqt2FBt/FBt_purity‚1 particle/accuracy_2_of_3‚1 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 /particle/accuracy_3_of_4‚0.9977324604988098 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9885455369949341 /particle/accuracy_4_of_4‚0.9302822351455688 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5687147974967957 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.16215123236179352 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.20078164339065552 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8298941850662231 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9952632188796997 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9976948499679565 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9824140667915344 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.18374758958816528 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6633416414260864 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7011898756027222 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.43518519401550293 & Purity/1FRt1SRqqt*FBt/FBt_purity‚1 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 trainer/global_step‚2399 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2202146202325821 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.21283231675624847 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4609375 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9912070035934448 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5732225179672241 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 8"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.651309609413147Å÷z+Ú•¥ ‚(:Š ¼ÓèÒú¶óö gpu.0.gpu‚100.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚35.0 gpu.0.powerWatts‚257.233' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚51.4466 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0®[èù¦ ‚(:î ¼ÓèÒæ‹ù disk.drbd2165.in‚0 disk.dm-1.out‚ 157.3359375# proc.memory.rssMB‚ 2824.19921875 disk.rbd3.in‚0 disk.sda2.out‚0 disk.rbd0.out‚0 disk.sda3.in‚0 disk.sda3.out‚ 157.3359375! disk.nvme0n1.in‚ 7971.25390625) proc.memory.availableMB‚ 2473.41015625 proc.cpu.threads‚17$ disk./.usageGB‚557.3342590332031 disk.drbd2165.out‚0# disk.nvme0c0n1.in‚ 7971.25390625 disk.sda.out‚ 157.3359375 disk.sda1.in‚0$ memory_percent‚84.90350246429443* proc.memory.percent‚17.237544059753418* disk./.usagePercent‚37.403169758309225 disk.rbd3.out‚0' disk.nvme0c0n1.out‚53918.0751953125 disk.rbd1.out‚ 9.6640625 disk.sda.in‚0 network.sent‚ 4.538146e+06 network.recv‚ 1.045735e+06 disk.rbd1.in‚0 disk.dm-0.in‚ 7971.25390625" disk.dm-0.out‚53918.0751953125 disk.dm-1.in‚0 disk.rbd0.in‚0 cpu‚25.582011272089513% disk.nvme0n1.out‚53918.0751953125 disk.sda2.in‚0 disk.sda1.out‚0}}Q ”§ ‚(:‰ ËÓèÒ¶ó÷ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚268.262' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.6524 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0&\ïõ¨ ‚(:ê ËÓèÒË“«ù disk.sda1.out‚0 disk.sda3.out‚ 157.84375* proc.memory.percent‚17.237544059753418 network.recv‚ 1.050539e+06 disk.rbd3.out‚0 disk.sda2.out‚0 cpu‚25.581270889463706 disk.dm-1.in‚0 disk.sda1.in‚0 disk.sda3.in‚0 disk.drbd2165.in‚0! disk.nvme0n1.in‚ 7971.25390625 disk.rbd1.in‚0 network.sent‚ 4.595573e+06* disk./.usagePercent‚37.403495399892776 disk.dm-0.in‚ 7971.25390625 disk.sda.out‚ 157.84375) proc.memory.availableMB‚ 2473.44140625 disk.rbd0.in‚0 disk.rbd3.in‚0# disk.nvme0c0n1.in‚ 7971.25390625 disk.sda2.in‚0 disk.rbd1.out‚ 9.6640625 proc.cpu.threads‚17 disk.dm-1.out‚ 157.84375 disk.rbd0.out‚0( disk.nvme0c0n1.out‚55286.84228515625 disk.sda.in‚0# disk./.usageGB‚557.339111328125$ memory_percent‚84.90331172943115# disk.dm-0.out‚55286.84228515625 disk.drbd2165.out‚0& disk.nvme0n1.out‚55286.84228515625# proc.memory.rssMB‚ 2824.19921875²/ ÑŸ© ‚(:” ÚÓèÒª‡Üö gpu.0.gpu‚100.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚340.626' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚68.12519999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÇtX'ÿª ‚(:ô ÚÓèÒƨíø* disk./.usagePercent‚37.403803120948936 disk.sda.out‚ 158.26953125 disk.rbd0.in‚0* proc.memory.percent‚17.237544059753418 proc.cpu.threads‚17( disk.nvme0c0n1.out‚55297.31005859375 disk.sda2.in‚0 disk.sda2.out‚0 disk.drbd2165.in‚0$ disk./.usageGB‚557.3436965942383# disk.nvme0c0n1.in‚ 7971.25390625 disk.rbd0.out‚0 disk.dm-0.in‚ 7971.25390625 disk.sda1.in‚0 disk.rbd3.out‚0 disk.sda.in‚0 disk.sda1.out‚0! disk.nvme0n1.in‚ 7971.25390625& disk.nvme0n1.out‚55297.31005859375 disk.dm-1.out‚ 158.26953125 disk.drbd2165.out‚0 network.sent‚ 4.600612e+06 disk.rbd1.in‚0) proc.memory.availableMB‚ 1501.50390625# proc.memory.rssMB‚ 2824.19921875 cpu‚25.580607280320965 network.recv‚ 1.054158e+06 disk.rbd3.in‚0# disk.dm-0.out‚55297.31005859375 disk.sda3.out‚ 158.26953125$ memory_percent‚90.83554744720459 disk.rbd1.out‚ 9.7265625 disk.dm-1.in‚0 disk.sda3.in‚0×}‘ž« ‚(:“ éÓèÒ•Ýóö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚292.548' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚58.50959999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ªäY7ô¬ ‚(:é éÓèÒÍÿ‚ù disk.drbd2165.in‚0& disk.nvme0c0n1.out‚55310.470703125 disk.sda.in‚0( proc.memory.availableMB‚ 1501.5078125 disk.sda1.in‚0 disk.sda2.in‚0 disk.rbd1.in‚0$ memory_percent‚90.83552360534668 disk.sda3.out‚ 158.7265625 disk.rbd3.in‚0! disk.nvme0n1.in‚ 7971.25390625 network.sent‚ 4.605616e+06* disk./.usagePercent‚37.404198140574266 disk.sda3.in‚0 disk.rbd0.in‚0 proc.cpu.threads‚17 network.recv‚ 1.05741e+06 disk.dm-0.in‚ 7971.25390625 disk.rbd3.out‚0# disk.nvme0c0n1.in‚ 7971.25390625$ disk.nvme0n1.out‚55310.470703125 disk.sda2.out‚0 disk.rbd1.out‚ 9.7265625 disk.drbd2165.out‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 158.7265625* proc.memory.percent‚17.237544059753418! disk.dm-0.out‚55310.470703125 disk.rbd0.out‚0$ disk./.usageGB‚557.3495826721191 disk.sda1.out‚0 disk.sda.out‚ 158.7265625# proc.memory.rssMB‚ 2824.19921875 cpu‚25.579817161890986³ð¸FŸ­ ‚(:” øÓèÒ÷²‚÷ gpu.0.gpu‚99.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚243.496' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚48.699200000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ä»÷® ‚(:ì øÓèÒ ­Ãù disk.sda.in‚0# disk.nvme0c0n1.in‚ 8254.22265625 disk.rbd1.in‚0 network.sent‚ 4.610674e+06) disk./.usagePercent‚37.40457114355166# disk.dm-0.out‚55318.92431640625! disk.nvme0n1.in‚ 8254.22265625 disk.sda2.in‚0# proc.memory.rssMB‚ 2824.19921875$ disk./.usageGB‚557.3551406860352 disk.rbd0.in‚0 disk.dm-0.in‚ 8254.22265625 disk.drbd2165.out‚0 disk.rbd3.in‚0& proc.memory.availableMB‚ 1501.59375 cpu‚25.579150800240782 disk.sda3.in‚0( disk.nvme0c0n1.out‚55318.92431640625 disk.dm-1.out‚ 158.9453125 disk.sda1.in‚0 disk.sda3.out‚ 158.9453125& disk.nvme0n1.out‚55318.92431640625$ memory_percent‚90.83499908447266 network.recv‚ 1.06069e+06 disk.sda.out‚ 158.9453125 disk.dm-1.in‚0 disk.rbd3.out‚0 disk.sda2.out‚0 proc.cpu.threads‚17 disk.rbd0.out‚0 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.rbd1.out‚ 9.7890625* proc.memory.percent‚17.237544059753418÷L-”¯ ‚(:‰ ‡ÔèÒò¤ä÷ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚295.976' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚59.1952 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ýŒÈû° ‚(:ð ‡ÔèÒÒúßø disk.drbd2165.in‚0 disk.sda2.in‚0 disk.sda.in‚0$ disk./.usageGB‚557.3614387512207( disk.nvme0c0n1.out‚55329.20166015625& disk.nvme0n1.out‚55329.20166015625 disk.rbd1.out‚ 9.7890625 disk.dm-1.in‚0 disk.rbd0.out‚0 disk.dm-0.in‚ 8254.22265625$ memory_percent‚90.83437919616699 disk.sda.out‚ 159.2890625 disk.drbd2165.out‚0( proc.memory.availableMB‚ 1501.6953125# proc.memory.rssMB‚ 2824.19921875* proc.memory.percent‚17.237544059753418* disk./.usagePercent‚37.404993811990686 disk.rbd3.out‚0 disk.sda2.out‚0 disk.sda3.in‚0 proc.cpu.threads‚17 network.sent‚ 4.615677e+06# disk.nvme0c0n1.in‚ 8254.22265625 disk.rbd1.in‚0 disk.sda1.in‚0 network.recv‚ 1.064191e+06! disk.nvme0n1.in‚ 8254.22265625 disk.dm-1.out‚ 159.2890625# disk.dm-0.out‚55329.20166015625 disk.sda3.out‚ 159.2890625 disk.sda1.out‚0 cpu‚25.578506077105246 disk.rbd3.in‚0 disk.rbd0.in‚0&¯õ22± - _wandbruntime‚4081  _runtime‚4081.VhfƲ À trainer/global_step‚2449 $ _timestamp‚1.784293897441807e+09 _runtime‚4081.659926865 _step‚198 & lr-AdamW/pg1‚0.0006733593718089962 & lr-AdamW/pg2‚0.0006733593718089962>- Ë³ Å trainer/global_step‚2449 $ _timestamp‚1.784293897441807e+09 _runtime‚4081.659926865 _step‚198 & lr-AdamW/pg1‚0.0006733593718089962 & lr-AdamW/pg2‚0.0006733593718089962ƺ6^í2´ - _wandbruntime‚4081  _runtime‚4081,Õ¬ð>µ èO F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚703852298530816 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚42113141 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚193296681949696 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚190322123657216 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚42123120 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚20901101 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2974592828928 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚21222013 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚706462356416000 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚13252002 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2301232710748 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚706462356416000 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚20901107 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6963940 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚704157153452032 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2303831400960 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚706152186641242 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚704157179625472 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2305176790528 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚13251747 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6288062 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6288057 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2974558292480 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚21222013 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚250 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚190322491748352 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2299888110426 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2305176790528 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 _step‚199 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚ 4081.66002505 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚20901101 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚704157179625472 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 trainer/global_step‚2449 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚706460984852992 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚706460984852992 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0pاñó'¶ íO G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2974558292480 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚21222013 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚250 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚190322491748352 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2299888110426 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2305176790528 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 _step‚199 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚ 4081.66002505 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚20901101 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚704157179625472 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 trainer/global_step‚2449 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚706460984852992 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚706460984852992 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚703852298530816 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚42113141 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚193296681949696 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚190322123657216 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚42123120 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚20901101 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2974592828928 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚21222013 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚706462356416000 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚13252002 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2301232710748 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚706462356416000 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚20901107 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6963940 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚704157153452032 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2303831400960 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚706152186641242 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚704157179625472 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2305176790528 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚13251747 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6288062 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6288057 IDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚20909307 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6290600 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚190399610064896 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚2449 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚704435180355584 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 _runtime‚4083.046420167 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚20909307 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6290605 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚42139818 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚21220518 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚704435180355584 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6966709 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚13257065 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚244 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2302156177106 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175p9_½%¹ øM UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2304756436480 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚21230505 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2975793622528 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚704435154182144 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚13257314 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚706430991885664 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2300811576672 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 _step‚200 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚706739910618624 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚42129825 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚704130205736960 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚20909313 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚13257065 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚244 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2302156177106 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚706741282188800 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚42129825 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚193375771778560 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚704130180308992 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚20909307 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6290600 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚190399610064896 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚2449 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚704435180355584 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 _runtime‚4083.046420167 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚20909307 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6290605 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚42139818 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚21220518 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚704435180355584 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6966709 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚720407337289216 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 trainer/global_step‚2499 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚42959172 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36633600 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚718057216131072 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚21647765 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚13518626 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚194196812132352 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚720092512663150 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚720407337289216 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3034413103616 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404724736 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚197231225235968 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2351492721152 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 _step‚203 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6415207 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 # _timestamp‚1.78429397627624e+09 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚21647765 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2346125223534 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚718057189957632 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚720093882691440 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚21311407 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚21637792 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚717746412867584 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚21311407 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2347469823856 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚720408708852224 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚288 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6415212 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚42949193 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2351492721152 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2350147331584 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚21311401 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚21637792 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚718057189957632 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7103131 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚717746387439616 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚197230820511232 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚718057216131072 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚13518338 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚194196444041216 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3034376470016 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚720093882691440 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚21311407 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚21637792 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚717746412867584 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚21311407 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2347469823856 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚720408708852224 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚288 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6415212 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚42949193 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2351492721152 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2350147331584 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚21311401 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚21637792 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚718057189957632 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7103131 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚717746387439616 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚197230820511232 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚718057216131072 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚13518338 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚194196444041216 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3034376470016 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚720407337289216 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 trainer/global_step‚2499 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚42959172 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36633600 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚718057216131072 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚21647765 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚13518626 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚194196812132352 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚720092512663150 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚720407337289216 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3034413103616 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404724736 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚197231225235968 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2351492721152 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 _step‚203 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6415207 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0ËBšå2Í - _wandbruntime‚4161  _runtime‚4161¿%MI Î óM WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2351072367104 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 % _timestamp‚1.7842939776615155e+09 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚21319607 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3035564539392 _step‚204 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚720687634625024 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚720371317907572 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6417750 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚268 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚718024269217792 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2352417763840 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚718335216861184 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚273 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚42965877 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚21319613 REDeviceStatsMonitor.on_train_batch_endI‘Y°/allocation.large_pool.allocated‚21319613 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚197309899705856 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7105888 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404717568 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚720372687935974 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚21319607 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚42975870 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚720687634625024 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚718335216861184 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚720686263054848 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 _runtime‚4161.878987534 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2351072367104 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚42965877 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚194274298540032 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚194273930448896 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3035601165824 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2348393290214 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚42975870 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚13523638 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36626432 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402ÁgÑZþ&Ï øM M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7105888 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404717568 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚720372687935974 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚21319607 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚42975870 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚720687634625024 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚718335216861184 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚720686263054848 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 _runtime‚4161.878987534 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2351072367104 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚42965877 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚194274298540032 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚194273930448896 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3035601165824 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2348393290214 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚42975870 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚13523638 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36626432 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2351072367104 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 % _timestamp‚1.7842939776615155e+09 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚21319607 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3035564539392 _step‚204 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚720687634625024 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚720371317907572 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6417750 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚268 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚718024269217792 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2352417763840 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚718335216861184 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚273 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚42965877 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚21319613 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚21319613 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚197309899705856 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008Ìpji2Ð - _wandbruntime‚4161  _runtime‚41612/µÑ ‚ 1loss/SRqqt1/detection_loss‚1.1131518153580937 /loss/FBt2/detection_loss‚0.4536631022670513 trainer/global_step‚2499 _runtime‚4161.965245341 2loss/FBt2/assignment_loss‚0.021580680615748853 /loss/FBt1/detection_loss‚1.3455774482605714 % _timestamp‚1.7842939777482908e+09 _step‚205 %loss/total_loss‚9.698745205869912 epoch‚6 0loss/FRt2/assignment_loss‚0.8115463876612572 2loss/SRqqt2/assignment_loss‚0.5384458983474004 2loss/FBt1/assignment_loss‚0.014706770261597463 /loss/FRt1/detection_loss‚0.5818461742319396 .loss/FRt2/detection_loss‚1.564935696077919 1loss/SRqqt2/detection_loss‚0.6148690438852139 0loss/FRt1/assignment_loss‚2.0017414326887075 2loss/SRqqt1/assignment_loss‚0.6366807562144131¸¼76 Ò ‡ 1loss/SRqqt2/detection_loss‚0.6148690438852139 0loss/FRt1/assignment_loss‚2.0017414326887075 2loss/SRqqt1/assignment_loss‚0.6366807562144131 1loss/SRqqt1/detection_loss‚1.1131518153580937 /loss/FBt2/detection_loss‚0.4536631022670513 trainer/global_step‚2499 _runtime‚4161.965245341 2loss/FBt2/assignment_loss‚0.021580680615748853 /loss/FBt1/detection_loss‚1.3455774482605714 % _timestamp‚1.7842939777482908e+09 _step‚205 %loss/total_loss‚9.698745205869912 epoch‚6 0loss/FRt2/assignment_loss‚0.8115463876612572 2loss/SRqqt2/assignment_loss‚0.5384458983474004 2loss/FBt1/assignment_loss‚0.014706770261597463 /loss/FRt1/detection_loss‚0.5818461742319396 .loss/FRt2/detection_loss‚1.564935696077919ÍÖ§Å”Ó ‚(:‰ áÔèÒ¬…ãö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚302.696' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚60.5392 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ê¡ÁýÔ ‚(:ò áÔèÒ˜Ûù disk.rbd3.in‚0 disk.sda2.in‚0) disk./.usagePercent‚37.40662073987078# disk.dm-0.out‚55390.43798828125# disk.nvme0c0n1.in‚ 8458.62890625( disk.nvme0c0n1.out‚55390.43798828125 disk.sda.in‚0 disk.dm-0.in‚ 8458.62890625 disk.sda3.in‚0 disk.sda3.out‚ 160.73828125 disk.rbd3.out‚0 disk.sda2.out‚0) proc.memory.percent‚17.23759174346924 disk.rbd0.in‚0& disk.nvme0n1.out‚55390.43798828125$ memory_percent‚90.82996845245361 disk.dm-1.in‚0 disk.rbd0.out‚0 disk.sda1.out‚0) proc.memory.availableMB‚ 1502.41796875 proc.cpu.threads‚17$ disk./.usageGB‚557.3856811523438 disk.dm-1.out‚ 160.73828125 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 9.9765625# proc.memory.rssMB‚ 2824.20703125 cpu‚25.574371941279445! disk.nvme0n1.in‚ 8458.62890625 network.sent‚ 4.703989e+06 network.recv‚ 1.085729e+06 disk.sda.out‚ 160.73828125 disk.sda1.in‚0æÑÌU”Õ ‚(:‰ ðÔèÒ¶—Üö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚268.022' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.6044 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0iŸø!úÖ ‚(:ï ðÔèÒª…Îø disk.drbd2165.in‚0 disk.rbd3.out‚0! disk.nvme0n1.in‚ 8809.26171875 disk.sda.out‚ 160.8671875) disk./.usagePercent‚37.40702215968447& disk.nvme0n1.out‚55397.02001953125 disk.dm-1.in‚0( proc.memory.availableMB‚ 1502.4140625$ disk./.usageGB‚557.3916625976562 disk.sda1.out‚0( disk.nvme0c0n1.out‚55397.02001953125 disk.sda2.in‚0 network.recv‚ 1.090269e+06 disk.sda1.in‚0# disk.nvme0c0n1.in‚ 8809.26171875 disk.sda.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.dm-0.in‚ 8809.26171875$ memory_percent‚90.82999229431152 disk.sda3.in‚0 disk.sda3.out‚ 160.8671875 disk.sda2.out‚0# proc.memory.rssMB‚ 2824.20703125 cpu‚25.573578774113017 disk.rbd3.in‚0 proc.cpu.threads‚17 network.sent‚ 4.766873e+06 disk.rbd1.in‚0 disk.dm-1.out‚ 160.8671875) proc.memory.percent‚17.23759174346924# disk.dm-0.out‚55397.02001953125 disk.drbd2165.out‚0 disk.rbd1.out‚ 10.0390625ÂO›¬ž× ‚(:“ ÿÔèÒÀ³ø gpu.0.gpu‚93.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚47.0 gpu.0.powerWatts‚339.413' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚67.88260000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‰sJ ÷Ø ‚(:ì ÿÔèÒΦóù cpu‚25.573005061252292& disk.nvme0n1.out‚55407.01708984375 disk.sda.in‚0 proc.cpu.threads‚17 disk.rbd3.out‚0 disk.sda2.out‚0# disk.nvme0c0n1.in‚ 8809.26171875! disk.nvme0n1.in‚ 8809.26171875 disk.rbd0.out‚0 disk.sda3.in‚0 disk.sda.out‚ 161.3203125# disk.dm-0.out‚55407.01708984375 disk.sda3.out‚ 161.3203125 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.rbd1.in‚0 network.recv‚ 1.094005e+06) disk./.usagePercent‚37.40790973780568$ disk./.usageGB‚557.4048881530762( disk.nvme0c0n1.out‚55407.01708984375 disk.dm-1.out‚ 161.3203125 disk.rbd0.in‚0) proc.memory.percent‚17.23759174346924 disk.rbd3.in‚0 network.sent‚ 4.771919e+06 disk.sda2.in‚0 disk.dm-0.in‚ 8809.26171875 disk.dm-1.in‚0# memory_percent‚90.8315658569336 disk.rbd1.out‚ 10.0390625 disk.sda1.in‚0 disk.sda1.out‚0& proc.memory.availableMB‚ 1502.15625# proc.memory.rssMB‚ 2824.20703125^™Ò ”Ù ‚(:‰ ŽÕèÒ·¦Óö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚266.062' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.2124 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0­Ì_ûÚ ‚(:ð ŽÕèÒ†¤¹ø disk.rbd0.in‚0 disk.sda1.out‚0 disk.sda3.out‚ 161.7734375# disk.nvme0c0n1.in‚ 8809.26171875 network.recv‚ 1.097453e+06 disk.drbd2165.in‚0 disk.rbd3.in‚0 disk.dm-1.out‚ 161.7734375) proc.memory.availableMB‚ 1502.15234375 proc.cpu.threads‚17 disk.sda.out‚ 161.7734375 disk.dm-1.in‚0 disk.sda3.in‚0 disk.drbd2165.out‚0) proc.memory.percent‚17.23759174346924& disk.nvme0n1.out‚55418.20654296875$ disk./.usageGB‚557.4101982116699 disk.rbd1.out‚ 10.1015625 disk.dm-0.in‚ 8809.26171875 cpu‚25.572234058370558 network.sent‚ 4.777032e+06 disk.sda2.out‚0 disk.sda1.in‚0 disk.rbd3.out‚0! disk.nvme0n1.in‚ 8809.26171875 disk.sda.in‚0( disk.nvme0c0n1.out‚55418.20654296875# memory_percent‚90.8315896987915# proc.memory.rssMB‚ 2824.20703125* disk./.usagePercent‚37.408266100293346 disk.sda2.in‚0 disk.rbd0.out‚0# disk.dm-0.out‚55418.20654296875 disk.rbd1.in‚0áï&–”Û ‚(:‰ ÕèÒËÓˆ÷ gpu.0.gpu‚100.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚335.265' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚67.053 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0|4WCäÜ ‚(:Ù ÕèÒÅŒÎù disk.nvme0n1.out‚ 55431.4375 disk.sda3.out‚ 162.12890625! disk.nvme0c0n1.out‚ 55431.4375 disk.rbd1.out‚ 10.1015625 disk.drbd2165.out‚0 network.sent‚ 4.782031e+06$ disk./.usageGB‚557.4156455993652 disk.sda.out‚ 162.12890625 disk.dm-1.in‚0 disk.sda3.in‚0$ proc.memory.availableMB‚1502.375 cpu‚25.571670759898918 proc.cpu.threads‚17) disk./.usagePercent‚37.40863167905225 disk.sda2.in‚0 disk.rbd1.in‚0 disk.dm-1.out‚ 162.12890625 disk.dm-0.in‚ 8809.26171875 disk.drbd2165.in‚0 disk.sda.in‚0 disk.rbd0.out‚0 disk.dm-0.out‚ 55431.4375 disk.sda1.in‚0) proc.memory.percent‚17.23759174346924 disk.sda2.out‚0# proc.memory.rssMB‚ 2824.20703125# disk.nvme0c0n1.in‚ 8809.26171875! disk.nvme0n1.in‚ 8809.26171875 disk.rbd3.out‚0 disk.rbd0.in‚0 disk.sda1.out‚0 disk.rbd3.in‚0$ memory_percent‚90.83023071289062 network.recv‚ 1.100899e+06Û«/72Ý - _wandbruntime‚4238  _runtime‚4238 ÁÀqÇÞ Á % _timestamp‚1.7842940547322028e+09 _runtime‚4238.949338451 _step‚206 & lr-AdamW/pg1‚0.0006715289457732211 & lr-AdamW/pg2‚0.0006715289457732211 trainer/global_step‚2549ˆBNaÌß Æ _step‚206 & lr-AdamW/pg1‚0.0006715289457732211 & lr-AdamW/pg2‚0.0006715289457732211 trainer/global_step‚2549 % _timestamp‚1.7842940547322028e+09 _runtime‚4238.949338451Ο=Ò;2à - _wandbruntime‚4238  _runtime‚4238alߣÍá éO G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚13784665 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚198070764425216 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚201164514298368 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3093749873152 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 trainer/global_step‚2549 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7242578 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2397808651776 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚22063544 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3093786506752 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6542362 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚201164919023104 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2397808651776 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚21721707 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚22063544 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚734353689725440 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚21721701 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚ý©¥£"26173440 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚43795224 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _step‚207 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚21721707 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚731957226463232 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404724736 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚734034208713348 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚731640476348416 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚731957226463232 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 % _timestamp‚1.7842940547331972e+09 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚734355061288448 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚21721701 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚22073517 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚275 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51~tÔÀô'â îO YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36633600 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _step‚207 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚21721707 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚731957226463232 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404724736 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚734034208713348 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚731640476348416 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚731957226463232 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 % _timestamp‚1.7842940547331972e+09 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚734355061288448 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚21721701 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚22073517 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚275 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚13784665 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚198070764425216 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚201164514298368 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3093749873152 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 trainer/global_step‚2549 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7242578 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2397808651776 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚22063544 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3093786506752 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6542362 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚201164919023104 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2397808651776 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚21721707 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚22063544 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚734353689725440 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚21721701 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚43795224 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚13789975 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚198148250832896 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚22072022 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚732235253366784 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36626432 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚21729913 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6544900 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7245335 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚2549 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3094967525888 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2398733694464 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚43811922 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚201243586449920 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚732235227193344 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2397388297728 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚201243181732352 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 _runtime‚4240.332413486 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2394630403322 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2398733694464 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚731918358126592 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚43801929 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7245075 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2393285802888 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚43801929 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚732235227193344 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6544905 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 _step‚208 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚198148618924032 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚260 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚734633987061248 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚731918383554560 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚734632615491072 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚21729907 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0W+:å ÷M F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚734632615491072 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚21729907 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚21729913 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2397388297728 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚265 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚734633987061248 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚22082009 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3094930899456 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚13789975 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚198148250832896 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚22072022 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚732235253366784 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36626432 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚21729913 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6544900 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7245335 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚2549 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3094967525888 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2398733694464 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚43811922 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚201243586449920 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚732235227193344 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2397388297728 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚201243181732352 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 _runtime‚4240.332413486 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2394630403322 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2398733694464 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚731918358126592 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚43801929 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7245075 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2393285802888 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚43801929 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚732235227193344 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6544905 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_traiÄ(‚`n_batch_end/segment.small_pool.current‚820 _step‚208 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚198148618924032 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚260 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚734633987061248 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚731918383554560ÐhF2æ - _wandbruntime‚4240  _runtime‚4240@ez°ç  /loss/FBt1/detection_loss‚0.9885365757509135 %loss/total_loss‚9.474142355363282 epoch‚6 2loss/SRqqt1/assignment_loss‚0.6924548583074189 1loss/SRqqt1/detection_loss‚1.1316001494205636 _runtime‚4240.419171266 trainer/global_step‚2549 _step‚209 /loss/FRt2/assignment_loss‚0.808599408610621 2loss/FBt2/assignment_loss‚0.036668016744398735 /loss/FRt1/detection_loss‚0.5212381899042549 .loss/FBt2/detection_loss‚0.511037321338479 % _timestamp‚1.7842940562022007e+09 0loss/FRt1/assignment_loss‚1.9771385738153777 2loss/SRqqt2/assignment_loss‚0.5931111386429225 2loss/FBt1/assignment_loss‚0.035684029297552655 /loss/FRt2/detection_loss‚1.5418635238234941 1loss/SRqqt2/detection_loss‚0.6362105697072853ÏRÓ è † 2loss/SRqqt2/assignment_loss‚0.5931111386429225 2loss/FBt1/assignment_loss‚0.035684029297552655 /loss/FRt2/detection_loss‚1.5418635238234941 1loss/SRqqt2/detection_loss‚0.6362105697072853 /loss/FBt1/detection_loss‚0.9885365757509135 %loss/total_loss‚9.474142355363282 epoch‚6 2loss/SRqqt1/assignment_loss‚0.6924548583074189 1loss/SRqqt1/detection_loss‚1.1316001494205636 _runtime‚4240.419171266 trainer/global_step‚2549 _step‚209 /loss/FRt2/assignment_loss‚0.808599408610621 2loss/FBt2/assignment_loss‚0.036668016744398735 /loss/FRt1/detection_loss‚0.5212381899042549 .loss/FBt2/detection_loss‚0.511037321338479 % _timestamp‚1.7842940562022007e+09 0loss/FRt1/assignment_loss‚1.9771385738153777Ñ°ypÊŸé ‚(:” ¬ÕèÒ¡é…÷ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚295.399' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚59.079800000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0øÕñê ‚(:æ ¬ÕèÒ”ž†ø disk.dm-0.in‚ 8952.4453125 disk.sda2.in‚0 disk.sda.in‚0 proc.cpu.threads‚17 network.sent‚ 4.786158e+06 disk.sda1.in‚0 disk.sda2.out‚0 disk.rbd1.in‚0 disk.sda.out‚ 162.296875 disk.dm-1.out‚ 162.296875# memory_percent‚90.8306360244751# disk./.usageGB‚557.421558380127 disk.sda3.out‚ 162.296875 disk.drbd2165.out‚0 disk.rbd3.in‚0# proc.memory.rssMB‚ 2824.20703125* disk./.usagePercent‚37.409028490730314" disk.dm-0.out‚55438.3857421875' disk.nvme0c0n1.out‚55438.3857421875 cpu‚25.571262783963117 disk.sda1.out‚0" disk.nvme0c0n1.in‚ 8952.4453125% disk.nvme0n1.out‚55438.3857421875) proc.memory.availableMB‚ 1502.30859375) proc.memory.percent‚17.23759174346924 disk.rbd0.in‚0 disk.drbd2165.in‚0 disk.rbd3.out‚0 disk.nvme0n1.in‚ 8952.4453125 network.recv‚ 1.103646e+06 disk.rbd0.out‚0 disk.sda3.in‚0 disk.rbd1.out‚ 10.2265625 disk.dm-1.in‚0Èh¬¿ë ‚(:’ »ÕèÒå­àö gpu.0.gpu‚26.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚46.0 gpu.0.powerWatts‚337.288' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚67.45760000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0™Ô™êØì ‚(:Í »ÕèÒõŒ†ù disk.sda2.out‚0 disk.rbd1.in‚0 disk.dm-0.in‚ 9120.69921875 disk.sda3.in‚0# disk.nvme0c0n1.in‚ 9120.69921875 disk.dm-0.out‚55450 disk.drbd2165.in‚0 proc.cpu.threads‚17 network.sent‚ 4.850312e+06$ disk./.usageGB‚557.4271812438965 disk.rbd3.in‚0 disk.sda2.in‚0 disk.sda.in‚0 disk.rbd0.out‚0 disk.sda1.in‚0* disk./.usagePercent‚37.409405845835785 disk.nvme0c0n1.out‚55450 disk.nvme0n1.out‚55450 disk.sda.out‚ 162.5703125 disk.dm-1.out‚ 162.5703125 network.recv‚ 1.108835e+06 disk.rbd3.out‚0 disk.rbd1.out‚ 10.2265625 disk.drbd2165.out‚0) proc.memory.availableMB‚ 1502.04296875) proc.memory.percent‚17.23759174346924 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.sda1.out‚0 disk.sda3.out‚ 162.5703125$ memory_percent‚90.83225727081299 cpu‚25.571817866218364! disk.nvme0n1.in‚ 9120.69921875# proc.memory.rssMB‚ 2824.20703125´ ¡ ”í ‚(:‰ ÊÕèÒŠœÖö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚266.957' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.3914 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0b {Èùî ‚(:î ÊÕèÒ¿ƒÈø disk.rbd1.out‚ 10.2890625 disk.drbd2165.out‚0 disk.sda3.out‚ 162.9453125) proc.memory.availableMB‚ 1501.96484375 disk.rbd3.in‚0# disk.nvme0c0n1.in‚ 9120.69921875 disk.dm-1.out‚ 162.9453125& disk.nvme0n1.out‚55459.44091796875 disk.sda.in‚0 network.sent‚ 4.854082e+06( disk./.usagePercent‚37.4097215031255# disk.dm-0.out‚55459.44091796875 disk.sda3.in‚0 disk.sda2.out‚0 disk.dm-1.in‚0 disk.rbd0.in‚0$ memory_percent‚90.83273410797119# proc.memory.rssMB‚ 2824.20703125 proc.cpu.threads‚17 disk.dm-0.in‚ 9120.69921875 disk.sda1.out‚0 disk.sda.out‚ 162.9453125 disk.rbd0.out‚0# disk./.usageGB‚557.431884765625 disk.sda1.in‚0( disk.nvme0c0n1.out‚55459.44091796875 disk.sda2.in‚0 disk.rbd1.in‚0 disk.drbd2165.in‚0 disk.rbd3.out‚0 network.recv‚ 1.112182e+06! disk.nvme0n1.in‚ 9120.69921875) proc.memory.percent‚17.23759174346924 cpu‚25.571980444497473~×[Fœï ‚(:‘ ÙÕèÒ®†æö gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚263.184' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚52.63680000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0÷¥Ûõð ‚(:ê ÙÕèÒÕÉÔø disk.sda2.out‚0) disk./.usagePercent‚37.41004944877687 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.rbd3.in‚0' disk.nvme0c0n1.out‚55471.7509765625! disk.nvme0n1.in‚ 9120.69921875$ memory_percent‚90.83454608917236 network.sent‚ 4.859151e+06 disk.dm-1.out‚ 163.234375# disk.nvme0c0n1.in‚ 9120.69921875% disk.nvme0n1.out‚55471.7509765625 disk.rbd1.out‚ 10.2890625 disk.sda.out‚ 163.234375) proc.memory.availableMB‚ 1501.66796875 proc.cpu.threads‚17 disk.rbd1.in‚0 disk.rbd0.out‚0 disk.drbd2165.out‚0 disk.sda2.in‚0 disk.sda.in‚0$ disk./.usageGB‚557.4367713928223 disk.dm-0.in‚ 9120.69921875# proc.memory.rssMB‚ 2824.20703125 disk.dm-1.in‚0 disk.sda3.out‚ 163.234375 network.recv‚ 1.116037e+06 disk.rbd0.in‚0" disk.dm-0.out‚55471.7509765625 disk.sda3.in‚0) proc.memory.percent‚17.23759174346924 cpu‚25.572598291116847 disk.sda1.in‚0 disk.rbd3.out‚0vÀËÅŸñ ‚(:” èÕèÒƒŒ„÷ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚289.503' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚57.900600000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0e±L ÷ò ‚(:ì èÕèÒ©¿Éù disk.sda.out‚ 163.671875 disk.drbd2165.out‚0# proc.memory.rssMB‚ 2824.20703125 disk.rbd1.out‚ 10.3515625# disk.dm-0.out‚55480.77392578125 disk.sda3.out‚ 163.671875( proc.memory.availableMB‚ 1501.7890625) proc.memory.percent‚17.23759174346924 cpu‚25.573144050503828 disk.rbd1.in‚0 network.sent‚ 4.864196e+06) disk./.usagePercent‚37.41034385744124 disk.sda.in‚0 disk.sda1.in‚0 network.recv‚ 1.119365e+06 disk.sda1.out‚0 disk.rbd3.in‚0 disk.rbd3.out‚0$ memory_percent‚90.83380699157715 disk.rbd0.out‚0! disk.nvme0n1.in‚ 9120.69921875& disk.nvme0n1.out‚55480.77392578125 disk.sda2.out‚0 disk.dm-1.in‚0 disk.dm-0.in‚ 9120.69921875 disk.drbd2165.in‚0# disk.nvme0c0n1.in‚ 9120.69921875 disk.sda2.in‚0 disk.dm-1.out‚ 163.671875 disk.rbd0.in‚0 disk.sda3.in‚0( disk.nvme0c0n1.out‚55480.77392578125 proc.cpu.threads‚17$ disk./.usageGB‚557.4411582946777žûì)2ó - _wandbruntime‚4318  _runtime‚4318§³‡HÃô ½ _runtime‚ 4318.2579226 _step‚210 & lr-AdamW/pg1‚0.0006705823335532986 & lr-AdamW/pg2‚0.0006705823335532986 trainer/global_step‚2599 # _timestamp‚1.78429413404074e+09z¿2Èõ  _step‚210 & lr-AdamW/pg1‚0.0006705823335532986 & lr-AdamW/pg2‚0.0006705823335532986 trainer/global_step‚2599 # _timestamp‚1.78429413404074e+09 _runtime‚ 4318.2579226ÒbGö2ö - _wandbruntime‚4318  _runtime‚4318* x¬ï'÷ éO VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚748301413724672 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚22489296 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚205098861080576 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚747973164706966 % _timestamp‚1.7842941340417676e+09 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚747974534735256 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2442779192832 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2444124582400 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2442779192832 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2438599449750 trainer/global_step‚2599 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚22489296 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚205098458452992 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚748300042161664 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚745857262968832 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2439944050072 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚201945084809216 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚748300042161664 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚22132007 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3153373643776 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚260 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚745857289142272 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚22499269 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚745857289142272 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚14050888 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7381381 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚201945452900352 _step‚211 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚22132001 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚14051148 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7381636 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚745857262968832 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚745534590685184 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚745534565257216 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚22132001 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚22499269 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚255 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640ÎôÄ{ô'ø îO @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚748300042161664 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚22132007 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3153373643776 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚260 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚745857289142272 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚22499269 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚745857289142272 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚14050888 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7381381 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚201945452900352 _step‚211 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚22132001 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚14051148 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7381636 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚745857262968832 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚745534590685184 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚745534565257216 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚22132001 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚22499269 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚255 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚748301413724672 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚22489296 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚205098861080576 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚747973164706966 % _timestamp‚1.7842941340417676e+09 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚747974534735256 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2442779192832 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2444124582400 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2442779192832 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2438599449750 trainer/global_step‚2599 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚22489296 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚205098458452992 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚748300042161664 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚745857262968832 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2439944050072 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚201945084809216 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448Ó7Ū_2ù - _wandbruntime‚4319  _runtime‚43199£ú òM S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚748251969951388 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7384127 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚746135289872384 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 trainer/global_step‚2599 _step‚212 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚746135263698944 REDeviceStatsMonitor.on_train_batch_end/allocatiÍ{éô"on.large_pool.allocated‚22140213 _runtime‚4319.645907754 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚748580339497472 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚205177538321408 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14056177 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚44647974 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚44637981 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚22497774 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2439522915996 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚44647974 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚22497774 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3154599013376 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2445049625088 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2443704228352 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚22140213 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚746135263698944 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚255 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚22140207 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚202022939308032 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3154564484096 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚205177538321408 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14056177 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚44647974 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚44637981 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚22497774 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2439522915996 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚44647974 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚22497774 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3154599013376 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2445049625088 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2443704228352 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚22140213 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚746135263698944 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚255 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚22140207 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚202022939308032 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3154564484096 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7384127 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚746135289872384 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 trainer/global_step‚2599 _step‚212Ô«\€2ü - _wandbruntime‚4319  _runtime‚4319ªq\«ý ‚ 1loss/SRqqt2/detection_loss‚0.6385018807743003 .loss/FBt1/detection_loss‚0.592583123320604 %loss/total_loss‚8.644039114913422 _runtime‚4319.730337901 _step‚213 2loss/FBt2/assignment_loss‚0.006466353575964934 1loss/SRqqt1/detection_loss‚0.9753495497992941 trainer/global_step‚2599 /loss/FRt1/detection_loss‚0.5496423833183027 /loss/FRt2/detection_loss‚1.5771816758694308 /loss/FBt2/detection_loss‚0.3953095154959506 % _timestamp‚1.7842941355133648e+09 0loss/FRt1/assignment_loss‚1.8238381085818602 0loss/FRt2/assignment_loss‚0.8228308005679038 2loss/SRqqt2/assignment_loss‚0.5438989941585306 epoch‚6 2loss/SRqqt1/assignment_loss‚0.7025492382841031 2loss/FBt1/assignment_loss‚0.015887491167175895ú…! þ ‡ trainer/global_step‚2599 /loss/FRt1/detection_loss‚0.5496423833183027 /loss/FRt2/detection_loss‚1.5771816758694308 /loss/FBt2/detection_loss‚0.3953095154959506 % _timestamp‚1.7842941355133648e+09 0loss/FRt1/assignment_loss‚1.8238381085818602 0loss/FRt2/assignment_loss‚0.8228308005679038 2loss/SRqqt2/assignment_loss‚0.5438989941585306 epoch‚6 2loss/SRqqt1/assignment_loss‚0.7025492382841031 2loss/FBt1/assignment_loss‚0.015887491167175895 1loss/SRqqt2/detection_loss‚0.6385018807743003 .loss/FBt1/detection_loss‚0.592583123320604 %loss/total_loss‚8.644039114913422 _runtime‚4319.730337901 _step‚213 2loss/FBt2/assignment_loss‚0.006466353575964934 1loss/SRqqt1/detection_loss‚0.9753495497992941ÕÌQÛažÿ ‚(:“ ÷ÕèÒ‰™Íö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚333.524' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚66.70479999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ðòµ÷€‚(:ì ÷ÕèÒò©Âø disk.rbd1.in‚0 proc.cpu.threads‚17 disk.sda1.in‚0 network.sent‚ 4.8693e+06 disk.drbd2165.out‚0 disk.rbd3.in‚0" disk.nvme0c0n1.in‚ 9268.4921875 disk.sda2.in‚0 disk.dm-1.in‚0) proc.memory.percent‚17.23759174346924$ disk./.usageGB‚557.4475440979004 disk.dm-0.in‚ 9268.4921875( disk.nvme0c0n1.out‚55491.70849609375 disk.nvme0n1.in‚ 9268.4921875 disk.rbd1.out‚ 10.3515625 disk.sda.out‚ 163.98046875 disk.sda3.in‚0 disk.sda2.out‚0 disk.sda.in‚0 disk.rbd3.out‚0& disk.nvme0n1.out‚55491.70849609375 disk.rbd0.out‚0$ memory_percent‚90.83681106567383 disk.sda1.out‚0 network.recv‚ 1.122948e+06) disk./.usagePercent‚37.41077241405356 disk.dm-1.out‚ 163.98046875 disk.rbd0.in‚0# disk.dm-0.out‚55491.70849609375' proc.memory.availableMB‚ 1501.296875 cpu‚25.573652757759223 disk.sda3.out‚ 163.98046875 disk.drbd2165.in‚0# proc.memory.rssMB‚ 2824.20703125˜ïP`“‚(:ˆ †ÖèÒÌ å÷ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚287.615' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚57.523 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0à?RJò‚‚(:ç †ÖèÒ…‡Šù disk.dm-0.in‚ 9268.4921875 disk.sda1.out‚0 disk.drbd2165.out‚0 disk.rbd1.in‚0 disk.dm-1.in‚0 disk.sda3.in‚0% disk.nvme0c0n1.out‚55502.94921875 disk.sda.out‚ 164.55078125 disk.rbd0.out‚0 proc.cpu.threads‚17# disk.nvme0n1.out‚55502.94921875 disk.rbd1.out‚ 10.4140625 disk.sda.in‚0) proc.memory.availableMB‚ 1501.37109375 disk.drbd2165.in‚0 disk.rbd3.in‚0 disk.sda2.in‚0 cpu‚25.574080716717738 disk.rbd3.out‚0" disk.nvme0c0n1.in‚ 9268.4921875$ memory_percent‚90.83635807037354) proc.memory.percent‚17.23759174346924 network.recv‚ 1.127517e+06) disk./.usagePercent‚37.41102202139944$ disk./.usageGB‚557.4512634277344 disk.dm-0.out‚55502.94921875 disk.sda1.in‚0 disk.nvme0n1.in‚ 9268.4921875 network.sent‚ 4.932196e+06 disk.rbd0.in‚0# proc.memory.rssMB‚ 2824.20703125 disk.sda3.out‚ 164.55078125 disk.sda2.out‚0 disk.dm-1.out‚ 164.55078125ó\ Y”ƒ‚(:‰ •ÖèÒ¸³Éö gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚336.177' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚67.2354 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0w884÷„‚(:ì •ÖèÒî‘ø) disk./.usagePercent‚37.39561548798886$ disk./.usageGB‚557.2216949462891 disk.sda.out‚ 165.60546875 disk.sda1.out‚0" disk.nvme0c0n1.in‚ 9268.4921875$ memory_percent‚90.83638191223145( proc.memory.availableMB‚ 1501.3671875 disk.rbd1.out‚ 10.4140625 disk.drbd2165.out‚0 cpu‚25.574638590843154 disk.rbd1.in‚0 disk.drbd2165.in‚0 disk.sda2.in‚0 disk.dm-1.out‚ 165.60546875 disk.rbd3.in‚0 disk.sda2.out‚0) proc.memory.percent‚17.23759174346924 disk.dm-0.in‚ 9268.4921875 proc.cpu.threads‚17 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.sda3.in‚0" disk.dm-0.out‚55520.4052734375 network.sent‚ 4.937231e+06 network.recv‚ 1.131229e+06 disk.sda.in‚0 disk.rbd0.out‚0 disk.sda3.out‚ 165.60546875 disk.nvme0n1.in‚ 9268.4921875% disk.nvme0n1.out‚55520.4052734375 disk.sda1.in‚0 disk.rbd3.out‚0' disk.nvme0c0n1.out‚55520.4052734375# proc.memory.rssMB‚ 2824.20703125Й½½”…‚(:‰ ¤ÖèÒÀŽòö gpu.0.gpu‚91.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚276.103' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.2206 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0.Mñ†‚(:æ ¤ÖèÒþã‘ù disk.nvme0n1.in‚ 9268.4921875& disk.nvme0n1.out‚55531.08447265625* disk./.usagePercent‚37.395961866182695 disk.sda1.out‚0 disk.rbd3.in‚0 disk.rbd3.out‚0$ memory_percent‚90.83595275878906) proc.memory.percent‚17.23759174346924 disk.sda2.in‚0 proc.cpu.threads‚17 disk.rbd1.in‚0 disk.rbd0.out‚0 disk.sda3.in‚0 disk.sda3.out‚ 166.046875 disk.dm-1.in‚0# disk.dm-0.out‚55531.08447265625 disk.sda1.in‚0" disk.nvme0c0n1.in‚ 9268.4921875 network.sent‚ 4.94357e+06 disk.sda.out‚ 166.046875 disk.drbd2165.in‚0 disk.drbd2165.out‚0% proc.memory.availableMB‚ 1501.4375 cpu‚25.574997781462613 network.recv‚ 1.134482e+06 disk.sda.in‚0 disk.dm-0.in‚ 9268.4921875( disk.nvme0c0n1.out‚55531.08447265625 disk.sda2.out‚0 disk.rbd1.out‚ 10.4765625 disk.dm-1.out‚ 166.046875 disk.rbd0.in‚0$ disk./.usageGB‚557.2268562316895# proc.memory.rssMB‚ 2824.20703125À7ľ”‡‚(:‰ ³ÖèÒ÷ïæö gpu.0.gpu‚94.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚39.0 gpu.0.powerWatts‚114.889' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.9778 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0b‡ïˆ‚(:ä ³ÖèÒ˜ýø disk.dm-1.in‚0 disk.rbd0.in‚0 network.recv‚ 1.13787e+06 disk.sda2.in‚0 disk.sda.out‚ 166.328125 disk.dm-1.out‚ 166.328125 disk.sda3.out‚ 166.328125 disk.rbd0.out‚0" disk.dm-0.out‚55560.6513671875( proc.memory.availableMB‚ 1501.4296875 proc.cpu.threads‚17 disk.rbd1.out‚ 10.4765625 disk.sda1.out‚0$ memory_percent‚90.83600044250488 network.sent‚ 4.94733e+06) disk./.usagePercent‚37.39653609108199' disk.nvme0c0n1.out‚55560.6513671875 disk.rbd3.out‚0 disk.drbd2165.out‚0 disk.rbd3.in‚0 disk.sda2.out‚0 disk.dm-0.in‚ 9379.1796875 disk.drbd2165.in‚0) proc.memory.percent‚17.23759174346924$ disk./.usageGB‚557.2354125976562" disk.nvme0c0n1.in‚ 9379.1796875 disk.nvme0n1.in‚ 9379.1796875 disk.sda1.in‚0 disk.sda3.in‚0 cpu‚25.575312091691607% disk.nvme0n1.out‚55560.6513671875 disk.rbd1.in‚0# proc.memory.rssMB‚ 2824.20703125 disk.sda.in‚0ë¼D ”‰‚(:‰ ÂÖèÒÅøÞö gpu.0.gpu‚99.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚287.327' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚57.4654 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0QäòŠ‚(:ç ÂÖèÒ öø# proc.memory.rssMB‚ 2824.20703125" disk.nvme0c0n1.in‚ 9379.1796875 network.sent‚ 4.952322e+06 disk.drbd2165.in‚0 disk.rbd3.out‚0 disk.sda.in‚0$ disk./.usageGB‚557.2410011291504 disk.sda1.in‚0 disk.dm-1.in‚0 disk.sda1.out‚0 disk.dm-1.out‚ 166.578125" disk.dm-0.out‚55569.8349609375$ memory_percent‚90.83600044250488( proc.memory.availableMB‚ 1501.4296875 disk.nvme0n1.in‚ 9379.1796875) proc.memory.percent‚17.23759174346924 proc.cpu.threads‚17 disk.drbd2165.out‚0 disk.rbd1.out‚ 10.6015625' disk.nvme0c0n1.out‚55569.8349609375 disk.sda2.in‚0 disk.sda.out‚ 166.578125 disk.rbd0.in‚0 disk.dm-0.in‚ 9379.1796875 network.recv‚ 1.141356e+06 disk.sda3.in‚0 disk.sda3.out‚ 166.578125 disk.rbd1.in‚0 disk.rbd0.out‚0 cpu‚25.575962354795514* disk./.usagePercent‚37.396911142119656 disk.rbd3.in‚0% disk.nvme0n1.out‚55569.8349609375 disk.sda2.out‚0ÛT²2‹- _wandbruntime‚4397  _runtime‚4397&Æ#pÆŒÀ trainer/global_step‚2649 $ _timestamp‚1.784294213161742e+09 _runtime‚4397.378967844 _step‚214 & lr-AdamW/pg1‚0.0006696148691497972 & lr-AdamW/pg2‚0.0006696148691497972¤šËÅ $ _timestamp‚1.784294213161742e+09 _runtime‚4397.378967844 _step‚214 & lr-AdamW/pg1‚0.0006696148691497972 & lr-AdamW/pg2‚0.0006696148691497972 trainer/global_step‚2649ÖGͤm2Ž- _wandbruntime‚4397  _runtime‚4397s’4 éO A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7519627 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚22915048 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3213229029376 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚22925021 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6796657 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 _runtime‚4397.380259483 _step‚215 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚762247766160896 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚761913490728874 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚209033002313728 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚762246394597888 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚759757325647872 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚209032599686144 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 % _timestamp‚1.7842942131628535e+09 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2486181163180 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2490440513024 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚45467328 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚759428654166016 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2490440513024 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚205819773284352 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚22542301 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚22915048 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚762247766160896 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚762246394597888 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚249 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚759428679593984 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚759757325647872 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚22925021 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚205819405193216 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3213194492928 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2489095123456 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚14316538 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚14316284 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820¨¬]›ô'îO \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚759428654166016 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2490440513024 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚205819773284352 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚22542301 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚22915048 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚762247766160896 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚762246394597888 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚249 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚759428679593984 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚759757325647872 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚22925021 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚205819405193216 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3213194492928 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2489095123456 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚14316538 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚14316284 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7519627 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚22915048 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3213229029376 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚22925021 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6796657 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 _runtime‚4397.380259483 _step‚215 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚762247766160896 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚761913490728874 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚209033002313728 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚762246394597888 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚759757325647872 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚209032599686144 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 % _timestamp‚1.7842942131628535e+09 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2486181163180 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2490440513024 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚45467328 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0×oж¢2‘- _wandbruntime‚4398  _runtime‚4398¨aèù&’óM \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚209111688713216 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2491365555712 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚762193666001698 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚762192295973296 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚45484026 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚22923526 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 trainer/global_step‚2649 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2491365555712 _step‚216 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6799200 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚760035300204544 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚22550507 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7522338 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3214429021184 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚45474033 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚22550513 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6799205 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚759706535944192 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚22923526 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚242 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚237 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚45474033 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚760035326377984 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚205896891600896 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚762526691933696 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14321538 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚22933513 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7842942145475633e+09 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2487104629538 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚45484026 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚762525320363520 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚22550507 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚14321780 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚4398.765042135þ¿tÕ“øM XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚209111286092800 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3214394491904 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚760035300204544 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚760035300204544 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚22550507 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7522338 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3214429021184 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚45474033 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚22550513 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6799205 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚759706535944192 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚22923526 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚242 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚237 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚45474033 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚760035326377984 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚205896891600896 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚762526691933696 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 TM–Å)EDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14321538 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚22933513 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7842942145475633e+09 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2487104629538 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚45484026 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚762525320363520 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚22550507 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚14321780 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚4398.765042135 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚209111688713216 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2491365555712 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚762193666001698 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚762192295973296 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚45484026 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚22923526 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 trainer/global_step‚2649 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2491365555712 _step‚216 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6799200 EDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6923807 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚23350773 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚209694093668352 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚212967112828928 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚22952601 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6923812 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚775855186779072 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 $ _timestamp‚1.784294292505797e+09 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚776192747034112 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚773322743074816 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚776192747034112 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2535411054080 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚22952607 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚773657362153472 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2535411054080 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _runtime‚4476.723135078 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚773657335980032 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚773657335980032 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚22952607 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2531073675966 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚255 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚14581863 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚212966710201344 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7658051 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚23340800 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚14581603 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7657796 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚23350773 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚209693725577216 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3273019160576 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚773322768502784 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 trainer/global_step‚2699 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚23340800 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚260 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚775853816750782 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2536756443648Q¶ŽÒk!¦íO P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚775855186779072 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 $ _timestamp‚1.784294292505797e+09 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚776192747034112 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚773322743074816 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚776192747034112 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2535411054080 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚22952607 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚773657362153472 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2535411054080 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _runtime‚4476.723135078 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚773657335980032 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚773657335980032 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚22952607 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2531073675966 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚255 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚14581863 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚212966710201344 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7658051 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚23340800 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚14581603 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7657796 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚23350773 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚209693725577216 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3273019160576 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚773322768502784 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 trainer/global_step‚2699 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚23340800 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚260 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚775853816750782 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2536756443648 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚776194118597120 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚773657362153472 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚46303380 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6923807 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚23350773 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 TDDeviceStatsMon®˜$Šˆitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚209694093668352 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚212967112828928 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚22952601 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6923812 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820ÛºUïh2§- _wandbruntime‚4478  _runtime‚4478·°’ðù&¨óM E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚773935336710144 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚23349278 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2537681486336 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚23359265 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚209771580076032 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚14587133 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚776132621995204 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚2699 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚773935336710144 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 _runtime‚4478.107987437 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚773935362883584 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚776133992023606 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚23349278 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14586882 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7660532 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚251 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6926350 _step‚220 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚46310085 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚773600650280960 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2537681486336 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2536336089600 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚22960813 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚213045795787776 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2533341742646 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚776473044369920 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2536336089600 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚46320078 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚46310085 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚246 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚46320078 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7660778 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚209771211984896 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3274215711744 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚22960807 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0VU þ&©øM =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚776132621995204 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚2699 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚773935336710144 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 _runtime‚4478.107987437 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚773935362883584 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚776133992023606 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚23349278 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14586882 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7660532 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚251 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6926350 _step‚220 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚46310085 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚773600650280960 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2537681486336 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2536336089600 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚22960813 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚213045795787776 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2533341742646 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚776473044369920 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2536336089600 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚46320078 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚46310085 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚246 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚46320078 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7660778 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚209771211984896 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3274215711744 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚22960807 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚773935336710144 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚23349278 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2537681486336 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚23359265 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚209771580076032 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚14587133 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175Ü8¡£2ª- _wandbruntime‚4478  _runtime‚4478Äí/Q «ƒ /loss/FRt1/detection_loss‚0.5644219324003118 1loss/SRqqt1/detection_loss‚0.9931908369358259 1loss/SRqqt2/detection_loss‚0.6514168644616151 3loss/FBt2/assignment_loss‚0.0057759630662809905 % _timestamp‚1.7842942939773316e+09 _runtime‚4478.194277024 /loss/FBt2/detection_loss‚0.5972484732488592 /loss/FRt2/detection_loss‚1.5331922485985887 /loss/FBt1/detection_loss‚0.9142869550734758 trainer/global_step‚2699 _step‚221 0loss/FRt1/assignment_loss‚1.9043279009674157 2loss/FBt1/assignment_loss‚0.020286013837707863 #loss/total_loss‚9.0290592657568 epoch‚6 0loss/FRt2/assignment_loss‚0.7583581258064855 2loss/SRqqt1/assignment_loss‚0.6002145625085009 3loss/SRqqt2/assignment_loss‚0.48633938885173317¾o0ଈ 0loss/FRt1/assignment_loss‚1.9043279009674157 2loss/FBt1/assignment_loss‚0.020286013837707863 #loss/total_loss‚9.0290592657568 epoch‚6 0loss/FRt2/assignment_loss‚0.7583581258064855 2loss/SRqqt1/assignment_loss‚0.6002145625085009 3loss/SRqqt2/assignment_loss‚0.48633938885173317 /loss/FRt1/detection_loss‚0.5644219324003118 1loss/SRqqt1/detection_loss‚0.9931908369358259 1loss/SRqqt2/detection_loss‚0.6514168644616151 3loss/FBt2/assignment_loss‚0.0057759630662809905 % _timestamp‚1.7842942939773316e+09 _runtime‚4478.194277024 /loss/FBt2/detection_loss‚0.5972484732488592 /loss/FRt2/detection_loss‚1.5331922485985887 /loss/FBt1/detection_loss‚0.9142869550734758 trainer/global_step‚2699 _step‚221Ý8:“Õ­‚(:Š œ×èÒ½ÎØö gpu.0.gpu‚100.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚343.177' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚68.6354 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Šè!ú®‚(:ï œ×èÒžžÞø disk.dm-1.out‚ 168.71484375 disk.sda3.out‚ 168.71484375 disk.drbd2165.in‚0 disk.sda.in‚0 network.sent‚ 5.04054e+06$ disk./.usageGB‚557.2640724182129 disk.drbd2165.out‚0 disk.rbd3.in‚0" disk.nvme0c0n1.in‚ 9670.8359375 disk.nvme0n1.in‚ 9670.8359375 disk.rbd1.in‚0 disk.rbd0.out‚0( disk.nvme0c0n1.out‚55628.62841796875 disk.sda2.in‚0# proc.memory.rssMB‚ 2824.20703125 cpu‚25.578718033373516 proc.cpu.threads‚17) disk./.usagePercent‚37.39845947568674 disk.sda3.in‚0 disk.rbd0.in‚0# disk.dm-0.out‚55628.62841796875 disk.rbd3.out‚0& disk.nvme0n1.out‚55628.62841796875 disk.rbd1.out‚ 10.7890625 disk.sda.out‚ 168.71484375 disk.dm-1.in‚0 disk.sda2.out‚0) proc.memory.availableMB‚ 1501.29296875 disk.dm-0.in‚ 9670.8359375$ memory_percent‚90.83683490753174) proc.memory.percent‚17.23759174346924 network.recv‚ 1.163262e+06 disk.sda1.in‚0 disk.sda1.out‚0ìÓþ”¯‚(:‰ «×èÒßø³÷ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚257.521' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚51.5042 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ËH9nÿ°‚(:ô «×èÒÙ´…ù disk.sda1.in‚0 disk.drbd2165.in‚0 disk.sda2.in‚0# proc.memory.rssMB‚ 2824.20703125 network.recv‚ 1.168072e+06 disk.rbd0.in‚0 disk.drbd2165.out‚0 disk.rbd1.in‚0 disk.sda3.in‚0# disk.nvme0c0n1.in‚ 9729.23828125) proc.memory.percent‚17.23759174346924 cpu‚25.578950916767088* disk./.usagePercent‚37.398971234747684 disk.dm-0.in‚ 9729.23828125# disk.dm-0.out‚55642.18212890625 disk.rbd3.out‚0( disk.nvme0c0n1.out‚55642.18212890625& disk.nvme0n1.out‚55642.18212890625 disk.rbd1.out‚ 10.7890625 disk.sda1.out‚0 disk.rbd3.in‚0$ memory_percent‚90.83693027496338) proc.memory.availableMB‚ 1501.27734375 proc.cpu.threads‚17 disk.sda.out‚ 169.10546875 disk.dm-1.out‚ 169.10546875 disk.sda3.out‚ 169.10546875! disk.nvme0n1.in‚ 9729.23828125 disk.sda2.out‚0 network.sent‚ 5.103473e+06 disk.rbd0.out‚0$ disk./.usageGB‚557.2716979980469 disk.sda.in‚0 disk.dm-1.in‚0B~bÊ”±‚(:‰ º×èÒ£Ãøö gpu.0.gpu‚87.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚333.089' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚66.6178 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚02aíYù²‚(:î º×èÒΰÇø disk.dm-0.in‚ 9936.86328125# disk.nvme0c0n1.in‚ 9936.86328125 disk.rbd1.in‚0 disk.rbd1.out‚ 10.8515625( proc.memory.availableMB‚ 1501.2734375 proc.cpu.threads‚17" disk.dm-0.out‚55651.9306640625 disk.sda3.out‚ 169.37109375 disk.drbd2165.out‚0' disk.nvme0c0n1.out‚55651.9306640625# proc.memory.rssMB‚ 2824.20703125! disk.nvme0n1.in‚ 9936.86328125% disk.nvme0n1.out‚55651.9306640625) proc.memory.percent‚17.23759174346924$ disk./.usageGB‚557.2764854431152 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda2.out‚0 network.recv‚ 1.171065e+06) disk./.usagePercent‚37.39929252420316 disk.dm-1.out‚ 169.37109375 disk.drbd2165.in‚0 disk.rbd3.in‚0 disk.sda2.in‚0 disk.sda1.in‚0 disk.sda3.in‚0 disk.rbd3.out‚0 disk.sda.out‚ 169.37109375 cpu‚25.579356679602576 network.sent‚ 5.10847e+06 disk.dm-1.in‚0 disk.sda1.out‚0 disk.sda.in‚0$ memory_percent‚90.83695411682129c ž³‚(:“ É×èÒôü…÷ gpu.0.gpu‚88.0 gpu.0.memory‚25, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚291.095' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚58.21900000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ïtBý´‚(:ò É×èÒÚ™Ðù disk.dm-0.in‚ 9936.86328125 disk.sda2.out‚0 network.sent‚ 5.113619e+06$ disk./.usageGB‚557.2818870544434) proc.memory.availableMB‚ 1501.69140625 disk.dm-1.in‚0 disk.dm-1.out‚ 169.73046875 disk.sda1.in‚0 cpu‚25.579796390268918 disk.sda.in‚0# disk.nvme0c0n1.in‚ 9936.86328125! disk.nvme0n1.in‚ 9936.86328125 disk.sda3.in‚0# memory_percent‚90.8344030380249 disk.sda.out‚ 169.73046875 disk.rbd0.out‚0# disk.dm-0.out‚55663.10498046875 disk.sda2.in‚0 network.recv‚ 1.174456e+06 disk.drbd2165.in‚0 disk.rbd3.in‚0( disk.nvme0c0n1.out‚55663.10498046875 disk.drbd2165.out‚0& disk.nvme0n1.out‚55663.10498046875) proc.memory.percent‚17.23759174346924) disk./.usagePercent‚37.39965503087164 disk.rbd3.out‚0 disk.rbd1.in‚0 disk.sda1.out‚0# proc.memory.rssMB‚ 2824.20703125 disk.rbd1.out‚ 10.8515625 disk.rbd0.in‚0 disk.sda3.out‚ 169.73046875 proc.cpu.threads‚17NÝÿ“µ‚(:ˆ Ø×èÒƒ‘ýö gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚289.215' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚57.843 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0I’¦Ší¶‚(:â Ø×èÒòë¦ù) disk./.usagePercent‚37.39999884899012# disk.dm-0.out‚55672.20166015625# memory_percent‚90.8362865447998 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.dm-0.in‚ 9936.86328125# disk.nvme0c0n1.in‚ 9936.86328125 disk.sda.in‚0 disk.sda.out‚169.875 disk.rbd0.out‚0 disk.sda3.out‚169.875 disk.sda2.out‚0# proc.memory.rssMB‚ 2824.20703125) proc.memory.percent‚17.23759174346924 cpu‚25.580249641279625 network.sent‚ 5.118666e+06$ disk./.usageGB‚557.2870101928711 disk.sda3.in‚0 disk.drbd2165.out‚0& disk.nvme0n1.out‚55672.20166015625 disk.rbd1.out‚ 10.9765625( disk.nvme0c0n1.out‚55672.20166015625 disk.sda2.in‚0 disk.rbd1.in‚0 network.recv‚ 1.177962e+06 disk.dm-1.in‚0 disk.dm-1.out‚169.875 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd2165.in‚0! disk.nvme0n1.in‚ 9936.86328125( proc.memory.availableMB‚ 1501.3828125¯‚Î2·- _wandbruntime‚4556  _runtime‚4556Š ÏÿǸÁ & lr-AdamW/pg2‚0.0006676176336810873 trainer/global_step‚2749 % _timestamp‚1.7842943722007298e+09 _runtime‚4556.417995297 _step‚222 & lr-AdamW/pg1‚0.0006676176336810873%è‹…Ì¹Æ & lr-AdamW/pg2‚0.0006676176336810873 trainer/global_step‚2749 % _timestamp‚1.7842943722007298e+09 _runtime‚4556.417995297 _step‚222 & lr-AdamW/pg1‚0.0006676176336810873Þ-vD°2º- _wandbruntime‚4556  _runtime‚4556NEpM«»éO VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚790140471033344 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚787557372485632 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚787216831983616 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 trainer/global_step‚2749 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2581726984704 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚47139432 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚47129453 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2583072374272 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3332386868224 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool\ÆôD!.peak‚ 46096207872 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚14847096 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3332350234624 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚790140471033344 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚789794142772690 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚216900396195840 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚275 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚787557372485632 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7796409 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 _step‚223 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚270 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚790139099470336 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚789795512800980 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚787557398659072 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚23766552 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7050962 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7796139 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 _runtime‚4556.419178571 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚14847371 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚787557398659072 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2577310789074 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚23362907 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚213568045961216 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚790139099470336 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚216900800920576 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 % _timestamp‚1.7842943722018504e+09 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚23776525 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚23766552 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2581726984704 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2578655389396 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚787557398659072 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚23766552 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7050962 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7796139 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 _runtime‚4556.419178571 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚14847371 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚787557398659072 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2577310789074 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚23362907 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚213568045961216 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚790139099470336 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚216900800920576 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 % _timestamp‚1.7842943722018504e+09 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚23776525 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚23766552 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2581726984704 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2578655389396 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚787557372485632 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚787216831983616 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 trainer/global_step‚2749 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2581726984704 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚47139432 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚47129453 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2583072374272 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3332386868224 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚14847096 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3332350234624 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚790140471033344 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚789794142772690 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚216900396195840ß¼N¸2½- _wandbruntime‚4557  _runtime‚4557¬J? ù&¾óM C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚14852661 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚213645900460032 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚790419396806144 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚787835373215744 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚787835399389184 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404717568 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2578234255320 % _timestamp‚1.7842943735879688e+09 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚787494713761792 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚23371113 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7799156 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚47156130 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2583997416960 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3333566581760 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14852398 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚787835399389184 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚2749 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚23371107 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚790419396806144 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚47156130 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7798898 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3333529955328 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚790074318045514 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 _runtime‚4557.805343708 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚23785017 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚263 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚23371113 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚790072948017112 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚47146137 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚23775030 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚23371107 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7053505 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚213645532368896 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚47146137 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2582652020224 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2579578855754 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1€-3˜z¿øM UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚23371113 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚790072948017112 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚47146137 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚23775030 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚23371107 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7053505 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚213645532368896 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚47146137 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2582652020224 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2579578855754 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚14852661 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚213645900460032 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚790419396806144 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚787835373215744 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚787835399389184 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404717568 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2578234255320 % _timestamp‚1.7842943735879688e+09 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ZFDeviceStatsMonitor.on_train_batch_eP¦}Ç„nd/requested_bytes.large_pool.freed‚787494713761792 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚23371113 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7799156 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚47156130 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2583997416960 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3333566581760 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14852398 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚787835399389184 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚2749 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚23371107 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚790419396806144 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚47156130 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7798898 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3333529955328 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚790074318045514 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 _runtime‚4557.805343708 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚23785017 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚263 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352àuQß»2À- _wandbruntime‚4557  _runtime‚4557zAt Á„ 2loss/SRqqt1/assignment_loss‚0.5875111257216693 3loss/SRqqt2/assignment_loss‚0.49766532810455627 1loss/SRqqt2/detection_loss‚0.6491421836396449 0loss/FRt1/assignment_loss‚1.9015779062267482 0loss/FRt2/assignment_loss‚0.8104254481680107 3loss/FBt1/assignment_loss‚0.0015277456801176268 /loss/FRt2/detection_loss‚1.5678954951621162 /loss/FBt2/detection_loss‚0.6259016112890095 epoch‚6 $ _timestamp‚1.784294373674357e+09 _step‚225 2loss/FBt2/assignment_loss‚0.018331369808680723 /loss/FBt1/detection_loss‚0.6843336221362863 trainer/global_step‚2749 _runtime‚4557.891286069 /loss/FRt1/detection_loss‚0.5430167317483663 1loss/SRqqt1/detection_loss‚1.1480278921579676 %loss/total_loss‚9.0353564598431731*je‰ 0loss/FRt2/assignment_loss‚0.8104254481680107 3loss/FBt1/assignment_loss‚0.0015277456801176268 /loss/FRt2/detection_loss‚1.5678954951621162 /loss/FBt2/detection_loss‚0.6259016112890095 epoch‚6 $ _timestamp‚1.784294373674357e+09 _step‚225 2loss/FBt2/assignment_loss‚0.018331369808680723 /loss/FBt1/detection_loss‚0.6843336221362863 trainer/global_step‚2749 _runtime‚4557.891286069 /loss/FRt1/detection_loss‚0.5430167317483663 1loss/SRqqt1/detection_loss‚1.1480278921579676 %loss/total_loss‚9.035356459843173 2loss/SRqqt1/assignment_loss‚0.5875111257216693 3loss/SRqqt2/assignment_loss‚0.49766532810455627 1loss/SRqqt2/detection_loss‚0.6491421836396449 0loss/FRt1/assignment_loss‚1.9015779062267482ᆹƒƒ’Â(:‡ ç×èÒ»•÷ gpu.0.gpu‚6.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚331.878' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚66.3756 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0T0Í­úÄ‚(:ï ç×èÒÚÂäø disk.drbd2165.out‚0) proc.memory.availableMB‚ 1501.15234375) proc.memory.percent‚17.23759174346924 disk.rbd3.out‚0% disk.nvme0n1.out‚55683.6005859375 disk.rbd1.in‚0 disk.sda1.out‚0# memory_percent‚90.8376932144165 network.recv‚ 1.181656e+06 disk.rbd3.in‚0 disk.sda2.out‚0 disk.dm-1.in‚0" disk.dm-0.out‚55683.6005859375 disk.drbd2165.in‚0 cpu‚25.580865953336527 network.sent‚ 5.123669e+06 disk.sda.in‚0 disk.sda.out‚ 170.15234375 disk.dm-0.in‚ 9936.86328125 disk.sda3.out‚ 170.15234375 proc.cpu.threads‚17) disk./.usagePercent‚37.40037133995245 disk.sda2.in‚0 disk.rbd0.out‚0$ disk./.usageGB‚557.2925605773926# disk.nvme0c0n1.in‚ 9936.86328125 disk.rbd1.out‚ 10.9765625 disk.dm-1.out‚ 170.15234375 disk.rbd0.in‚0 disk.sda1.in‚0 disk.sda3.in‚0! disk.nvme0n1.in‚ 9936.86328125# proc.memory.rssMB‚ 2824.20703125' disk.nvme0c0n1.out‚55683.6005859375ý­w%”Å‚(:‰ ö×èÒ¦¿Þö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚273.707' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.7414 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0´ˆó,ùÆ‚(:î ö×èÒÈâ–ù disk.sda.in‚0 disk.dm-0.in‚ 10114.1171875# proc.memory.rssMB‚ 2824.20703125! disk.nvme0n1.in‚ 10114.1171875 disk.rbd1.in‚0 disk.rbd3.out‚0) proc.memory.percent‚17.23759174346924' disk.nvme0c0n1.out‚55692.5888671875 disk.sda1.out‚0# disk./.usageGB‚557.297679901123% disk.nvme0n1.out‚55692.5888671875 disk.rbd1.out‚ 11.0390625 disk.sda1.in‚0 cpu‚25.581195481263805# disk.nvme0c0n1.in‚ 10114.1171875 disk.dm-1.out‚ 170.30078125 disk.rbd0.out‚0 disk.sda3.in‚0) proc.memory.availableMB‚ 1501.33203125 proc.cpu.threads‚17 network.recv‚ 1.186415e+06" disk.dm-0.out‚55692.5888671875( disk./.usagePercent‚37.4007149020634 disk.sda.out‚ 170.30078125 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.drbd2165.in‚0 disk.rbd3.in‚0$ memory_percent‚90.83659648895264 network.sent‚ 5.186594e+06 disk.sda2.out‚0 disk.sda3.out‚ 170.30078125 disk.drbd2165.out‚0 disk.sda2.in‚02@*2žÇ‚(:“ …ØèÒ‰¨÷ö gpu.0.gpu‚80.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚320.425' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚64.08500000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ó=îWóÈ‚(:è …ØèÒå̦ù disk.sda3.in‚0 disk.sda3.out‚ 170.765625 disk.rbd3.in‚0 disk.sda2.out‚0$ memory_percent‚90.83662033081055 disk.sda.in‚0 disk.sda.out‚ 170.765625" disk.dm-0.out‚55704.8798828125 disk.drbd2165.in‚0 disk.rbd3.out‚0# disk.nvme0c0n1.in‚ 10114.1171875 disk.dm-1.in‚0# proc.memory.rssMB‚ 2824.20703125 network.sent‚ 5.190677e+06 disk.sda1.in‚0 disk.drbd2165.out‚0$ disk./.usageGB‚557.3022537231445 disk.rbd0.in‚0 disk.dm-0.in‚ 10114.1171875) proc.memory.percent‚17.23759174346924 cpu‚25.581742880947512 disk.rbd1.in‚0' disk.nvme0c0n1.out‚55704.8798828125 network.recv‚ 1.189081e+06 disk.sda1.out‚0 disk.sda2.in‚0 proc.cpu.threads‚17 disk.rbd0.out‚0! disk.nvme0n1.in‚ 10114.1171875% disk.nvme0n1.out‚55704.8798828125 disk.rbd1.out‚ 11.0390625' proc.memory.availableMB‚ 1501.328125) disk./.usagePercent‚37.40102185509695 disk.dm-1.out‚ 170.765625ö$žÉ‚(:“ ”ØèÒ‹Ç ÷ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚272.86' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚54.571999999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÃÐhôÊ‚(:é ”ØèÒÌ×áø( disk.nvme0c0n1.out‚55713.44775390625 disk.sda2.in‚0 disk.rbd1.out‚ 11.1015625$ memory_percent‚90.83664417266846 proc.cpu.threads‚17! disk.nvme0n1.in‚ 10114.1171875& disk.nvme0n1.out‚55713.44775390625 disk.rbd3.in‚0# proc.memory.rssMB‚ 2824.20703125 cpu‚25.582078457314132 disk.rbd0.out‚0# disk.dm-0.out‚55713.44775390625 disk.rbd1.in‚0) proc.memory.availableMB‚ 1501.32421875 disk.drbd2165.out‚0 disk.dm-0.in‚ 10114.1171875 disk.sda1.in‚0 network.sent‚ 5.19573e+06 network.recv‚ 1.192391e+06 disk.sda.out‚ 170.96875 disk.dm-1.out‚ 170.96875 disk.rbd0.in‚0 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.rbd3.out‚0) disk./.usagePercent‚37.40137232941132$ disk./.usageGB‚557.3074760437012 disk.sda.in‚0 disk.dm-1.in‚0# disk.nvme0c0n1.in‚ 10114.1171875 disk.sda2.out‚0) proc.memory.percent‚17.23759174346924 disk.sda3.in‚0 disk.sda3.out‚ 170.96875¡ZÐ̞˂(:“ £ØèÒ°³âö gpu.0.gpu‚87.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚330.528' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚66.10560000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¿¯†üïÌ‚(:ä £ØèÒµ¾‹ù proc.cpu.threads‚17 disk.dm-0.in‚ 10114.1171875 disk.dm-1.out‚ 171.26171875 disk.drbd2165.out‚0$ disk.nvme0c0n1.out‚ 55725.4921875 disk.sda2.in‚0* disk./.usagePercent‚37.401805238151724 disk.rbd1.in‚0 disk.sda.in‚0 disk.drbd2165.in‚0$ memory_percent‚90.83671569824219% proc.memory.availableMB‚ 1501.3125 cpu‚25.582455541252997 disk.rbd0.in‚0 disk.sda1.out‚0 disk.sda3.in‚0 disk.rbd3.out‚0 disk.sda2.out‚0) proc.memory.percent‚17.23759174346924 network.sent‚ 5.200836e+06 disk.rbd1.out‚ 11.1015625 disk.dm-1.in‚0 disk.rbd0.out‚0 disk.rbd3.in‚0 network.recv‚ 1.195775e+06$ disk./.usageGB‚557.3139266967773 disk.sda1.in‚0# disk.nvme0c0n1.in‚ 10114.1171875" disk.nvme0n1.out‚ 55725.4921875 disk.sda.out‚ 171.26171875 disk.dm-0.out‚ 55725.4921875 disk.sda3.out‚ 171.26171875! disk.nvme0n1.in‚ 10114.1171875# proc.memory.rssMB‚ 2824.20703125hŽ:5žÍ‚(:“ ²ØèÒ•«þö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚289.743' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚57.94859999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¤F¬ðñ΂(:æ ²ØèÒÇ›•ù disk.dm-1.out‚ 171.421875 disk.rbd1.out‚ 11.1640625) disk./.usagePercent‚37.40208454237158 disk.dm-1.in‚0" disk.nvme0n1.in‚10321.95703125 network.recv‚ 1.199086e+06 disk.sda.in‚0 disk.drbd2165.in‚0 disk.sda2.out‚0 cpu‚25.58272584139758 proc.cpu.threads‚17 network.sent‚ 5.205879e+06 disk.dm-0.in‚10321.95703125 disk.dm-0.out‚55735.44921875 disk.sda1.out‚0 disk.rbd3.in‚0 disk.sda3.in‚0 disk.sda3.out‚ 171.421875 disk.rbd3.out‚0$ disk./.usageGB‚557.3180885314941 disk.rbd0.out‚0$ disk.nvme0c0n1.in‚10321.95703125# disk.nvme0n1.out‚55735.44921875) proc.memory.availableMB‚ 1500.94921875) proc.memory.percent‚17.23759174346924 disk.sda.out‚ 171.421875 disk.drbd2165.out‚0% disk.nvme0c0n1.out‚55735.44921875 disk.sda2.in‚0 disk.rbd1.in‚0$ memory_percent‚90.83893299102783# proc.memory.rssMB‚ 2824.20703125 disk.rbd0.in‚0 disk.sda1.in‚0i[Ë¿2Ï- _wandbruntime‚4635  _runtime‚4635³™õÉÇÐÁ & lr-AdamW/pg2‚0.0006665879908945809 trainer/global_step‚2799 % _timestamp‚1.7842944513168008e+09 _runtime‚4635.534025189 _step‚226 & lr-AdamW/pg1‚0.0006665879908945809EShÌÑÆ & lr-AdamW/pg1‚0.0006665879908945809 & lr-AdamW/pg2‚0.0006665879908945809 trainer/global_step‚2799 % _timestamp‚1.7842944513168008e+09 _runtime‚4635.534025189 _step‚226â–s8Y2Ò- _wandbruntime‚4635  _runtime‚4635àùîÄï'ÓéO LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚220834798805504 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3392029832704 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚47965505 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚801457408991232 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚804086823469568 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚23773201 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚801457408991232 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚24202277 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7934416 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2623547902182 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 trainer/global_step‚2799 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚217442366345216 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚23773207 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚220834396177920 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚801110920892416 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2628042915328 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2629388304896 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚804085451906560 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2629388304896 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚24202277 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚24192304 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚24192304 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚248 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚804085451906560 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚23773207 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚803734468794598 _runtime‚4635.535249701 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚801110920892416 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2628042915328 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2629388304896 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚804085451906560 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2629388304896 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚24202277 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚24192304 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚24192304 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚248 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚804085451906560 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚23773207 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚803734468794598 _runtime‚4635.535249701 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3392029832704 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚47965505 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚801457408991232 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚804086823469568 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚23773201 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚801457408991232 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚24202277 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7934416 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2623547902182 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 trainer/global_step‚2799 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚217442366345216ã®òu2Õ- _wandbruntime‚4636  _runtime‚4636,êù&ÖóM J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚24200782 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2630313347584 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 trainer/global_step‚2799 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚217519852752896 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2624471368428 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚23781407 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚801735409721344 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚24210769 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚220913078701568 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚47982189 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 % _timestamp‚1.7842944527043393e+09 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚15118052 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚240 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7937397 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚24200782 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7180655 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚47992182 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15117812 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚4636.921714564 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7180650 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚801735435894784 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚24210769 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2630313347584 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚804365749242368 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 _step‚228 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚47982189 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚23781407 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚801388802670592 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3393225948672 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚804014644067422 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚23781413 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚220913481321984 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚47992182 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚47992182 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚47982189 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 % _timestamp‚1.7842944527043393e+09 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚15118052 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚240 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7937397 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚24200782 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7180655 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚47992182 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15117812 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚4636.921714564 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7180650 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚801735435894784 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚24210769 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2630313347584 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚804365749242368 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0äOQÞ2Ø- _wandbruntime‚4637  _runtime‚4637;Ö¬SÙ /loss/FRt1/assignment_loss‚1.953524782462339 2loss/SRqqt1/assignment_loss‚0.5271734728583727 2loss/SRqqt2/assignment_loss‚0.5577628337830003 epoch‚6 _step‚229 2loss/FBt1/assignment_loss‚0.013331911076806835 1loss/SRqqt1/detection_loss‚1.0090463550846007 1loss/SRqqt2/detection_loss‚0.6030512474043744 /loss/FRt2/assignment_loss‚0.805513495297143 2loss/FBt2/assignment_loss‚0.003712097065862106 /loss/FRt1/detection_loss‚0.5263695556174771 /loss/FBt1/detection_loss‚0.6699812796087591 /loss/FRt2/detection_loss‚1.5573020393306372 /loss/FBt2/detection_loss‚0.5015698421525602 %loss/total_loss‚8.728338911741933 trainer/global_step‚2799 % _timestamp‚1.7842944527913191e+09 _runtime‚4637.008253865^øÜB Ú† /loss/FRt1/assignment_loss‚1.953524782462339 2loss/SRqqt1/assignment_loss‚0.5271734728583727 2loss/SRqqt2/assignment_loss‚0.5577628337830003 epoch‚6 _step‚229 2loss/FBt1/assignment_loss‚0.013331911076806835 1loss/SRqqt1/detection_loss‚1.0090463550846007 1loss/SRqqt2/detection_loss‚0.6030512474043744 /loss/FRt2/assignment_loss‚0.805513495297143 2loss/FBt2/assignment_loss‚0.003712097065862106 /loss/FRt1/detection_loss‚0.5263695556174771 /loss/FBt1/detection_loss‚0.6699812796087591 /loss/FRt2/detection_loss‚1.5573020393306372 /loss/FBt2/detection_loss‚0.5015698421525602 %loss/total_loss‚8.728338911741933 trainer/global_step‚2799 % _timestamp‚1.7842944527913191e+09 _runtime‚4637.008253865åGxw§“Û‚(:ˆ ÁØèÒÓøüö gpu.0.gpu‚96.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚39.0 gpu.0.powerWatts‚330.145' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚66.029 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0–ºzÿûÜ‚(:ð ÁØèÒüšƒú' disk.nvme0c0n1.out‚55755.8935546875$ disk./.usageGB‚557.3300857543945 disk.dm-1.out‚ 173.53515625 disk.sda1.in‚0 disk.sda3.out‚ 173.53515625 disk.rbd3.in‚0 disk.sda2.out‚0 disk.rbd1.in‚0* proc.memory.percent‚17.438864707946777 network.sent‚ 5.211019e+06* disk./.usagePercent‚37.402889686066764 disk.sda.in‚0" disk.nvme0n1.in‚10321.95703125 cpu‚25.59004724573532 proc.cpu.threads‚17 disk.drbd2165.in‚0 disk.sda1.out‚0 disk.dm-1.in‚0 disk.rbd0.out‚0 disk.sda3.in‚0 disk.sda2.in‚0% proc.memory.availableMB‚ 1468.6875 disk.rbd0.in‚0 disk.drbd2165.out‚0# proc.memory.rssMB‚ 2857.18359375 disk.dm-0.in‚10321.95703125 disk.rbd1.out‚ 11.1640625$ memory_percent‚91.03584289550781 disk.sda.out‚ 173.53515625 disk.rbd3.out‚0$ disk.nvme0c0n1.in‚10321.95703125% disk.nvme0n1.out‚55755.8935546875 network.recv‚ 1.202502e+06" disk.dm-0.out‚55755.8935546875ç2Ý- _wandbruntime‚4664  _runtime‚4664ÝÓêÞ±{ 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.4069550037384033 particle/accuracy_1_of_2‚1 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6561165452003479 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.2398989051580429 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.5606131553649902 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.23540712893009186 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.1538461595773697 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7142857313156128 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.24986112117767334 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.1538461595773697 jet/accuracy_3_of_5‚1 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9994588494300842 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.06196476146578789 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.2838928699493408 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.28809648752212524 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9959074854850769 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.7455533146858215 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4775197207927704 6 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.699999988079071 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 _runtime‚4664.153255027 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 8"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.544463038444519 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9134171009063721 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8723124265670776 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.2348373681306839 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9333804845809937 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9847925901412964 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6847169399261475 7 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.1363653540611267 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.21701562404632568 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.25106167793273926 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6260399222373962 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.5086568593978882 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.18408605456352234 particle/accuracy_3_of_3‚1 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.24273188412189484 *jet/accuracy_1_of_1‚0.2929002046585083 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.2738322615623474 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.2760877013206482 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.24888718128204346 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7647770643234253 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.23738592863082886 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 *jet/accuracy_1_of_3‚0.9096570014953613 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.038905125111341476 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.5885416269302368 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7857142686843872 particle/accuracy_3_of_5‚1 8"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.550493597984314 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3054240643978119 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.6562433838844299 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 ,jet/accuracy_4_of_4‚0.013899499550461769 _step‚230 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9820979237556458 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.5597426295280457 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 particle/accuracy_1_of_3‚1 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7462803721427917 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5375000238418579 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 *jet/accuracy_2_of_4‚0.7410203814506531 /particle/accuracy_3_of_4‚0.9968253374099731 ) Purity/2FRt*SRqqt2FBt/FRt_purity‚0.75 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.12393977493047714 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.23738592863082886 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 jet/accuracy_2_of_5‚1 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.00099ìÝítÍ)98140158131719 particle/accuracy_1_of_1‚1 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9490740299224854 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5278207659721375 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9848985075950623 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.625 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.2546727657318115 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.2836368978023529 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.2801907956600189 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 jet/accuracy_1_of_5‚1 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7546296119689941 jet/accuracy_5_of_5‚0 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5521286725997925 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8243387341499329 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9230769276618958 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.6423898935317993 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.6536340713500977 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.49074071645736694 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4453125 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 &particle/f_score‚0.616946280002594 particle/accuracy_5_of_5‚0 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.10185185074806213 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.3201597332954407 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 *particle/specificity‚0.908310055732727 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8247501850128174 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9958513379096985 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.13931135833263397 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.6129003167152405 6 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.994129478931427 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.24056680500507355 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.13365322351455688 *jet/accuracy_2_of_2‚0.1141267865896225 7validation_average_jet_accuracy‚0.33639466762542725 )loss/val_total_loss‚8.739537871576378 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9973902106285095 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.5549519658088684 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.49156084656715393 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9992673993110657 particle/accuracy_1_of_4‚1 particle/accuracy_1_of_5‚1 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9939182996749878 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.3318908214569092 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03167566657066345 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.6385296583175659 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9928439855575562 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9949094653129578 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6539293527603149 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5899032354354858 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.5721417665481567 8"Purity/*FRt*SRqqt1FBt/event_purity‚0.660298228263855 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 7"Purity/*FRt1SRqqt*FBt/event_purity‚0.30547034740448 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.6170231103897095 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.17898452281951904 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.13811901211738586 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.03216676414012909 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9983028173446655 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 +jet/accuracy_3_of_3‚0.06072627753019333 particle/accuracy_2_of_4‚1 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6133136749267578 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8434996604919434 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4693404734134674 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7222222089767456 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.7455533146858215 )jet/accuracy_1_of_4‚0.956003725528717 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.04956776276230812 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.22928276658058167 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9907358884811401 (particle/accuracy‚0.8411652445793152 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.5561188459396362 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2512911260128021 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.13365322351455688 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5155993103981018 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 +particle/sensitivity‚0.5956609845161438 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.29242539405822754 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.23572687804698944 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8298762440681458 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.2668008506298065 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.10499999672174454 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5155993103981018 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.3256276547908783 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 *jet/accuracy_1_of_2‚0.7627582550048828 particle/accuracy_4_of_5‚1 % _timestamp‚1.7842944799354892e+09 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9988387227058411 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5218316316604614 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6013457179069519 )jet/accuracy_2_of_3‚0.465303510427475 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7546296119689941 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.2239583283662796 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.5640196800231934 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 particle/accuracy_2_of_5‚1 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.39814817905426025 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.12628254294395447 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.5981268286705017 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7195798754692078 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.5820218920707703 jet/accuracy_4_of_5‚0 trainer/global_step‚2799 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.5235716700553894 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.32499998807907104 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.5840862989425659 /particle/accuracy_4_of_4‚0.9287430047988892 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2477862536907196 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.2761039733886719 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9879558086395264 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8331444263458252 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.20845063030719757 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.31297045946121216 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9953094720840454 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 *jet/accuracy_3_of_4‚0.0632137805223465 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.02568456344306469 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9697970151901245 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5575304627418518 6 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.823167085647583 particle/accuracy_2_of_2‚1 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 particle/accuracy_2_of_3‚1 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.10185185074806213 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9992673993110657 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 epoch‚6 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6215153932571411 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8331444263458252 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 7 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.1369437426328659 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625Ÿ¼ô.¼=߶{ ("Purity/*FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5155993103981018 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 +particle/sensitivity‚0.5956609845161438 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.29242539405822754 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.23572687804698944 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8298762440681458 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.2668008506298065 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.10499999672174454 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5155993103981018 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.3256276547908783 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 *jet/accuracy_1_of_2‚0.7627582550048828 particle/accuracy_4_of_5‚1 % _timestamp‚1.7842944799354892e+09 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9988387227058411 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5218316316604614 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6013457179069519 )jet/accuracy_2_of_3‚0.465303510427475 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7546296119689941 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.2239583283662796 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.5640196800231934 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 particle/accuracy_2_of_5‚1 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.39814817905426025 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.12628254294395447 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.5981268286705017 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7195798754692078 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.5820218920707703 jet/accuracy_4_of_5‚0 trainer/global_step‚2799 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.5235716700553894 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.32499998807907104 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.5840862989425659 /particle/accuracy_4_of_4‚0.9287430047988892 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2477862536907196 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.2761039733886719 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9879558086395264 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8331444263458252 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.20845063030719757 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.31297045946121216 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9953094720840454 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 *jet/accuracy_3_of_4‚0.0632137805223465 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.02568456344306469 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9697970151901245 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5575304627418518 6 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.823167085647583 particle/accuracy_2_of_2‚1 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 particle/accuracy_2_of_3‚1 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.10185185074806213 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9992673993110657 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 epoch‚6 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6215153932571411 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8331444263458252 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 7 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.1369437426328659 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.4069550037384033 particle/accuracy_1_of_2‚1 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6561165452003479 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.2398989051580429 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.5606131553649902 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.23540712893009186 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.1538461595773697 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7142857313156128 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.24986112117767334 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.1538461595773697 jet/accuracy_3_of_5‚1 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9994588494300842 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.06196476146578789 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.2838928699493408 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.28809648752212524 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9959074854850769 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.7455533146858215 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4775197207927704 6 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.699999988079071 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 _runtime‚4664.153255027 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 8"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.544463038444519 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9134171009063721 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8723124265670776 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.2348373681306839 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9333804845809937 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9847925901412964 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6847169399261475 7 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.1363653540611267 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.21701562404632568 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.25106167793273926 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6260399222373962 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.5086568593978882 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.18408605456352234 particle/accuracy_3_of_3‚1 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.24273188412189484 *jet/accuracy_1_of_1‚0.2929002046585083 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.2738322615623474 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.2760877013206482 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.24888718128204346 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7647770643234253 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.23738592863082886 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 *jet/accuracy_1_of_3‚0.9096570014953613 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.038905125111341476 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.5885416269302368 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7857142686843872 particle/accuracy_3_of_5‚1 8"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.550493597984314 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3054240643978119 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.6562433838844299 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 ,jet/accuracy_4_of_4‚0.013899499550461769 _step‚230 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9820979237556458 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.5597426295280457 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 particle/accuracy_1_of_3‚1 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7462803721427917 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5375000238418579 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 *jet/accuracy_2_of_4‚0.7410203814506531 /particle/accuracy_3_of_4‚0.9968253374099731 ) Purity/2FRt*SRqqt2FBt/FRt_purity‚0.75 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.12393977493047714 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.23738592863082886 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 jet/accuracy_2_of_5‚1 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 particle/accuracy_1_of_1‚1 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9490740299224854 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5278207659721375 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9848985075950623 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.625 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.2546727657318115 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.2836368978023529 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.2801907956600189 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 jet/accuracy_1_of_5‚1 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7546296119689941 jet/accuracy_5_of_5‚0 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5521286725997925 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8243387341499329 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9230769276618958 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.6423898935317993 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.6536340713500977 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.49074071645736694 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4453125 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 &particle/f_score‚0.616946280002594 particle/accuracy_5_of_5‚0 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.10185185074806213 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.3201597332954407 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 *particle/specificity‚0.908310055732727 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8247501850128174 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9958513379096985 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.13931135833263397 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.6129003167152405 6 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.994129478931427 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.24056680500507355 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.13365322351455688 *jet/accuracy_2_of_2‚0.1141267865896225 7validation_average_jet_accuracy‚0.33639466762542725 )loss/val_total_loss‚8.739537871576378 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9973902106285095 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.5549519658088684 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.49156084656715393 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9992673993110657 particle/accuracy_1_of_4‚1 particle/accuracy_1_of_5‚1 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9939182996749878 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.3318908214569092 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03167566657066345 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.6385296583175659 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9928439855575562 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9949094653129578 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6539293527603149 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5899032354354858 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.5721417665481567 8"Purity/*FRt*SRqqt1FBt/event_purity‚0.660298228263855 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 7"Purity/*FRt1SRqqt*FBt/event_purity‚0.30547034740448 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.6170231103897095 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.17898452281951904 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.13811901211738586 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.03216676414012909 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9983028173446655 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 +jet/accuracy_3_of_3‚0.06072627753019333 particle/accuracy_2_of_4‚1 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6133136749267578 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8434996604919434 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4693404734134674 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7222222089767456 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.7455533146858215 )jet/accuracy_1_of_4‚0.956003725528717 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.04956776276230812 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.22928276658058167 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9907358884811401 (particle/accuracy‚0.8411652445793152 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.5561188459396362 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2512911260128021 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.13365322351455688 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129æÅH“à‚(:ˆ ÐØèÒêäîö gpu.0.gpu‚19.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚33.0 gpu.0.powerWatts‚108.527' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.7054 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ì¿Çòþá‚(:ó ÐØèÒÌËŸù$ disk.nvme0c0n1.in‚10321.95703125) disk./.usagePercent‚37.40312495699074$ disk./.usageGB‚557.3335914611816 disk.rbd0.in‚0# disk.dm-0.out‚55769.83740234375 disk.sda1.in‚0 disk.sda3.out‚ 173.90234375( disk.nvme0c0n1.out‚55769.83740234375 disk.dm-0.in‚10321.95703125 disk.sda3.in‚0 disk.drbd2165.out‚0) proc.memory.availableMB‚ 1091.05859375 disk.dm-1.in‚0 disk.drbd2165.in‚0" disk.nvme0n1.in‚10321.95703125 disk.sda2.out‚0 disk.rbd3.in‚0& disk.nvme0n1.out‚55769.83740234375 disk.rbd1.out‚ 11.2265625 disk.sda.out‚ 173.90234375 disk.dm-1.out‚ 173.90234375! proc.memory.rssMB‚ 3234.765625 cpu‚25.597126675451655 proc.cpu.threads‚17 disk.rbd3.out‚0 network.sent‚ 5.273701e+06 network.recv‚ 1.20661e+06 disk.rbd1.in‚0 disk.rbd0.out‚0$ memory_percent‚93.34070682525635) proc.memory.percent‚19.74344253540039 disk.sda2.in‚0 disk.sda1.out‚0 disk.sda.in‚0ÎqN4”â‚(:‰ ßØèÒ¥ÞÔø gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚41.0 gpu.0.powerWatts‚271.718' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.3436 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ñæ³÷ã‚(:ì ßØèÒ¾ïèù$ disk.nvme0n1.out‚57968.705078125 disk.sda2.in‚0 disk.rbd1.out‚ 11.2265625 disk.sda.in‚0 disk.drbd2165.in‚0 disk.dm-1.in‚0 disk.rbd0.out‚0 disk.sda1.in‚0 disk.drbd2165.out‚0 disk.dm-1.out‚ 174.64453125$ memory_percent‚84.62934494018555 proc.cpu.threads‚17 network.sent‚ 5.331052e+06& disk.nvme0c0n1.out‚57968.705078125' proc.memory.availableMB‚ 2518.328125 network.recv‚ 1.211076e+06$ disk.nvme0c0n1.in‚10321.95703125) proc.memory.percent‚17.25139617919922 cpu‚25.592026371501717* disk./.usagePercent‚37.403314914581145" disk.nvme0n1.in‚10321.95703125 disk.sda.out‚ 174.64453125 disk.dm-0.in‚10321.95703125! disk.dm-0.out‚57968.705078125 disk.sda3.in‚0 disk.rbd3.out‚0 disk.sda2.out‚0 disk.rbd0.in‚0 disk.sda3.out‚ 174.64453125 disk.rbd1.in‚0 proc.memory.rssMB‚ 2826.46875$ disk./.usageGB‚557.3364219665527 disk.sda1.out‚0 disk.rbd3.in‚0¾E.ö•ä‚(:Š îØèÒÆêö gpu.0.gpu‚100.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚325.017' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚65.0034 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0V½õå‚(:ê îØèÒ†Úùø disk.sda.in‚0 network.sent‚ 5.335526e+06 disk.rbd3.in‚0 disk.sda1.out‚0 disk.sda3.in‚0 disk.sda3.out‚ 176.9140625( disk.nvme0c0n1.out‚57981.43408203125 disk.nvme0n1.in‚ 10499.515625$ memory_percent‚84.63051319122314) proc.memory.availableMB‚ 2518.13671875 disk.rbd0.in‚0 disk.rbd0.out‚0 network.recv‚ 1.214564e+06$ disk./.usageGB‚557.3407554626465 disk.sda2.out‚0 proc.memory.rssMB‚ 2826.46875 cpu‚25.591372820770857 proc.cpu.threads‚17 disk.dm-0.in‚ 10499.515625# disk.dm-0.out‚57981.43408203125 disk.rbd3.out‚0 disk.rbd1.in‚0 disk.dm-1.in‚0) proc.memory.percent‚17.25139617919922 disk.drbd2165.out‚0" disk.nvme0c0n1.in‚ 10499.515625& disk.nvme0n1.out‚57981.43408203125 disk.sda2.in‚0 disk.rbd1.out‚ 11.2890625 disk.sda.out‚ 176.9140625 disk.dm-1.out‚ 176.9140625) disk./.usagePercent‚37.40360573914004 disk.sda1.in‚0 disk.drbd2165.in‚0Äû”æ‚(:‰ ýØèÒ¬Âìö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚266.957' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.3914 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ǫ‰Üç‚(:Ñ ýØèÒÍèœù disk.drbd2165.out‚0 disk.rbd3.out‚0 disk.rbd1.in‚0 network.sent‚ 5.341056e+06 disk.dm-1.out‚178 disk.rbd0.out‚0( disk.nvme0c0n1.out‚57994.71826171875 disk.dm-1.in‚0 disk.sda3.in‚0) proc.memory.availableMB‚ 2518.13671875 network.recv‚ 1.217997e+06$ disk./.usageGB‚557.3455543518066 disk.sda2.in‚0) proc.memory.percent‚17.25139617919922) disk./.usagePercent‚37.40392779661811# disk.dm-0.out‚57994.71826171875& disk.nvme0n1.out‚57994.71826171875 disk.sda2.out‚0 disk.rbd3.in‚0 disk.nvme0n1.in‚ 10511.328125 disk.sda1.out‚0" disk.nvme0c0n1.in‚ 10511.328125 disk.sda.out‚178 proc.memory.rssMB‚ 2826.46875 disk.rbd0.in‚0 disk.dm-0.in‚ 10511.328125 disk.sda1.in‚0 disk.rbd1.out‚ 11.2890625 disk.sda.in‚0$ memory_percent‚84.63051319122314 cpu‚25.59065748299468 proc.cpu.threads‚17 disk.sda3.out‚178 disk.drbd2165.in‚0ú]?•è‚(:Š ŒÙèÒ’²÷ gpu.0.gpu‚100.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚259.878' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚51.9756 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0™Oõ;xé‚(:ë ŒÙèÒÍäÈø) disk./.usagePercent‚37.40427571085714" disk.nvme0c0n1.in‚ 10511.328125 disk.sda2.ouMZ t~t‚0# disk.dm-0.out‚58003.69287109375 disk.sda1.out‚0 disk.sda3.out‚ 178.36328125 proc.memory.rssMB‚ 2826.46875 proc.cpu.threads‚17 network.sent‚ 5.34609e+06 disk.sda3.in‚0( disk.nvme0c0n1.out‚58003.69287109375& disk.nvme0n1.out‚58003.69287109375 network.recv‚ 1.22132e+06 disk.dm-1.in‚0 disk.dm-0.in‚ 10511.328125 disk.drbd2165.out‚0 disk.rbd3.out‚0) proc.memory.percent‚17.25139617919922 disk.sda.in‚0 disk.dm-1.out‚ 178.36328125 disk.rbd0.in‚0 disk.nvme0n1.in‚ 10511.328125 disk.sda2.in‚0 disk.rbd1.in‚0) proc.memory.availableMB‚ 2518.13671875 disk.rbd0.out‚0 disk.drbd2165.in‚0 disk.rbd3.in‚0 disk.rbd1.out‚ 11.4140625 disk.sda1.in‚0$ disk./.usageGB‚557.3507385253906 disk.sda.out‚ 178.36328125$ memory_percent‚84.63051319122314 cpu‚25.590062682345845ßæöõ”ê‚(:‰ ›ÙèÒÆœÚö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚286.551' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚57.3102 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Q>ØBöë‚(:ë ›ÙèÒåÝþø$ disk./.usageGB‚557.3566246032715 disk.dm-1.out‚ 178.60546875 disk.rbd0.out‚0 disk.drbd2165.in‚0" disk.nvme0c0n1.in‚ 10511.328125 disk.sda3.in‚0$ memory_percent‚84.63263511657715 proc.memory.rssMB‚ 2826.46875 disk.dm-1.in‚0 disk.dm-0.in‚ 10511.328125 disk.rbd3.in‚0& disk.nvme0n1.out‚58014.88037109375( proc.memory.availableMB‚ 2517.7890625 network.sent‚ 5.351163e+06 network.recv‚ 1.22496e+06 disk.sda.out‚ 178.60546875# disk.dm-0.out‚58014.88037109375 disk.sda1.out‚0 disk.sda3.out‚ 178.60546875 disk.sda2.out‚0) proc.memory.percent‚17.25139617919922 disk.rbd0.in‚0 disk.rbd1.in‚0 cpu‚25.589354450117305 proc.cpu.threads‚17) disk./.usagePercent‚37.40467073048247 disk.sda.in‚0 disk.sda1.in‚0 disk.rbd3.out‚0 disk.sda2.in‚0 disk.drbd2165.out‚0( disk.nvme0c0n1.out‚58014.88037109375 disk.nvme0n1.in‚ 10511.328125 disk.rbd1.out‚ 11.4140625Þöðç2ì- _wandbruntime‚4748  _runtime‚4748°ý•ÓÇíÁ _step‚231 & lr-AdamW/pg1‚0.0006655377524736606 & lr-AdamW/pg2‚0.0006655377524736606 trainer/global_step‚2849 % _timestamp‚1.7842945639473586e+09 _runtime‚4748.164511797ûßʾÌîÆ % _timestamp‚1.7842945639473586e+09 _runtime‚4748.164511797 _step‚231 & lr-AdamW/pg1‚0.0006655377524736606 & lr-AdamW/pg2‚0.0006655377524736606 trainer/global_step‚2849çªi­2ï- _wandbruntime‚4748  _runtime‚4748pµ¯‡ï'ðéO PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚24315513 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚24687016 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚819185426931712 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7314180 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚821865628391936 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚821866999954944 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2681546849792 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚819185426931712 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚32439296 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 400530432 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚821506532880556 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚818830944161792 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚237 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚818830918733824 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚15420981 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚24677043 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2680201460224 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2676958747054 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8106801 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚221365167067136 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7842945639484386e+09 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚48992550 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚821866999954944 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚24687016 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚224824066195968 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3459267219968%ǹô'ñîO O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚821506532880556 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚818830944161792 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚237 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚818830918733824 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚15420981 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚24677043 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2680201460224 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2676958747054 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8106801 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚221365167067136 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7842945639484386e+09 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚48992550 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚821866999954944 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚24687016 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚224824066195968 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3459267219968 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚24315513 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚24687016 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚819185426931712 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7314180 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚821865628391936 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚821866999954944 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2681546849792 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚819185426931712 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚32439296 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 400530432è…¢úY2ò- _wandbruntime‚4749  _runtime‚4749ZˆG§Ÿ$óñM H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚233 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7316718 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚821786708153380 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚819108800512000 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚822145925727744 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚24685521 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2677882213412 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚821785338124978 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚49009234 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚819463427661824 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚819463453835264 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2681126495744 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2682471892480 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚15426292 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8109341 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3460479867392 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2681126495744 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚24323719 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚24323713 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15426059 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚49019227 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2676537612978 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚24685521 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚224903165774336 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚819108825939968 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚819463427661824 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 _step‚233 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚224902765251072 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚49019227 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚24695508 _runtime‚ 4749.55126793 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚822144554157568 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚24323713 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚819463453835264 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3460512299520 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7316723 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚24695508 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚221442653474816 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚221442285383680 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 $ _timestamp‚1.784294565333859e+09 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚32432128 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚49009234 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚24323719 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 UIDeviceStatsMonitor.on_train_bñ}¢Xatch_end/inactive_split.small_pool.allocated‚8109569 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚822145925727744 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚228 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175õj?ü&ôöM LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚821786708153380 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚819108800512000 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚822145925727744 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚24685521 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2677882213412 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚821785338124978 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚49009234 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚819463427661824 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚819463453835264 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2681126495744 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2682471892480 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚15426292 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8109341 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3460479867392 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2681126495744 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚24323719 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚24323713 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15426059 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚49019227 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2676537612978 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚24685521 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚224903165774336 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚819108825939968 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚819463427661824 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 _step‚233 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚224902765251072 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚49019227 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚24695508 _runtime‚ 4749.55126793 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚822144554157568 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚24323713 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚819463453835264 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3460512299520 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7316723 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚24695508 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚221442653474816 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚221442285383680 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 $ _timestamp‚1.784294565333859e+09 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚32432128 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚49009234 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚24323719 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8109569 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚822145925727744 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚228 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚233 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7316718é~cC2õ- _wandbruntime‚4749  _runtime‚4749k„¾,ö€ _runtime‚4749.637931543 epoch‚7 0loss/FRt1/assignment_loss‚1.8962077410513414 0loss/FRt2/assignment_loss‚0.8257072914509752 2loss/SRqqt1/assignment_loss‚0.6810664673766982 2loss/FBt2/assignment_loss‚0.012558240322306726 /loss/FRt1/detection_loss‚0.5649927906039965 .loss/FRt2/detection_loss‚1.424265550339923 1loss/SRqqt1/detection_loss‚1.1086272542917375 2loss/SRqqt2/assignment_loss‚0.5314286962767817 1loss/SRqqt2/detection_loss‚0.5904223879450927 /loss/FBt1/detection_loss‚0.4651869035951677 /loss/FBt2/detection_loss‚0.8461847920677558 trainer/global_step‚2849 $ _timestamp‚1.784294565420942e+09 _step‚234 %loss/total_loss‚9.136778826569534 1loss/FBt1/assignment_loss‚0.19013071124775704Ÿ¸¯Œ ÷… trainer/global_step‚2849 $ _timestamp‚1.784294565420942e+09 _step‚234 %loss/total_loss‚9.136778826569534 1loss/FBt1/assignment_loss‚0.19013071124775704 _runtime‚4749.637931543 epoch‚7 0loss/FRt1/assignment_loss‚1.8962077410513414 0loss/FRt2/assignment_loss‚0.8257072914509752 2loss/SRqqt1/assignment_loss‚0.6810664673766982 2loss/FBt2/assignment_loss‚0.012558240322306726 /loss/FRt1/detection_loss‚0.5649927906039965 .loss/FRt2/detection_loss‚1.424265550339923 1loss/SRqqt1/detection_loss‚1.1086272542917375 2loss/SRqqt2/assignment_loss‚0.5314286962767817 1loss/SRqqt2/detection_loss‚0.5904223879450927 /loss/FBt1/detection_loss‚0.4651869035951677 /loss/FBt2/detection_loss‚0.8461847920677558ê9·âÕŸø‚(:” ªÙèÒ‰ÂÉö gpu.0.gpu‚90.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚285.002' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚57.000400000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Œm õù‚(:ê ªÙèÒ–›½ø disk.dm-0.in‚ 10511.328125$ disk./.usageGB‚557.3616256713867 disk.drbd2165.in‚0 disk.sda2.in‚0 disk.rbd0.out‚0 disk.sda1.in‚0' proc.memory.availableMB‚ 2517.296875) proc.memory.percent‚17.25139617919922) disk./.usagePercent‚37.40500635635986 disk.sda3.in‚0 disk.sda3.out‚ 178.81640625 disk.rbd0.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 178.81640625 proc.memory.rssMB‚ 2826.46875 proc.cpu.threads‚17 disk.sda1.out‚0 network.sent‚ 5.356156e+06& disk.nvme0n1.out‚58024.77490234375 disk.sda2.out‚0# disk.dm-0.out‚58024.77490234375$ memory_percent‚84.63563919067383 cpu‚25.58876565527824 network.recv‚ 1.228059e+06 disk.drbd2165.out‚0( disk.nvme0c0n1.out‚58024.77490234375 disk.sda.out‚ 178.81640625 disk.rbd3.out‚0 disk.rbd3.in‚0 disk.nvme0n1.in‚ 10511.328125 disk.sda.in‚0" disk.nvme0c0n1.in‚ 10511.328125 disk.rbd1.in‚0 disk.rbd1.out‚ 11.47656259¹j Ÿú‚(:” ¹ÙèÒ¯‡àö gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚279.876' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚55.975199999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‹È¯ì÷û‚(:ì ¹ÙèÒ¨µ‚ù disk.sda3.in‚0 proc.memory.rssMB‚ 2826.46875 disk.sda.in‚0 disk.sda.out‚ 180.84765625! disk.dm-0.out‚58048.037109375 disk.drbd2165.out‚0 proc.cpu.threads‚17 disk.rbd1.in‚0 disk.dm-1.in‚0 disk.sda1.out‚0 disk.rbd3.in‚0 disk.sda2.in‚0 network.sent‚ 5.419162e+06 disk.rbd3.out‚0( proc.memory.availableMB‚ 2517.1796875) disk./.usagePercent‚37.40538934363109 disk.rbd0.in‚0 disk.dm-0.in‚10667.08984375 disk.sda1.in‚0& disk.nvme0c0n1.out‚58048.037109375$ memory_percent‚84.63635444641113 network.recv‚ 1.233047e+06 disk.sda3.out‚ 180.84765625$ disk.nvme0c0n1.in‚10667.08984375 disk.rbd1.out‚ 11.4765625 disk.dm-1.out‚ 180.84765625 disk.rbd0.out‚0" disk.nvme0n1.in‚10667.08984375) proc.memory.percent‚17.25139617919922 cpu‚25.588064587577932$ disk./.usageGB‚557.3673324584961 disk.drbd2165.in‚0$ disk.nvme0n1.out‚58048.037109375 disk.sda2.out‚0—Mt”ü‚(:‰ ÈÙèÒ×ü÷ö gpu.0.gpu‚90.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚258.358' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚51.6716 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0èÖ |ïý‚(:ä ÈÙèÒ¢²ù disk.rbd3.out‚0# disk.nvme0n1.out‚58055.56640625 network.recv‚ 1.23632e+06) disk./.usagePercent‚37.40569604065711 disk.rbd1.in‚0 disk.sda1.in‚0 disk.sda3.in‚0 proc.memory.rssMB‚ 2826.46875 disk.dm-1.in‚0 disk.rbd0.in‚0$ disk.nvme0c0n1.in‚10667.08984375 cpu‚25.587525737432074 disk.rbd3.in‚0$ proc.memory.availableMB‚2517.125 disk.sda2.in‚0 disk.rbd1.out‚ 11.5390625 disk.sda.in‚0 disk.sda.out‚ 182.06640625 disk.dm-1.out‚ 182.06640625% disk.nvme0c0n1.out‚58055.56640625 proc.cpu.threads‚17$ disk./.usageGB‚557.3719024658203 network.sent‚ 5.424165e+06 disk.dm-0.in‚10667.08984375 disk.dm-0.out‚58055.56640625" disk.nvme0n1.in‚10667.08984375$ memory_percent‚84.63668823242188) proc.memory.percent‚17.25139617919922 disk.sda1.out‚0 disk.sda2.out‚0 disk.rbd0.out‚0 disk.sda3.out‚ 182.06640625 disk.drbd2165.in‚0 disk.drbd2165.out‚0%€  žþ‚(:“ ×ÙèÒ•½œ÷ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚40.0 gpu.0.powerWatts‚272.542' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚54.50839999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚04Ìôÿ‚(:é ×ÙèÒö¬‚ù disk.rbd1.out‚ 11.5390625 network.sent‚ 5.42916e+06 disk.rbd0.out‚0 disk.sda3.out‚ 183.8203125 cpu‚25.586843063498478 disk.rbd1.in‚0* disk./.usagePercent‚37.405961520470115 disk.sda.out‚ 183.8203125 disk.sda.in‚0 disk.dm-0.in‚10667.08984375 disk.dm-0.out‚58073.37109375 disk.dm-1.out‚ 183.8203125$ disk.nvme0c0n1.in‚10667.08984375# memory_percent‚84.6367597579956) proc.memory.availableMB‚ 2517.11328125 disk.rbd0.in‚0 disk.sda3.in‚0 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.rbd3.out‚0" disk.nvme0n1.in‚10667.08984375 network.recv‚ 1.239826e+06 disk.dm-1.in‚0 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda2.out‚0 proc.cpu.threads‚17 disk.sda1.in‚0 disk.rbd3.in‚0$ disk./.usageGB‚557.3758583068848% disk.nvme0c0n1.out‚58073.37109375# disk.nvme0n1.out‚58073.37109375# proc.memory.rssMB‚ 2826.47265625) proc.memory.percent‚17.25142002105713ÖÉ*Ÿ€‚(:” æÙèÒšçö gpu.0.gpu‚90.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚267.174' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚53.434799999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0œÑp?ÿ‚(:ô æÙèÒ­¥÷ø disk.dm-1.out‚ 185.12109375 disk.sda3.out‚ 185.12109375 disk.sda.in‚0 disk.dm-1.in‚0 network.recv‚ 1.242914e+06 disk.drbd2165.in‚0 disk.sda2.in‚0 network.sent‚ 5.434199e+06 disk.drbd2165.out‚0 disk.sda.out‚ 185.12109375$ memory_percent‚84.63692665100098 cpu‚25.58626472518988 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.rbd1.out‚ 11.6015625) disk./.usagePercent‚37.40648940800572 disk.dm-0.in‚10667.08984375 disk.sda1.out‚0 disk.sda3.in‚0$ disk.nvme0c0n1.in‚10667.08984375 disk.sda2.out‚0 disk.sda1.in‚0 disk.rbd3.in‚0( disk.nvme0c0n1.out‚58083.74365234375& disk.nvme0n1.out‚58083.74365234375 disk.rbd1.in‚0$ disk./.usageGB‚557.3837242126465# disk.dm-0.out‚58083.74365234375 disk.rbd3.out‚0( proc.memory.availableMB‚ 2517.0859375" disk.nvme0n1.in‚10667.08984375# proc.memory.rssMB‚ 2826.47265625) proc.memory.percent‚17.25142002105713 proc.cpu.threads‚17%Ç‚2‚- _wandbruntime‚4826  _runtime‚4826Ë[̗ƃÀ _runtime‚ 4826.86022968 _step‚235 & lr-AdamW/pg1‚0.0006644669858731779 & lr-AdamW/pg2‚0.0006644669858731779 trainer/global_step‚2899 % _timestamp‚1.7842946426430433e+09/pË„Å _step‚235 & lr-AdamW/pg1‚0.0006644669858731779 & lr-AdamW/pg2‚0.0006644669858731779 trainer/global_step‚2899 % _timestamp‚1.7842946426430433e+09 _runtime‚ 4826.86022968ëÞ}˜2…- _wandbruntime‚4826  _runtime‚4826 /6ï'†éO VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404724736 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚15686978 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚228758507589120 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2726517390848 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚835813352391168 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚835811980828160 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚833085489610752 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2721851259840 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8245934 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8245653 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚225239487451136 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3518983504384 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚24725807 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚835448228930754 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3519020137984 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2726517390848 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚835813352391168 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚835811980828160 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚833085489610752 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2721851259840 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8245934 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8245653 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚225239487451136 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3518983504384 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚24725807 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚835448228930754 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3519020137984 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 IDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15692214 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚228836769290752 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 _step‚237 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2722774726086 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚833002914848768 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚25111273 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚24734019 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8248346 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 trainer/global_step‚2899 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚833363464167424 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2727442426368 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚49855279 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚24734013 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚49855279 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2724119326520 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404717568 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3520200149504 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚833002889420800 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7443873 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚25121260 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚833363490340864 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2728787823104 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚25121260 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚836090906593792 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1þþþ&ŠøM E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚25121260 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚836090906593792 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚228837174008320 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3520163523072 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7443868 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚225316973858816 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚835725664146886 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚49845286 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚833363490340864 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2727442426368 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 % _timestamp‚1.7842946440304308e+09 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚24734019 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15692214 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚228836769290752 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 _step‚237 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2722774726086 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚833002914848768 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚25111273 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚24734019 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8248346 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 trainer/global_step‚2899 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚833363464167424 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2727442426368 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚49855279 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚24734013 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚49855279 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2724119326520 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404717568 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3520200149504 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚833002889420800 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7443873 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚25121260 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚833363490340864 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2728787823104 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154íHãèR2‹- _wandbruntime‚4828  _runtime‚4828®Çë Œ€ .loss/FRt2/detection_loss‚1.625148044576346 _step‚238 3loss/SRqqt2/assignment_loss‚0.48809020994688784 3loss/FBt1/assignment_loss‚0.0009566921531533663 -loss/FBt1/detection_loss‚0.64700299840317 %loss/total_loss‚8.494542342015231 /loss/FRt1/detection_loss‚0.5045936283719958 1loss/SRqqt1/detection_loss‚0.9162040750851628 /loss/FBt2/detection_loss‚0.5028799613732297 epoch‚7 0loss/FRt1/assignment_loss‚1.8102929233847296 0loss/SRqqt1/assignment_loss‚0.56415260742024 2loss/FBt2/assignment_loss‚0.007359138290192589 1loss/SRqqt2/detection_loss‚0.6750603939877304 trainer/global_step‚2899 % _timestamp‚1.7842946441176598e+09 _runtime‚4828.334583551 0loss/FRt2/assignment_loss‚0.7528016690223936:oü … epoch‚7 0loss/FRt1/assignment_loss‚1.8102929233847296 0loss/SRqqt1/assignment_loss‚0.56415260742024 2loss/FBt2/assignment_loss‚0.007359138290192589 1loss/SRqqt2/detection_loss‚0.6750603939877304 trainer/global_step‚2899 % _timestamp‚1.7842946441176598e+09 _runtime‚4828.334583551 0loss/FRt2/assignment_loss‚0.7528016690223936 .loss/FRt2/detection_loss‚1.625148044576346 _step‚238 3loss/SRqqt2/assignment_loss‚0.48809020994688784 3loss/FBt1/assignment_loss‚0.0009566921531533663 -loss/FBt1/detection_loss‚0.64700299840317 %loss/total_loss‚8.494542342015231 /loss/FRt1/detection_loss‚0.5045936283719958 1loss/SRqqt1/detection_loss‚0.9162040750851628 /loss/FBt2/detection_loss‚0.5028799613732297îzŽ$Y”Ž‚(:‰ õÙèÒ™ˆ÷ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚278.522' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.7044 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0z]S÷‚(:ì õÙèÒö­·ù$ disk.nvme0c0n1.in‚10853.26953125 disk.rbd0.out‚0$ memory_percent‚84.63788032531738 disk.drbd2165.in‚0# proc.memory.rssMB‚ 2826.47265625) proc.memory.percent‚17.25142002105713" disk.nvme0n1.in‚10853.26953125 disk.rbd1.in‚0 disk.dm-1.in‚0 proc.cpu.threads‚17 disk.sda3.in‚0 disk.sda3.out‚ 188.08203125 disk.rbd3.in‚0 disk.sda.in‚0 disk.sda1.out‚0# disk.nvme0n1.out‚58114.41796875 disk.sda1.in‚0 network.recv‚ 1.246261e+06 disk.sda2.out‚0 disk.rbd1.out‚ 11.6015625 disk.dm-1.out‚ 188.08203125 disk.rbd0.in‚0 disk.dm-0.in‚10853.26953125( proc.memory.availableMB‚ 2516.9296875 network.sent‚ 5.439201e+06 disk.drbd2165.out‚0% disk.nvme0c0n1.out‚58114.41796875 disk.sda.out‚ 188.08203125) disk./.usagePercent‚37.40677434439133 disk.rbd3.out‚0 disk.sda2.in‚0 disk.dm-0.out‚58114.41796875 cpu‚25.585528955102273$ disk./.usageGB‚557.3879699707031åá^úž‚(:“ „ÚèÒ‹Á¬ø gpu.0.gpu‚88.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚258.057' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚51.61140000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‡hêû‘‚(:ð „ÚèÒΖÃù) proc.memory.percent‚17.25142002105713 proc.cpu.threads‚17 disk.rbd1.in‚0" disk.nvme0n1.in‚10853.26953125 cpu‚25.58500772716923 disk.sda3.in‚0 disk.drbd2165.in‚0# proc.memory.rssMB‚ 2826.47265625 disk.dm-1.out‚ 189.4453125 network.recv‚ 1.250902e+06$ disk.nvme0c0n1.in‚10853.26953125& disk.nvme0n1.out‚58123.83642578125$ disk./.usageGB‚557.3923225402832 disk.sda2.in‚0# disk.dm-0.out‚58123.83642578125 disk.sda1.in‚0 disk.drbd2165.out‚0 disk.rbd3.in‚0 disk.sda1.out‚0 disk.sda3.out‚ 189.4453125$ memory_percent‚84.63797569274902( proc.memory.availableMB‚ 2516.9140625 disk.sda.out‚ 189.4453125 disk.dm-1.in‚0 network.sent‚ 5.502366e+06 disk.sda2.out‚0 disk.rbd1.out‚ 11.6640625 disk.sda.in‚0 disk.dm-0.in‚10853.26953125( disk.nvme0c0n1.out‚58123.83642578125( disk./.usagePercent‚37.4070664489879 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.rbd3.out‚0©"\Ÿ’‚(:” “ÚèÒå§ê÷ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚277.903' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚55.580600000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0w‚÷—“‚(:ô “ÚèÒꤎù network.recv‚ 1.2545e+06 disk.sda3.out‚ 190.12890625 disk.drbd2165.out‚0$ disk./.usageGB‚557.3973770141602"2Ïdæh disk.nvme0n1.in‚10853.26953125 disk.sda.out‚ 190.12890625 disk.dm-0.in‚10853.26953125 disk.sda1.out‚0$ memory_percent‚84.63809490203857 disk.sda2.out‚0 disk.rbd1.in‚0 disk.rbd3.in‚0) proc.memory.availableMB‚ 2516.89453125 proc.cpu.threads‚17 disk.sda.in‚0 disk.dm-1.in‚0 disk.sda3.in‚0$ disk.nvme0c0n1.in‚10853.26953125( disk.nvme0c0n1.out‚58137.37158203125& disk.nvme0n1.out‚58137.37158203125 disk.sda1.in‚0 disk.sda2.in‚0# disk.dm-0.out‚58137.37158203125# proc.memory.rssMB‚ 2826.47265625 network.sent‚ 5.506876e+06) disk./.usagePercent‚37.40740565897077 disk.rbd0.out‚0 disk.rbd3.out‚0 disk.rbd1.out‚ 11.6640625 disk.dm-1.out‚ 190.12890625 disk.rbd0.in‚0 disk.drbd2165.in‚0) proc.memory.percent‚17.25142002105713 cpu‚25.584392685448375 ã&‰ž”‚(:“ ¢ÚèÒ¥þøö gpu.0.gpu‚99.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚46.0 gpu.0.powerWatts‚327.145' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚65.42899999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0éjœÒ•‚(:õ ¢ÚèÒìäù network.sent‚ 5.512259e+06 network.recv‚ 1.256958e+06# disk./.usageGB‚557.403148651123 disk.sda3.in‚0) proc.memory.availableMB‚ 2517.12109375 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.dm-0.in‚10853.26953125 disk.sda3.out‚ 190.32421875" disk.nvme0n1.in‚10853.26953125 disk.rbd1.out‚ 11.7890625$ memory_percent‚84.63671207427979 proc.cpu.threads‚17 disk.drbd2165.in‚0# proc.memory.rssMB‚ 2826.47265625 cpu‚25.583827870771824 disk.dm-1.out‚ 190.32421875# disk.dm-0.out‚58145.05126953125 disk.drbd2165.out‚0 disk.sda2.out‚0 disk.rbd1.in‚0 disk.sda.out‚ 190.32421875 disk.dm-1.in‚0 disk.rbd3.in‚0 disk.rbd3.out‚0) proc.memory.percent‚17.25142002105713) disk./.usagePercent‚37.40779299837007 disk.sda1.in‚0 disk.sda2.in‚0 disk.sda.in‚0$ disk.nvme0c0n1.in‚10853.26953125& disk.nvme0n1.out‚58145.05126953125 disk.sda1.out‚0( disk.nvme0c0n1.out‚58145.05126953125 7þŠ”–‚(:‰ ±ÚèÒ÷ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚251.813' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚50.3626 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0hÁý—‚(:ò ±ÚèÒŸªù$ memory_percent‚84.63830947875977 disk.rbd0.in‚0 disk.rbd0.out‚0# disk.dm-0.out‚58170.58349609375 disk.drbd2165.in‚0 disk.rbd3.out‚0" disk.nvme0n1.in‚11009.60546875 disk.rbd1.out‚ 11.7890625# proc.memory.rssMB‚ 2826.47265625) disk./.usagePercent‚37.40835672696047# disk./.usageGB‚557.411548614502 disk.dm-0.in‚11009.60546875 disk.sda3.out‚ 192.41796875 disk.sda.in‚0 proc.cpu.threads‚17 cpu‚25.58316622106679 disk.sda1.in‚0( disk.nvme0c0n1.out‚58170.58349609375' proc.memory.availableMB‚ 2516.859375 network.recv‚ 1.261844e+06 disk.sda3.in‚0 disk.drbd2165.out‚0$ disk.nvme0c0n1.in‚11009.60546875 disk.sda2.in‚0 disk.rbd1.in‚0) proc.memory.percent‚17.25142002105713 disk.dm-1.in‚0 disk.dm-1.out‚ 192.41796875 disk.sda.out‚ 192.41796875 network.sent‚ 5.518866e+06 disk.rbd3.in‚0 disk.sda2.out‚0 disk.sda1.out‚0& disk.nvme0n1.out‚58170.58349609375 ±âŸ˜‚(:” ÀÚèÒœŠ€÷ gpu.0.gpu‚100.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚46.0 gpu.0.powerWatts‚330.913' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚66.18260000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0S¢Úæü™‚(:ñ ÀÚèÒØîù network.sent‚ 5.522451e+06) disk./.usagePercent‚37.40841612070841 disk.dm-0.in‚11009.60546875$ disk./.usageGB‚557.4124336242676 disk.rbd1.out‚ 11.8515625" disk.dm-0.out‚58184.9697265625) proc.memory.availableMB‚ 2517.07421875 cpu‚25.582599322812804 proc.cpu.threads‚17 disk.rbd3.in‚0" disk.nvme0n1.in‚11009.60546875 disk.dm-1.in‚0 disk.dm-1.out‚ 194.17578125 disk.rbd0.in‚0 disk.sda3.out‚ 194.17578125# memory_percent‚84.6369981765747 disk.rbd1.in‚0 disk.sda.out‚ 194.17578125 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda3.in‚0 disk.drbd2165.out‚0' disk.nvme0c0n1.out‚58184.9697265625% disk.nvme0n1.out‚58184.9697265625 disk.sda2.out‚0 disk.rbd0.out‚0# proc.memory.rssMB‚ 2826.47265625 network.recv‚ 1.26472e+06 disk.sda2.in‚0 disk.rbd3.out‚0$ disk.nvme0c0n1.in‚11009.60546875 disk.sda.in‚0 disk.drbd2165.in‚0) proc.memory.percent‚17.25142002105713NW{2š- _wandbruntime‚4906  _runtime‚4906®û\Ç›Á & lr-AdamW/pg1‚0.0006633757598664704 & lr-AdamW/pg2‚0.0006633757598664704 trainer/global_step‚2949 % _timestamp‚1.7842947220958347e+09 _runtime‚4906.313012473 _step‚239±Oô”ÌœÆ trainer/global_step‚2949 % _timestamp‚1.7842947220958347e+09 _runtime‚4906.313012473 _step‚239 & lr-AdamW/pg1‚0.0006633757598664704 & lr-AdamW/pg2‚0.0006633757598664704ïµÚ=a2- _wandbruntime‚4906  _runtime‚4906G7“*ï'žéO VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 % _timestamp‚1.7842947220969296e+09 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚15951667 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚846985526116352 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚50674633 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3577923236864 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8383192 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚849387184924372 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2774178711040 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚25538520 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚846985499942912 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3577884506112 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2769432973270 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚849758333264384 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 406821888 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 trainer/global_step‚2949 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚846985499942912 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚849759704827392 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2774178711040 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚849388554952662 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 _runtime‚4906.314338505 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚25136113 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚50664654 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚25538520 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8383479 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚229113439744000 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚38730752 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚849759704827392 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚25528547 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚15951959 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚232691324250112 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 _step‚240 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚846985526116352 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440á;—ô'ŸîO KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚25538520 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚846985499942912 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3577884506112 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2769432973270 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚849758333264384 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 406821888 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 trainer/global_step‚2949 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚846985499942912 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚849759704827392 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2774178711040 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚849388554952662 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 _runtime‚4906.314338505 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚25136113 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚50664654 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚25538520 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8383479 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚229113439744000 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚38730752 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚849759704827392 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚25528547 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚15951959 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚232691324250112 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 _step‚240 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚846985526116352 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 % _timestamp‚1.7842947220969296e+09 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚15951667 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚846985526116352 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚50674633 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3577923236864 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8383192 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚849387184924372 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2774178711040 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0ð!G 2 - _wandbruntime‚4907  _runtime‚4907Kº^ûY¡óM YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2775103753728 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚846897003757568 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8386217 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚25537025 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚847263526846464 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚232770384872448 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2773758356992 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2769011839194 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2775103753728 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚25537025 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚241 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚232769978057728 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚849665990168794 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 _runtime‚4907.698518306 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2770356439628 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚850038630600192 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚25547012 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚50691331 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7571023 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚25144319 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3579051906048 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚25547012 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚15957240 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚847263500673024 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15956960 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995M{^Ýþ&¢øM PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚50691331 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7571023 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚25144319 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3579051906048 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚25547012 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚15957240 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚847263500673024 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15956960 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2775103753728 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚846897003757568 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8386217 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚25537025 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚847263526846464 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚232770384872448 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2773758356992 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2769011839194 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2775103753728 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚25537025 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚241 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚232769978057728 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚849665990168794 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 _runtime‚4907.698518306 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2770356439628 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚850038630600192 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚25547012 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 FDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2814325486056 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚25546407 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚863706057263616 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚25964272 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚232988128219136 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚16216270 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚863704685700608 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2820494641664 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚860513185460224 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 _runtime‚4985.117928805 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚860885562621952 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚860885536448512 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚25546407 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2819149252096 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚25964272 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚860885536448512 UADeviceStatsMonitor.on_t Éöõrain_batch_start/requested_bytes.all.freed‚863327510946280 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640×'Vó'µíO C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚236624576911872 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚51510685 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚863328880974570 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2815670086378 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3636816783872 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚2999 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3636851320320 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2814325486056 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚25546407 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚863706057263616 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚25964272 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚232988128219136 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚16216270 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚863704685700608 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2820494641664 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚860513185460224 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 _runtime‚4985.117928805 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚860885562621952 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚860885536448512 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚25546407 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2819149252096 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚25964272 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚860885536448512 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚863327510946280 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 _step‚244 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 IDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚16221530 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚236703248515584 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3638001979904 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2821419684352 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚863607686219104 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚51527383 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _runtime‚4986.499289717 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚51517390 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚863984983036416 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚245 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7698168 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚233065614626816 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚25554619 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7698173 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚236703651136000 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚863984983036416 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2821419684352 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚25972764 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2820074287616 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚16221280 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚2999 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚51527383 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚25962777 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚25554619 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2815248952302 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993|x¹œý&¸÷M M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚863607686219104 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚51527383 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _runtime‚4986.499289717 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚51517390 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚863984983036416 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚245 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7698168 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚233065614626816 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚25554619 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7698173 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚236703651136000 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚863984983036416 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2821419684352 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚25972764 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2820074287616 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚16221280 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚2999 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚51527383 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚25962777 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚25554619 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2815248952302 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚861163563352064 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚861163537178624 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2820074287616 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚25554613 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8523112 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚16221530 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚236703248515584 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3638001979904 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2821419684352 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0õÿQë62¹- _wandbruntime‚4986  _runtime‚4986-íî( º„ _runtime‚4986.585346217 2loss/SRqqt1/assignment_loss‚0.6222469412246365 3loss/SRqqt2/assignment_loss‚0.48360527123239694 /loss/FRt1/detection_loss‚0.5314222626538213 0loss/SRqqt2/detection_loss‚0.656230333952303 %loss/total_loss‚8.485540940280341 % _timestamp‚1.7842948023684368e+09 /loss/FBt1/detection_loss‚0.7395281142788008 2loss/FBt1/assignment_loss‚0.016598710730931998 2loss/FBt2/assignment_loss‚0.011757675715358832 1loss/SRqqt1/detection_loss‚0.9129828140007944 trainer/global_step‚2999 _step‚246 0loss/FRt1/assignment_loss‚1.7981793946321074 0loss/FRt2/assignment_loss‚0.7211819180292101 /loss/FRt2/detection_loss‚1.5419412603550333 0loss/FBt2/detection_loss‚0.44986624347494686 epoch‚7§M]q»‰ /loss/FBt1/detection_loss‚0.7395281142788008 2loss/FBt1/assignment_loss‚0.016598710730931998 2loss/FBt2/assignment_loss‚0.011757675715358832 1loss/SRqqt1/detection_loss‚0.9129828140007944 trainer/global_step‚2999 _step‚246 0loss/FRt1/assignment_loss‚1.7981793946321074 0loss/FRt2/assignment_loss‚0.7211819180292101 /loss/FRt2/detection_loss‚1.5419412603550333 0loss/FBt2/detection_loss‚0.44986624347494686 epoch‚7 _runtime‚4986.585346217 2loss/SRqqt1/assignment_loss‚0.6222469412246365 3loss/SRqqt2/assignment_loss‚0.48360527123239694 /loss/FRt1/detection_loss‚0.5314222626538213 0loss/SRqqt2/detection_loss‚0.656230333952303 %loss/total_loss‚8.485540940280341 % _timestamp‚1.7842948023684368e+09ö8 ty”¼‚(:‰ šÛèÒËÎÐö gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚46.0 gpu.0.powerWatts‚336.569' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚67.3138 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0µ±u̽‚(:ç šÛèÒ©öÃø network.recv‚ 1.283915e+06 disk.sda1.in‚0 disk.nvme0c0n1.in‚ 11130.4375' disk.nvme0c0n1.out‚58290.2939453125 network.sent‚ 5.603337e+06 disk.sda.in‚0 côÂ& disk.sda3.in‚0 disk.drbd2165.in‚0 disk.nvme0n1.in‚ 11130.4375) proc.memory.availableMB‚ 1545.69921875 disk.rbd0.in‚0 disk.sda3.out‚ 205.37890625 disk.drbd2165.out‚0 disk.rbd3.out‚0 disk.sda2.out‚0) disk./.usagePercent‚37.41078982256589$ disk./.usageGB‚557.4478034973145 disk.rbd3.in‚0# proc.memory.rssMB‚ 2826.47265625 proc.cpu.threads‚17 disk.sda.out‚ 205.37890625 disk.rbd0.out‚0 disk.rbd1.in‚0 disk.dm-1.in‚0" disk.dm-0.out‚58290.2939453125) proc.memory.percent‚17.25142002105713 disk.rbd1.out‚ 12.0390625 disk.dm-1.out‚ 205.37890625 disk.dm-0.in‚ 11130.4375% disk.nvme0n1.out‚58290.2939453125 cpu‚25.579587826098518 disk.sda1.out‚0 disk.sda2.in‚0$ memory_percent‚90.56580066680908ûØü”¾‚(:‰ ©ÛèÒÙîÿö gpu.0.gpu‚99.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚293.752' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚58.7504 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0±‰9Ìö¿‚(:ë ©ÛèÒ§­ù network.recv‚ 1.287569e+06 disk.sda.in‚0 disk.rbd0.in‚0 disk.sda1.in‚0" disk.nvme0n1.in‚11260.09765625* disk./.usagePercent‚37.411021253376845$ disk./.usageGB‚557.4512519836426 disk.sda.out‚ 206.05078125" disk.nvme0n1.out‚ 58300.5234375 cpu‚25.579940316018934 disk.dm-1.out‚ 206.05078125 disk.rbd0.out‚0 disk.sda3.out‚ 206.05078125 disk.drbd2165.out‚0 disk.dm-0.out‚ 58300.5234375 disk.drbd2165.in‚0$ disk.nvme0c0n1.in‚11260.09765625 disk.sda2.out‚0 disk.dm-0.in‚11260.09765625$ disk.nvme0c0n1.out‚ 58300.5234375$ memory_percent‚90.56751728057861 disk.sda3.in‚0 disk.rbd3.out‚0) proc.memory.availableMB‚ 1545.41796875# proc.memory.rssMB‚ 2826.47265625 network.sent‚ 5.611462e+06 disk.dm-1.in‚0 disk.sda1.out‚0 disk.rbd1.out‚ 12.0390625 proc.cpu.threads‚17 disk.rbd3.in‚0 disk.sda2.in‚0 disk.rbd1.in‚0) proc.memory.percent‚17.25142002105713€˜XÏ”À‚(:‰ ¸ÛèÒõÜÚö gpu.0.gpu‚97.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚238.946' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚47.7892 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0j„) ðÁ‚(:å ¸ÛèÒÎʽø network.sent‚ 5.614279e+06 disk.sda1.out‚0( disk.nvme0c0n1.out‚58308.30419921875 disk.sda2.in‚0 disk.rbd3.out‚0# disk.dm-0.out‚58308.30419921875 disk.drbd2165.in‚0 disk.rbd3.in‚0& disk.nvme0n1.out‚58308.30419921875) proc.memory.percent‚17.25142002105713 disk.rbd0.out‚0 network.recv‚ 1.289924e+06 disk.sda3.in‚0 disk.sda2.out‚0 disk.sda.in‚0 disk.sda1.in‚0 disk.drbd2165.out‚0 disk.rbd1.out‚ 12.1015625 disk.sda.out‚206.875 disk.rbd0.in‚0( proc.memory.availableMB‚ 1545.4140625 proc.cpu.threads‚17) disk./.usagePercent‚37.41142088113779 disk.dm-0.in‚11362.26171875" disk.nvme0n1.in‚11362.26171875 disk.rbd1.in‚0$ memory_percent‚90.56754112243652# proc.memory.rssMB‚ 2826.47265625 cpu‚25.58045249418911 disk.sda3.out‚206.875 disk.dm-1.in‚0$ disk./.usageGB‚557.4572067260742$ disk.nvme0c0n1.in‚11362.26171875 disk.dm-1.out‚206.875==”‚(:‰ ÇÛèÒ„ÎÄø gpu.0.gpu‚90.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚293.271' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚58.6542 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0·´ôÂ(:é ÇÛèÒÉøžù disk.sda2.in‚0 disk.dm-0.in‚11362.26171875 cpu‚25.580699880909616* disk./.usagePercent‚37.411782875791204$ disk.nvme0c0n1.out‚ 58319.9921875 disk.rbd1.out‚ 12.1015625 disk.drbd2165.out‚0 disk.rbd3.out‚0$ disk.nvme0c0n1.in‚11362.26171875 disk.sda1.out‚0 proc.cpu.threads‚17 network.sent‚ 5.683383e+06 network.recv‚ 1.296769e+06" disk.nvme0n1.in‚11362.26171875 disk.rbd0.out‚0 disk.drbd2165.in‚0' proc.memory.availableMB‚ 1545.171875) proc.memory.percent‚17.25142002105713 disk.rbd0.in‚0 disk.sda3.out‚ 207.27734375$ disk./.usageGB‚557.4626007080078" disk.nvme0n1.out‚ 58319.9921875 disk.sda2.out‚0 disk.sda.in‚0 disk.sda.out‚ 207.27734375 disk.dm-1.in‚0 disk.dm-1.out‚ 207.27734375 disk.dm-0.out‚ 58319.9921875 disk.rbd3.in‚0$ memory_percent‚90.56901931762695# proc.memory.rssMB‚ 2826.47265625 disk.rbd1.in‚0 disk.sda1.in‚0 disk.sda3.in‚0 ¼ŽgŸÄ‚(:” ÖÛèÒËòÝö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚40.0 gpu.0.powerWatts‚111.112' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.222399999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Qì©üÅ‚(:ñ ÖÛèÒŽþõø' proc.memory.availableMB‚ 1545.171875 network.sent‚ 5.686937e+06 network.recv‚ 1.299058e+06 disk.sda.out‚ 207.5234375) proc.memory.percent‚17.25142002105713 cpu‚25.580957165859907 disk.drbd2165.out‚0 disk.sda2.in‚0 proc.cpu.threads‚17 disk.rbd0.out‚0 disk.sda1.in‚0( disk.nvme0c0n1.out‚58328.58740234375 disk.rbd1.in‚0 disk.sda.in‚0$ disk.nvme0c0n1.in‚11362.26171875# proc.memory.rssMB‚ 2826.47265625$ disk./.usageGB‚557.4673538208008 disk.rbd1.out‚ 12.1015625 disk.drbd2165.in‚0$ memory_percent‚90.56901931762695 disk.dm-1.out‚ 207.5234375 disk.rbd0.in‚0 disk.dm-0.in‚11362.26171875# disk.dm-0.out‚58328.58740234375 disk.sda3.in‚0 disk.rbd3.in‚0" disk.nvme0n1.in‚11362.26171875& disk.nvme0n1.out‚58328.58740234375 disk.dm-1.in‚0) disk./.usagePercent‚37.41210186117887 disk.sda1.out‚0 disk.sda3.out‚ 207.5234375 disk.rbd3.out‚0 disk.sda2.out‚0‹&B2Æ- _wandbruntime‚5064  _runtime‚5064žL?™ÇÇÁ & lr-AdamW/pg2‚0.0006611322112935787 trainer/global_step‚3049 % _timestamp‚1.7842948802147493e+09 _runtime‚5064.431940326 _step‚247 & lr-AdamW/pg1‚0.0006611322112935787û%“¬ÌÈÆ % _timestamp‚1.7842948802147493e+09 _runtime‚5064.431940326 _step‚247 & lr-AdamW/pg1‚0.0006611322112935787 & lr-AdamW/pg2‚0.0006611322112935787 trainer/global_step‚3049÷nŠÏ 2É- _wandbruntime‚5064  _runtime‚5064[KšÁî'ÊèO ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚240 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚877651038136832 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚240559233038848 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2865465182720 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚16480894 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3696751996416 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8658119 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚32439296 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚874785572954112 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚877652409699840 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚25956713 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚877267836968188 _runtime‚5064.433127847 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 400530432 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚26390024 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚25956713 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚26380051 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2866810572288 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚877651038136832 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 $ _timestamp‚1.784294880215782e+09 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚26390024 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚877269206996478 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚240558832508416 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚52336758 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572ÈF†ó'ËíO UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚877267836968188 _runtime‚5064.433127847 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 400530432 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚26390024 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚25956713 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚26380051 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2866810572288 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚877651038136832 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 $ _timestamp‚1.784294880215782e+09 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚26390024 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚877269206996478 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚240558832508416 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚52336758 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚240 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚877651038136832 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚240559233038848 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2865465182720 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚16480894 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3696751996416 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8658119 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚32439296 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚874785572954112 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚877652409699840 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚25956713 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973ø"m2Ì- _wandbruntime‚5065  _runtime‚5065N,š$(ÍóM K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚16486379 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚25964913 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2866390218240 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚25964919 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8661056 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚240637918029312 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2862830665844 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7825323 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8660831 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 _runtime‚5065.818264836 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚236939935010816 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚26388529 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2866390218240 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚877931335472640 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚877929963902464 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚874685181575168 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚874685156147200 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2867735614976 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3697983018496 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2861486065410 F=DeviceStatsMonitor.on_train_batch_e-ªTÑnd/segment.large_pool.peak‚1175 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 400523264 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚877548012241012 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚877546642212610 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚16486149 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚52363435 _step‚249 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚52363435 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚875063599857664 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚25964913 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚875063573684224 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7842948816009088e+09 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚25964919 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚877931335472640 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚225 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚875063573684224 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚52353442 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008ªU›Ìþ&ÎøM WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚877931335472640 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚877929963902464 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚874685181575168 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚874685156147200 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2867735614976 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3697983018496 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2861486065410 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 400523264 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚877548012241012 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚877546642212610 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚16486149 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚52363435 _step‚249 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚52363435 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚875063599857664 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚25964913 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚875063573684224 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7842948816009088e+09 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚25964919 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚877931335472640 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚225 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚875063573684224 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚52353442 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚16486379 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚25964913 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2866390218240 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚25964919 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8661056 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚240637918029312 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2862830665844 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7825323 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8660831 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 _runtime‚5065.818264836 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚236939935010816 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚26388529 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2866390218240ùVo'2Ï- _wandbruntime‚5065  _runtime‚5065'•(´Ð€ 1loss/SRqqt1/detection_loss‚0.9910043190761756 /loss/FBt1/detection_loss‚0.9049958847463131 /loss/FRt2/detection_loss‚1.6247557934665704 $loss/total_loss‚8.99420034595566 trainer/global_step‚3049 % _timestamp‚1.7842948816880589e+09 /loss/FRt1/assignment_loss‚1.815268963854411 0loss/FRt2/assignment_loss‚0.7212028736320437 2loss/SRqqt2/assignment_loss‚0.5283779270901913 1loss/SRqqt2/detection_loss‚0.6143830186623559 2loss/FBt2/assignment_loss‚0.040513683937791714 /loss/FRt1/detection_loss‚0.5322769578126051 /loss/FBt2/detection_loss‚0.5351119849986087 epoch‚7 _runtime‚5065.904990722 _step‚250 2loss/SRqqt1/assignment_loss‚0.6433073021540342 1loss/FBt1/assignment_loss‚0.04300163652455902Æd† Ñ… /loss/FRt1/assignment_loss‚1.815268963854411 0loss/FRt2/assignment_loss‚0.7212028736320437 2loss/SRqqt2/assignment_loss‚0.5283779270901913 1loss/SRqqt2/detection_loss‚0.6143830186623559 2loss/FBt2/assignment_loss‚0.040513683937791714 /loss/FRt1/detection_loss‚0.5322769578126051 /loss/FBt2/detection_loss‚0.5351119849986087 epoch‚7 _runtime‚5065.904990722 _step‚250 2loss/SRqqt1/assignment_loss‚0.6433073021540342 1loss/FBt1/assignment_loss‚0.04300163652455902 1loss/SRqqt1/detection_loss‚0.9910043190761756 /loss/FBt1/detection_loss‚0.9049958847463131 /loss/FRt2/detection_loss‚1.6247557934665704 $loss/total_loss‚8.99420034595566 trainer/global_step‚3049 % _timestamp‚1.7842948816880589e+09ú”\Ú±žÒ‚(:“ åÛèÒÉ”âö gpu.0.gpu‚99.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚300.808' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚60.16160000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚03yÛ’øÓ‚(:í åÛèÒö‘‡ù network.recv‚ 1.30188e+06$ disk./.usageGB‚557.4727592468262! disk.dm-0.out‚58340.521484375$ disk.nvme0c0n1.in‚11362.26171875" disk.nvme0n1.in‚11362.26171875 disk.sda2.out‚0$ memory_percent‚90.56921005249023 cpu‚25.581306346254504 network.sent‚ 5.689796e+06 disk.dm-0.in‚11362.26171875 disk.rbd1.out‚ 12.1640625 disk.sda.out‚ 207.68359375) disk./.usagePercent‚37.41246462385489 disk.sda3.out‚ 207.68359375 disk.drbd2165.in‚0 disk.rbd3.out‚0 disk.rbd1.in‚0 disk.dm-1.in‚0& disk.nvme0c0n1.out‚58340.521484375 disk.sda1.out‚0 disk.rbd3.in‚0 disk.dm-1.out‚ 207.68359375' proc.memory.availableMB‚ 1545.140625 disk.sda1.in‚0 disk.sda3.in‚0 disk.sda2.in‚0 disk.sda.in‚0# proc.memory.rssMB‚ 2826.47265625) proc.memory.percent‚17.25142002105713 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.drbd2165.out‚0$ disk.nvme0n1.out‚58340.521484375Rýè)ŸÔ‚(:” ôÛèÒÚœßö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚284.671' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚56.934200000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0íwVÈóÕ‚(:è ôÛèÒƒÞ ù disk.dm-1.in‚0 disk.drbd2165.in‚0$ disk.nvme0c0n1.in‚11545.45703125& disk.nvme0n1.out‚58349.53076171875 disk.sda2.out‚0$ proc.memory.availableMB‚1545.375$ disk./.usageGB‚557.4780311584473 disk.sda.in‚0 disk.sda1.in‚0$ memory_percent‚90.56777954101562 disk.rbd3.in‚0 disk.rbd1.out‚ 12.1640625# proc.memory.rssMB‚ 2826.47265625) proc.memory.percent‚17.25142002105713 network.recv‚ 1.305428e+06 disk.rbd0.in‚0 disk.sda3.out‚ 207.96875 disk.rbd1.in‚0 proc.cpu.threads‚17 disk.dm-1.out‚ 207.96875# disk.dm-0.out‚58349.53076171875 disk.sda1.out‚0 disk.sda3.in‚0 disk.rbd3.out‚0 disk.rbd0.out‚0( disk.nvme0c0n1.out‚58349.53076171875 disk.sda2.in‚0) disk./.usagePercent‚37.41281842626721" disk.nvme0n1.in‚11545.45703125 disk.sda.out‚ 207.96875 disk.dm-0.in‚11545.45703125 disk.drbd2165.out‚0 cpu‚25.581554948355013 network.sent‚ 5.692995e+06:ó“”Ö‚(:‰ ƒÜèÒ©ã‰÷ gpu.0.gpu‚99.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚337.816' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚67.5632 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0^^T”õׂ(:ê ƒÜèÒIJ®ù network.sent‚ 5.695854e+06 disk.sda1.out‚0 network.recv‚ 1.30802e+06 disk.rbd3.in‚0$ memory_percent‚90.56780338287354 proc.cpu.threads‚17 disk.sda.out‚ 208.21875# disk.dm-0.out‚58362.42041015625 disk.sda1.in‚0 cpu‚25.5820957501741 disk.dm-1.out‚ 208.21875 disk.rbd0.in‚0 disk.dm-0.in‚11545.45703125# proc.memory.rssMB‚ 2826.47265625 disk.dm-1.in‚0 disk.rbd0.out‚0 disk.sda3.out‚ 208.21875 disk.drbd2165.in‚0 disk.drbd2165.out‚0" disk.nvme0n1.in‚11545.45703125& disk.nvme0n1.out‚58362.42041015625 disk.sda2.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 12.2265625) disk./.usagePercent‚37.41304217685214$ disk./.usageGB‚557.4813652038574$ disk.nvme0c0n1.in‚11545.45703125 disk.sda2.in‚0) proc.memory.availableMB‚ 1545.37109375) proc.memory.percent‚17.25142002105713 disk.sda.in‚0 disk.sda3.in‚0 disk.rbd3.out‚0( disk.nvme0c0n1.out‚58362.42041015625ØmŽŸØ‚(:” ’ÜèÒðëÐö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚291.919' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚58.383799999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0õÄãÙ‚(:ö ’ÜèÒùĹø$ disk.nvme0c0n1.in‚11545.45703125" disk.nvme0n1.in‚11545.45703125 disk.sda2.out‚0# proc.memory.rssMB‚ 2826.47265625 disk.sda1.out‚0 network.recv‚ 1.310535e+06 disk.sda.in‚0 disk.dm-1.out‚ 208.54296875 disk.rbd0.out‚0$ memory_percent‚90.56603908538818( disk.nvme0c0n1.out‚58370.69091796875 disk.rbd1.in‚0 disk.rbd1.out‚ 12.2265625 disk.dm-0.in‚11545.45703125 disk.sda3.in‚0 disk.drbd2165.in‚0& disk.nvme0n1.out‚58370.69091796875) proc.memory.availableMB‚ 1545.66015625 disk.sda2.in‚0# disk.dm-0.out‚58370.69091796875 cpu‚25.582352166235534 network.sent‚ 5.698671e+06$ disk./.usageGB‚557.4869422912598 disk.sda.out‚ 208.54296875 disk.drbd2165.out‚0 disk.rbd3.in‚0 disk.rbd3.out‚0) proc.memory.percent‚17.25142002105713 proc.cpu.threads‚17 disk.sda1.in‚0 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.sda3.out‚ 208.54296875) disk./.usagePercent‚37.41341645986719©zù¸žÚ‚(:“ ¡ÜèÒ›³ÿö gpu.0.gpu‚99.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚330.913' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚66.18260000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0yø0NüÛ‚(:ñ ¡ÜèÒÙ®æø( proc.memory.availableMB‚ 1545.6640625) proc.memory.percent‚17.25142002105713 cpu‚25.58259394352388 disk.sda3.in‚0( disk.nvme0c0n1.out‚58384.01611328125 disk.rbd3.out‚0 disk.sda3.out‚ 208.8203125" disk.nvme0n1.in‚11545.45703125 disk.sda2.in‚0 proc.cpu.threads‚17 disk.dm-0.in‚11545.45703125$ disk.nvme0c0n1.in‚11545.45703125 disk.dm-1.out‚ 208.8203125# proc.memory.rssMB‚ 2826.47265625 disk.rbd0.out‚0 network.recv‚ 1.313052e+06) disk./.usagePercent‚37.41342209203295$ disk./.usageGB‚557.4870262145996 disk.sda1.out‚0 disk.drbd2165.in‚0# disk.dm-0.out‚58384.01611328125 disk.rbd3.in‚0 disk.rbd1.in‚0 disk.rbd0.in‚0$ memory_percent‚90.56601524353027 disk.sda1.in‚0& disk.nvme0n1.out‚58384.01611328125 disk.sda.out‚ 208.8203125 disk.dm-1.in‚0 disk.sda2.out‚0 disk.sda.in‚0 disk.drbd2165.out‚0 network.sent‚ 5.701631e+06 disk.rbd1.out‚ 12.2890625·‘ãX2Ü- _wandbruntime‚5143  _runtime‚5143ŠhÉ"ÇÝÁ _runtime‚5143.586018119 _step‚251 & lr-AdamW/pg1‚0.0006599800328263267 & lr-AdamW/pg2‚0.0006599800328263267 trainer/global_step‚3099 % _timestamp‚1.7842949593686168e+09©§×ÌÞÆ & lr-AdamW/pg1‚0.0006599800328263267 & lr-AdamW/pg2‚0.0006599800328263267 trainer/global_step‚3099 % _timestamp‚1.7842949593686168e+09 _runtime‚5143.586018119 _step‚251ûÃæz2ß- _wandbruntime‚5143  _runtime‚5143ò9²÷€!àèO UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚891208162990096 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 trainer/global_step‚3099 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2913126502912 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚16745280 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚244493356055040 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚244492955524608 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚26367013 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚26367007 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚26815776 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚891597390573056 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚888301363277824 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚891598762136064 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚240736768987136 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚888301388705792 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2911781113344 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚26805803 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚53182789 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚891597390573056 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚249 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8795599 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚240736400896000 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3756554628608 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8795355 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 400530432 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2911781113344 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚888685609459712 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚891598762136064 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚888685609459712 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚26815776 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2908144312594 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7842949593699422e+09 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚240736768987136 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚888301388705792 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2911781113344 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚26805803 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚53182789 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚891597390573056 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚249 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8795599 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚240736400896000 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3756554628608 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8795355 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 400530432 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2911781113344 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚888685609459712 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚891598762136064 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚888685609459712 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚26815776 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2908144312594 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7842949593699422e+09 IDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚888963610189824 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚241 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚26375219 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚53199487 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚53189494 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚888963610189824 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚246 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2912706148864 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚891877687908864 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚26375219 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚888963636363264 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2912706148864 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚26375213 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚26824268 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3757742675456 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚891488338262920 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚888963636363264 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8798351 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2914051545600 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚888579270483968 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚26824268 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚26814281 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚26814281 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚16750578 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚888579245056000 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚240814255394816 _step‚253 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2907723178518 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2909067778952 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0·*ŒÜþ&äøM K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚888963610189824 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚246 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2912706148864 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚891877687908864 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚26375219 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚888963636363264 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2912706148864 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚26375213 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚26824268 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3757742675456 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚891488338262920 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚888963636363264 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8798351 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2914051545600 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚888579270483968 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚26824268 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚26814281 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚26814281 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚16750578 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚888579245056000 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚240814255394816 _step‚253 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2907723178518 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2909067778952 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 % _timestamp‚1.7842949607568145e+09 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚244572032599552 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚240813887303680 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7952468 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3757777204736 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚244571629979136 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 _runtime‚5144.974203817 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚891876316338688 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚888963610189824 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚241 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚26375219 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚53199487 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚53189494 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0ý¼Ž¹0”å‚(:‰ °ÜèÒþÔÚö gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚304.288' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚60.8576 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0!Õ8Ÿ±æ‚(:ó °ÜèÒŒËÖù disk.dm-1.in‚0( disk.nvme0c0n1.out‚58393.17919921875 cpu‚25.581958221343875 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda1.out‚0 disk.rbd3.in‚0 disk.rbd1.in‚0$ disk.nvme0c0n1.in‚11638.76953125 proc.cpu.threads‚17 disk.sda.out‚ 209.0859375# disk.dm-0.out‚58393.17919921875 disk.sda1.in‚0 disk.drbd2165.out‚0 disk.rbd3.out‚0 network.recv‚ 1} .M.318367e+06$ disk./.usageGB‚557.4920349121094 disk.sda.in‚0 disk.dm-0.in‚11638.76953125 disk.sda3.in‚0 network.sent‚ 5.76789e+06 disk.drbd2165.in‚0) proc.memory.availableMB‚ 1545.66015625* disk./.usagePercent‚37.413758229925406 disk.sda2.in‚0 disk.rbd1.out‚ 12.2890625& disk.nvme0n1.out‚58393.17919921875 disk.dm-1.out‚ 209.0859375 disk.sda3.out‚ 209.0859375" disk.nvme0n1.in‚11638.76953125$ memory_percent‚90.56603908538818# proc.memory.rssMB‚ 2826.47265625) proc.memory.percent‚17.25142002105713 disk.sda2.out‚0ûÑl¬2ç- _wandbruntime‚5145  _runtime‚5145%øè€ 2loss/SRqqt1/assignment_loss‚0.5122573016857817 2loss/FBt2/assignment_loss‚0.001854085924641122 /loss/FRt2/detection_loss‚1.5381714835290632 1loss/SRqqt2/detection_loss‚0.5899892145694642 _step‚254 0loss/SRqqt1/detection_loss‚0.945008428183735 /loss/FBt1/detection_loss‚0.8554699103392306 /loss/FBt2/detection_loss‚0.4963646022246239 %loss/total_loss‚8.339428072886935 _runtime‚5145.059191144 epoch‚7 2loss/SRqqt2/assignment_loss‚0.5165423228982532 1loss/FBt1/assignment_loss‚0.05706429587859826 /loss/FRt1/detection_loss‚0.5332142492643916 trainer/global_step‚3099 % _timestamp‚1.7842949608422558e+09 /loss/FRt1/assignment_loss‚1.715582370407097 0loss/FRt2/assignment_loss‚0.5779098079820569¢Ž¨a é… trainer/global_step‚3099 % _timestamp‚1.7842949608422558e+09 /loss/FRt1/assignment_loss‚1.715582370407097 0loss/FRt2/assignment_loss‚0.5779098079820569 2loss/SRqqt1/assignment_loss‚0.5122573016857817 2loss/FBt2/assignment_loss‚0.001854085924641122 /loss/FRt2/detection_loss‚1.5381714835290632 1loss/SRqqt2/detection_loss‚0.5899892145694642 _step‚254 0loss/SRqqt1/detection_loss‚0.945008428183735 /loss/FBt1/detection_loss‚0.8554699103392306 /loss/FBt2/detection_loss‚0.4963646022246239 %loss/total_loss‚8.339428072886935 _runtime‚5145.059191144 epoch‚7 2loss/SRqqt2/assignment_loss‚0.5165423228982532 1loss/FBt1/assignment_loss‚0.05706429587859826 /loss/FRt1/detection_loss‚0.5332142492643916þ’d”ê‚(:‰ ¿ÜèҤߠø gpu.0.gpu‚92.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚253.749' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚50.7498 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ó¡|úë‚(:ï ¿ÜèÒݸ«ù disk.sda2.out‚0 disk.rbd1.in‚0% disk.nvme0n1.out‚58404.3740234375 disk.rbd1.out‚ 12.3515625 disk.sda.in‚0 disk.dm-0.in‚11638.76953125 disk.rbd3.in‚0 disk.rbd0.out‚0 disk.sda1.out‚0 disk.rbd3.out‚0$ memory_percent‚90.56758880615234 proc.cpu.threads‚17 network.sent‚ 5.770749e+06" disk.nvme0n1.in‚11638.76953125 disk.sda1.in‚0 disk.sda3.out‚ 209.34765625& proc.memory.availableMB‚ 1545.40625 network.recv‚ 1.32084e+06* disk./.usagePercent‚37.414399272791144$ disk.nvme0c0n1.in‚11638.76953125" disk.dm-0.out‚58404.3740234375# proc.memory.rssMB‚ 2826.47265625 disk.sda.out‚ 209.34765625$ disk./.usageGB‚557.5015869140625) proc.memory.percent‚17.25142002105713 cpu‚25.58143514163069 disk.dm-1.in‚0 disk.dm-1.out‚ 209.34765625 disk.drbd2165.in‚0 disk.drbd2165.out‚0' disk.nvme0c0n1.out‚58404.3740234375 disk.sda2.in‚0 disk.rbd0.in‚0 disk.sda3.in‚0V4•lì‚(:… ÎÜèÒÞ„ø÷ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚293.8' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚58.76 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0·¾×üí‚(:ñ ÎÜèÒÜ–øø disk.rbd0.in‚0$ memory_percent‚90.56565761566162 disk.sda2.in‚0 disk.sda.out‚ 209.54296875 disk.dm-0.in‚11638.76953125 disk.sda1.in‚0 disk.sda3.in‚0 disk.drbd2165.in‚0) proc.memory.availableMB‚ 1545.72265625 cpu‚25.580915736660923 disk.drbd2165.out‚0" disk.nvme0n1.in‚11638.76953125# proc.memory.rssMB‚ 2826.47265625 network.sent‚ 5.843295e+06 disk.rbd1.out‚ 12.3515625$ disk./.usageGB‚557.5070610046387 disk.rbd1.in‚0 disk.rbd0.out‚0! disk.dm-0.out‚58414.009765625$ disk.nvme0n1.out‚58414.009765625 disk.rbd3.out‚0$ disk.nvme0c0n1.in‚11638.76953125& disk.nvme0c0n1.out‚58414.009765625 network.recv‚ 1.327352e+06* disk./.usagePercent‚37.414766643602775 disk.sda2.out‚0 disk.sda.in‚0 disk.rbd3.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 209.54296875 disk.sda1.out‚0 disk.sda3.out‚ 209.54296875) proc.memory.percent‚17.25142002105713 proc.cpu.threads‚17u¸% žî‚(:“ ÝÜèÒ™¼Óø gpu.0.gpu‚90.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚275.782' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚55.15639999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0,Ù“œúï‚(:ï ÝÜèÒèûÓù disk.sda2.out‚0 disk.rbd1.in‚0 disk.sda3.in‚0) proc.memory.availableMB‚ 1545.23046875$ disk./.usageGB‚557.5119934082031 disk.sda2.in‚0 disk.dm-0.in‚11638.76953125) disk./.usagePercent‚37.41509766134455 disk.rbd0.out‚0 disk.drbd2165.out‚0& disk.nvme0c0n1.out‚58425.013671875 disk.sda3.out‚ 209.81640625 cpu‚25.580291889832164 proc.cpu.threads‚17 disk.rbd3.out‚0 disk.dm-1.out‚ 209.81640625 network.sent‚ 5.847058e+06" disk.nvme0n1.in‚11638.76953125$ disk.nvme0n1.out‚58425.013671875 disk.rbd0.in‚0 disk.sda1.out‚0# memory_percent‚90.5686616897583 disk.sda1.in‚0 disk.rbd3.in‚0# proc.memory.rssMB‚ 2826.47265625 disk.rbd1.out‚ 12.4140625 disk.drbd2165.in‚0) proc.memory.percent‚17.25142002105713 disk.sda.out‚ 209.81640625! disk.dm-0.out‚58425.013671875$ disk.nvme0c0n1.in‚11638.76953125 disk.sda.in‚0 disk.dm-1.in‚0 network.recv‚ 1.330857e+06Ž”ÊŸð‚(:” ìÜèÒœ˜†÷ gpu.0.gpu‚99.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚39.0 gpu.0.powerWatts‚113.907' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.781399999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚03pƒ²ìñ‚(:á ìÜèÒøˆ‰ø% disk.nvme0c0n1.out‚58434.42578125 disk.drbd2165.out‚0 disk.rbd1.out‚ 12.4140625 disk.sda.in‚0 disk.dm-1.in‚0 cpu‚25.579742980408277$ disk./.usageGB‚557.5171661376953 disk.sda2.in‚0 disk.rbd1.in‚0# proc.memory.rssMB‚ 2826.47265625 proc.cpu.threads‚17 network.sent‚ 5.851536e+06# disk.nvme0n1.out‚58434.42578125 network.recv‚ 1.334001e+06 disk.rbd0.in‚0 disk.dm-0.out‚58434.42578125! disk.nvme0n1.in‚ 11774.3671875$ memory_percent‚90.56637287139893) proc.memory.availableMB‚ 1545.60546875) proc.memory.percent‚17.25142002105713) disk./.usagePercent‚37.41544480756098 disk.dm-0.in‚ 11774.3671875 disk.sda1.out‚0 disk.rbd0.out‚0 disk.sda1.in‚0 disk.sda3.out‚ 210.03125 disk.rbd3.out‚0# disk.nvme0c0n1.in‚ 11774.3671875 disk.sda2.out‚0 disk.sda.out‚ 210.03125 disk.dm-1.out‚ 210.03125 disk.drbd2165.in‚0 disk.sda3.in‚0 disk.rbd3.in‚0æ]Ó”ò‚(:‰ ûÜèÒïå¾ö gpu.0.gpu‚99.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚274.718' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.9436 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ï9FAùó‚(:î ûÜèÒÛé„ù network.sent‚ 5.857137e+06) disk./.usagePercent‚37.41582267468152 disk.drbd2165.out‚0 disk.rbd3.out‚0 disk.sda2.in‚0 disk.rbd1.out‚ 12.5390625 disk.rbd0.in‚0) proc.memory.percent‚17.25142002105713$ disk./.usageGB‚557.5227966308594 disk.sda3.in‚0 disk.sda.in‚0 disk.dm-0.in‚ 11774.3671875 proc.cpu.threads‚17 disk.sda.out‚ 210.33203125 disk.rbd0.out‚0 disk.sda1.out‚0$ proc.memory.availableMB‚1545.125 cpu‚25.579218970525424 disk.rbd1.in‚0 disk.dm-1.in‚0# disk.dm-0.out‚58444.27197265625 disk.rbd3.in‚0& disk.nvme0n1.out‚58444.27197265625 disk.dm-1.out‚ 210.33203125$ memory_percent‚90.56930541992188 disk.sda3.out‚ 210.33203125# disk.nvme0c0n1.in‚ 11774.3671875 disk.sda1.in‚0# proc.memory.rssMB‚ 2826.47265625 disk.drbd2165.in‚0( disk.nvme0c0n1.out‚58444.27197265625 disk.sda2.out‚0 network.recv‚ 1.337531e+06! disk.nvme0n1.in‚ 11774.3671875›ø3š2ô- _wandbruntime‚5222  _runtime‚5222‰8ÿÆõÀ $ _timestamp‚1.784295037986224e+09 _runtime‚5222.203348309 _step‚255 & lr-AdamW/pg1‚0.0006588076831414599 & lr-AdamW/pg2‚0.0006588076831414599 trainer/global_step‚3149`³ÒËöÅ _runtime‚5222.203348309 _step‚255 & lr-AdamW/pg1‚0.0006588076831414599 & lr-AdamW/pg2‚0.0006588076831414599 trainer/global_step‚3149 $ _timestamp‚1.784295037986224e+09ÿïªÐ2÷- _wandbruntime‚5222  _runtime‚5222|D&§î'øèO JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚905543743009280 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚902585645965312 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚902585645965312 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2959442433536 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚17009308 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚54018841 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚248426752259584 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚27241528 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚905148489012004 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2954381425702 _step‚256 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚54008862 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚27231555 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 $ _timestamp‚1.784295037987244e+09 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3816030979584 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚905545114572288 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0x¤Æ1ó'ùíO UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚905148489012004 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2954381425702 _step‚256 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚54008862 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚27231555 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 $ _timestamp‚1.784295037987244e+09 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3816030979584 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚905545114572288 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚905543743009280 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚902585645965312 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚902585645965312 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2959442433536 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚17009308 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚54018841 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚248426752259584 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚27241528€½p†¼2ú- _wandbruntime‚5223  _runtime‚5223ló! ûóM H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚5223.598910193 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚54035539 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 % _timestamp‚1.7842950393814042e+09 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2959022079488 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚27240033 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚238 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚17014552 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚26785519 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2960367476224 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚902863646695424 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17014309 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8079618 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚27250020 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚905428664284828 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚248505432768000 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚905427294256426 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2960367476224 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚902863672868864 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8934929 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚54035539 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚244688207687680Žó%Íþ&üøM N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚902863646695424 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17014309 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8079618 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚27250020 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚905428664284828 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚248505432768000 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚905427294256426 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2960367476224 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚902863672868864 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8934929 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚54035539 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚244688207687680 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚5223.598910193 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚54035539 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 % _timestamp‚1.7842950393814042e+09 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2959022079488 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚27240033 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚238 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚17014552 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚26785519 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2960367476224 EDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚919490095445504 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚252360245250560 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚54844914 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36633600 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚27657307 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404724736 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚248485409755136 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9068912 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚919491467008512 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚27187607 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚276 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8204225 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3875240220160 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚919088815033912 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚916485708644352 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚27187607 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚919491467008512 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚248485041664000 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 trainer/global_step‚3199 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚17272866 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3875203586560 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚916485682470912 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚27187613 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 _runtime‚5301.038658463 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3005758364160 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3004412974592 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 % _timestamp‚1.7842951168213565e+09 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚54854893 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9068641 YJDeviceStatsMonitor.on_train_batch_start/requesteY­¥Šƒd_bytes.small_pool.current‚ 1344600322 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚27667280 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3004412974592 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3005758364160 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8204230 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2999273938488 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚916089541095424 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚17273142 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚27667280 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚919090185062202 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚916089566523392 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚27657307 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚252360649975296 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3000618538810 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚54854893 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9068641 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚27667280 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3004412974592 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3005758364160 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8204230 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2999273938488 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚916089541095424 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚17273142 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚27667280 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚919090185062202 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚916089566523392 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚27657307 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚252360649975296 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3000618538810 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚919490095445504 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚252360245250560 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚54844914 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36633600 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚27657307 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404724736 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚248485409755136 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9068912 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚919491467008512 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚27187607 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚276 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8204225 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3875240220160 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚919088815033912 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚916485708644352 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚27187607 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚919491467008512 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚248485041664000 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 trainer/global_step‚3199 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚17272866 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3875203586560 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚916485682470912 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚27187613 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 _runtime‚5301.038658463„gÁ­2- _wandbruntime‚5302  _runtime‚5302ÆV¦€ø&‘òM NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚27195819 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚916367448301568 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚27675772 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚252438915420672 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404717568 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8206768 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚919770392781312 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚919368990306736 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7842951182075315e+09 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3005338010112 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8206773 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚916763709374464 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9071635 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚916763683201024 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚919367620278334 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚916367422873600 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚54871591 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9071371 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚264 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚54871591 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚248562896162816 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚916763709374464 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3876387348992 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3005338010112 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚27195813 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 _runtime‚ 5302.42491145 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚54861598 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚27195819 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚269 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3001542005168 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚27195813 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚919770392781312 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚916763683201024 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚252439320138240 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚248562528071680 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17278139 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚919769021211136 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚27675772 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _step‚261 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚27665785 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36626432 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚54861598 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚248562528071680 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17278139 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚919769021211136 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚27675772 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _step‚261 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚27665785 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36626432 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚54861598 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7842951182075315e+09 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3005338010112 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8206773 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚916763709374464 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9071635 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚916763683201024 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚919367620278334 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚916367422873600 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚54871591 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9071371 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚264 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_en$ZÑYÁd/active.all.allocated‚54871591 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚248562896162816 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚916763709374464 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3876387348992 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3005338010112 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚27195813 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 _runtime‚ 5302.42491145 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚54861598…¶4N2“- _wandbruntime‚5302  _runtime‚5302MžŽN”þ /loss/FRt1/assignment_loss‚1.802340090871454 2loss/FBt2/assignment_loss‚0.016629295093608432 1loss/SRqqt2/detection_loss‚0.5947722648892352 $loss/total_loss‚8.50900749233729 _runtime‚5302.511391734 0loss/FRt2/assignment_loss‚0.6944851599880869 2loss/SRqqt1/assignment_loss‚0.6479983876064102 1loss/FBt1/assignment_loss‚0.01565584851787149 /loss/FRt1/detection_loss‚0.5542864623492495 1loss/SRqqt1/detection_loss‚1.0190033700317145 /loss/FBt2/detection_loss‚0.5536953490435966 epoch‚7 /loss/FRt2/detection_loss‚1.4413634365782035 -loss/FBt1/detection_loss‚0.58458590029906 trainer/global_step‚3199 _step‚262 2loss/SRqqt2/assignment_loss‚0.5841919270688002 % _timestamp‚1.7842951182944293e+09Qÿ}u •ƒ /loss/FRt1/assignment_loss‚1.802340090871454 2loss/FBt2/assignment_loss‚0.016629295093608432 1loss/SRqqt2/detection_loss‚0.5947722648892352 $loss/total_loss‚8.50900749233729 _runtime‚5302.511391734 0loss/FRt2/assignment_loss‚0.6944851599880869 2loss/SRqqt1/assignment_loss‚0.6479983876064102 1loss/FBt1/assignment_loss‚0.01565584851787149 /loss/FRt1/detection_loss‚0.5542864623492495 1loss/SRqqt1/detection_loss‚1.0190033700317145 /loss/FBt2/detection_loss‚0.5536953490435966 epoch‚7 /loss/FRt2/detection_loss‚1.4413634365782035 -loss/FBt1/detection_loss‚0.58458590029906 trainer/global_step‚3199 _step‚262 2loss/SRqqt2/assignment_loss‚0.5841919270688002 % _timestamp‚1.7842951182944293e+09†–j-9•–‚(:Š ÕÝèÒëÁÝö gpu.0.gpu‚100.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚334.442' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚66.8884 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0«ŒÿPø—‚(:í ÕÝèÒâôÏø disk.rbd0.in‚0# disk.nvme0c0n1.in‚ 11869.3828125' disk.nvme0c0n1.out‚58503.2470703125% disk.nvme0n1.out‚58503.2470703125 disk.sda2.out‚0 network.sent‚ 5.9443e+06 network.recv‚ 1.358785e+06 disk.dm-1.in‚0 disk.rbd1.out‚ 12.7265625# proc.memory.rssMB‚ 2848.62890625 disk.sda.out‚ 211.6796875 disk.dm-0.in‚ 11869.3828125) proc.memory.availableMB‚ 1508.13671875 disk.sda1.in‚0 disk.sda1.out‚0* disk./.usagePercent‚37.417992850549275 disk.dm-1.out‚ 211.6796875" disk.dm-0.out‚58503.2470703125 disk.drbd2165.out‚0 disk.rbd3.out‚0$ memory_percent‚90.79506397247314* proc.memory.percent‚17.386651039123535 proc.cpu.threads‚17 disk.sda3.out‚ 211.6796875 disk.drbd2165.in‚0! disk.nvme0n1.in‚ 11869.3828125 cpu‚25.579198828817482 disk.rbd0.out‚0 disk.sda3.in‚0$ disk./.usageGB‚557.5551338195801 disk.rbd3.in‚0 disk.sda2.in‚0 disk.rbd1.in‚0 disk.sda.in‚0Í0Úcž˜‚(:“ äÝèÒ“…Ïö gpu.0.gpu‚98.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚333.764' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚66.75280000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚07‹»™‚(:õ äÝèÒÏÕãù disk.rbd1.out‚ 12.7265625# disk.dm-0.out‚58514.35302734375 disk.sda1.in‚0 disk.rbd3.in‚0& disk.nvme0n1.out‚58514.35302734375# proc.memory.rssMB‚ 2804.54296875 cpu‚25.586584523246362) disk./.usagePercent‚37.41830466772598 disk.sda.out‚ 212.07421875 disk.dm-1.in‚0 disk.sda1.out‚0$ disk.nvme0c0n1.in‚11950.11328125( disk.nvme0c0n1.out‚58514.35302734375 disk.sda2.out‚0 disk.rbd1.in‚0$ disk./.usageGB‚557.5597801208496) proc.memory.availableMB‚ 1551.95703125 disk.drbd2165.in‚0 network.sent‚ 6.00778e+06 network.recv‚ 1.362998e+06 disk.sda.in‚0 disk.sda2.in‚0 proc.cpu.threads‚17 disk.dm-0.in‚11950.11328125 disk.rbd3.out‚0" disk.nvme0n1.in‚11950.11328125$ memory_percent‚90.52760601043701 disk.rbd0.out‚0 disk.sda3.in‚0 disk.dm-1.out‚ 212.07421875 disk.rbd0.in‚0 disk.sda3.out‚ 212.07421875 disk.drbd2165.out‚0) proc.memory.percent‚17.11757183074951ÁÞ¹2š- _wandbruntime‚5328  _runtime‚5328B¬=›¦{ 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9917945861816406 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.32362937927246094 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9888414740562439 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.31245407462120056 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5511826276779175 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.27736037969589233 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8450490832328796 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7675358057022095 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.10250352323055267 jet/accuracy_4_of_5‚0 8"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.583917498588562 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.22780254483222961 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.04584156349301338 *jet/accuracy_1_of_1‚0.3152037262916565 *jet/accuracy_2_of_3‚0.4989437758922577 particle/accuracy_1_of_3‚1 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.25915148854255676 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8540437817573547 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9962303638458252 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.11904759705066681 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9221678972244263 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.2486414611339569 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5516781210899353 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 +jet/accuracy_2_of_2‚0.12819913029670715 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5618631839752197 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 *jet/accuracy_3_of_4‚0.0851360484957695 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 particle/accuracy_1_of_5‚1 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.24318218231201172 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2585538327693939 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 * Purity/2FRt*SRqqt2FBt/FRt_purity‚0.875 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 6 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.989953875541687 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 particle/accuracy_2_of_3‚1 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.32165491580963135 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9907755255699158 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 *jet/accuracy_1_of_3‚0.9164043664932251 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 9"Purity/*FRt2SRqqt0FBt/event_purity‚0.1310705989599228 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.03906332701444626 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6407055854797363 *particle/sensitivity‚0.650070309638977 +"Purity/2FRt*SRqqt2FBt/event_purity‚0.75 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt2SRqqt*FBt/event_purity‚0.1287681609392166 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.6248905658721924 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9932422637939453 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.6917309761047363 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 ( Purity/2FRt1SRqqt1FBt/FRt_purity‚0.5 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.27459731698036194 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.31755614280700684 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6304114460945129 /particle/accuracy_3_of_4‚0.9977324604988098 jet/accuracy_3_of_5‚1 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.5954941511154175 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.26365503668785095 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.5485102534294128 8"Purity/0FRt1SRqqt*FBt/event_purity‚0.541951060295105 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.6584806442260742 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7824074029922485 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 particle/accuracy_3_of_3‚1 'particle/f_score‚0.6464165449142456 6validation_average_jet_accuracy‚0.3553381562232971 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.1538461595773697 jet/accuracy_2_of_5‚1 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5202104449272156 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 (particle/accuracy‚0.8472843170166016 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.1538461595773697 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6297965049743652 +jet/accuracy_4_of_4‚0.01628045178949833 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.20024651288986206 particle/accuracy_1_of_1‚1 trainer/global_step‚3199 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 /particle/accuracy_4_of_4‚0.9665483832359314 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7091474533081055 8"Purity/*FRt0SRqqt1FBt/event_purity‚0.871862530708313 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6399733424186707 particle/accuracy_2_of_5‚1 epoch‚7 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5461381077766418 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6392346620559692 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.08692508935928345 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.3311716616153717 *jet/accuracy_3_of_3‚0.0649457573890686 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.27561891078948975 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8578987121582031 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6425414085388184 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_4_of_5‚1 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.10185185074806213 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.37731483578681946 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5461381077766418 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6624951362609863 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.7599063515663147 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.7599063515663147 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.21672314405441284 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_2_of_4‚1 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9230769276618958 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.24785397946834564 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8805978894233704 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.46733301877975464 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7754346132278442 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.11964596807956696 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5570833086967468 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.10185185074806213 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2593758702278137 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7384882569313049 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8504912257194519 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.2552083134651184 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.3844505548477173 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9490740299224854 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 )loss/val_total_loss‚8.570087227488525 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9864411354064941 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3261294364929199 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.2508465349674225 8"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.637895941734314 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.32499998807907104 *jet/accuracy_1_of_4‚0.9543108940124512 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.5555911660194397 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5582782626152039 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.23164281249046326 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 6 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.242638498544693 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9970898628234863 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9799078106880188 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8450490832328796 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4502028524875641 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.07394292950630188 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.25785526633262634 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9946191906929016 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6508790254592896 *jet/accuracy_1_of_2‚0.7680672407150269 jet/accuracy_1_of_5‚1 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.11438649892807007 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9978564977645874 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4140625 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 particle/accuracy_1_of_2‚1 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.20801803469657898 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.2660827934741974 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.10250352323055267 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.04312572255730629 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 particle/accuracy_2_of_2‚1 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 particle/accuracy_3_of_5‚1 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9424775242805481 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.6541901230812073 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.2355712652206421 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.8571428656578064 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9970898628234863 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 % _timestamp‚1.7842951447583234e+09 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.11666665971279144 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.28781184554100037 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9857678413391113 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 particle/accuracy_1_of_4‚1 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6119791269302368 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9956039190292358 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5887719988822937 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9973999857902527 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 +particle/specificity‚0.9012231230735779 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6658190488815308 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6344972848892212 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8447314500808716 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2552650272846222 *jet/accuracy_2_of_4‚0.7480053901672363 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9285714030265808 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.033914048224687576 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.6937228441238403 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.4877724051475525 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9939466118812561 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9935346245765686 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 particle/accuracy_5_of_5‚0 _runtime‚5328.976143859 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9962709546089172 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.5817515254020691 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.5929579734802246 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.6202080845832825 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.4080488681793213 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.5246586799621582 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6984246373176575 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.5601385831832886 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9946585297584534 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.27752208709716797 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.11465616524219513 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.2956721782684326 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.2660827934741974 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7824074029922485 jet/accuracy_5_of_5‚0 _step‚263 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612V»ajª(œ«{ particle/accuracy_1_of_3‚1 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.25915148854255676 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8540437817573547 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9962303638458252 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.11904759705066681 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9221678972244263 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.2486414611339569 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5516781210899353 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 +jet/accuracy_2_of_2‚0.12819913029670715 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5618631839752197 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 *jet/accuracy_3_of_4‚0.0851360484957695 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 particle/accuracy_1_of_5‚1 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.24318218231201172 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2585538327693939 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 * Purity/2FRt*SRqqt2FBt/FRt_purity‚0.875 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 6 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.989953875541687 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 particle/accuracy_2_of_3‚1 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.32165491580963135 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9907755255699158 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 *jet/accuracy_1_of_3‚0.9164043664932251 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 9"Purity/*FRt2SRqqt0FBt/event_purity‚0.1310705989599228 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.03906332701444626 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6407055854797363 *particle/sensitivity‚0.650070309638977 +"Purity/2FRt*SRqqt2FBt/event_purity‚0.75 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt2SRqqt*FBt/event_purity‚0.1287681609392166 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.6248905658721924 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9932422637939453 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.6917309761047363 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 ( Purity/2FRt1SRqqt1FBt/FRt_purity‚0.5 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.27459731698036194 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.31755614280700684 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6304114460945129 /particle/accuracy_3_of_4‚0.9977324604988098 jet/accuracy_3_of_5‚1 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.5954941511154175 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.26365503668785095 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.5485102534294128 8"Purity/0FRt1SRqqt*FBt/event_purity‚0.541951060295105 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.6584806442260742 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7824074029922485 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 particle/accuracy_3_of_3‚1 'particle/f_score‚0.6464165449142456 6validation_average_jet_accuracy‚0.3553381562232971 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.1538461595773697 jet/accuracy_2_of_5‚1 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5202104449272156 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 (particle/accuracy‚0.8472843170166016 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.1538461595773697 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6297965049743652 +jet/accuracy_4_of_4‚0.01628045178949833 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.20024651288986206 particle/accuracy_1_of_1‚1 trainer/global_step‚3199 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 /particle/accuracy_4_of_4‚0.9665483832359314 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7091474533081055 8"Purity/*FRt0SRqqt1FBt/event_purity‚0.871862530708313 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6399733424186707 particle/accuracy_2_of_5‚1 epoch‚7 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5461381077766418 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6392346620559692 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.08692508935928345 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.3311716616153717 *jet/accuracy_3_of_3‚0.0649457573890686 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.27561891078948975 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8578987121582031 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6425414085388184 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_4_of_5‚1 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.10185185074806213 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.37731483578681946 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5461381077766418 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6624951362609863 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.7599063515663147 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.7599063515663147 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.21672314405441284 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_2_of_4‚1 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9230769276618958 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.24785397946834564 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8805978894233704 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.46733301877975464 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7754346132278442 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.11964596807956696 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5570833086967468 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.10185185074806213 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2593758702278137 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7384882569313049 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8504912257194519 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.2552083134651184 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.3844505548477173 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9490740299224854 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 )loss/val_total_loss‚8.570087227488525 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9864411354064941 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3261294364929199 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.2508465349674225 8"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.637895941734314 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.32499998807907104 *jet/accuracy_1_of_4‚0.9543108940124512 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.5555911660194397 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5582782626152039 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.23164281249046326 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 6 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.242638498544693 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9970898628234863 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9799078106880188 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8450490832328796 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4502028524875641 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.07394292950630188 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.25785526633262634 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9946191906929016 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6508790254592896 *jet/accuracy_1_of_2‚0.7680672407150269 jet/accuracy_1_of_5‚1 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.11438649892807007 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9978564977645874 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4140625 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 particle/accuracy_1_of_2‚1 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.20801803469657898 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.2660827934741974 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.10250352323055267 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.04312572255730629 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 particle/accuracy_2_of_2‚1 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.0001860119082266464SßJé8 particle/accuracy_3_of_5‚1 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9424775242805481 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.6541901230812073 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.2355712652206421 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.8571428656578064 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9970898628234863 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 % _timestamp‚1.7842951447583234e+09 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.11666665971279144 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.28781184554100037 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9857678413391113 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 particle/accuracy_1_of_4‚1 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6119791269302368 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9956039190292358 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5887719988822937 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9973999857902527 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 +particle/specificity‚0.9012231230735779 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6658190488815308 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6344972848892212 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8447314500808716 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2552650272846222 *jet/accuracy_2_of_4‚0.7480053901672363 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9285714030265808 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.033914048224687576 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.6937228441238403 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.4877724051475525 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9939466118812561 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9935346245765686 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 particle/accuracy_5_of_5‚0 _runtime‚5328.976143859 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9962709546089172 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.5817515254020691 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.5929579734802246 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.6202080845832825 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.4080488681793213 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.5246586799621582 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6984246373176575 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.5601385831832886 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9946585297584534 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.27752208709716797 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.11465616524219513 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.2956721782684326 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.2660827934741974 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7824074029922485 jet/accuracy_5_of_5‚0 _step‚263 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9917945861816406 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.32362937927246094 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9888414740562439 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.31245407462120056 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5511826276779175 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.27736037969589233 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8450490832328796 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7675358057022095 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.10250352323055267 jet/accuracy_4_of_5‚0 8"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.583917498588562 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.22780254483222961 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.04584156349301338 *jet/accuracy_1_of_1‚0.3152037262916565 *jet/accuracy_2_of_3‚0.4989437758922577‡ß l@Ÿ‚(:” óÝèÒ–„×ö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚40.0 gpu.0.powerWatts‚298.104' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚59.620799999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0{êˆäÿž‚(:ô óÝèÒ´Ïšú network.sent‚ 6.01279e+06* disk./.usagePercent‚37.418918317785554# memory_percent‚86.1722469329834( proc.memory.availableMB‚ 2265.5390625 network.recv‚ 1.366785e+06$ disk./.usageGB‚557.5689239501953# disk.dm-0.out‚59492.84423828125 disk.sda1.out‚0 disk.sda2.in‚0 cpu‚25.586291512372554& disk.nvme0n1.out‚59492.84423828125 disk.rbd0.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 212.43359375 disk.rbd3.in‚0( disk.nvme0c0n1.out‚59492.84423828125 disk.sda2.out‚0# proc.memory.rssMB‚ 2825.52734375 proc.cpu.threads‚17 disk.sda.out‚ 212.43359375" disk.nvme0n1.in‚11950.11328125 disk.rbd1.in‚0 disk.rbd0.out‚0 disk.sda1.in‚0 disk.drbd2165.out‚0) proc.memory.percent‚17.24565029144287 disk.sda.in‚0 disk.dm-0.in‚11950.11328125 disk.sda3.in‚0 disk.sda3.out‚ 212.43359375 disk.drbd2165.in‚0 disk.rbd3.out‚0$ disk.nvme0c0n1.in‚11950.11328125 disk.rbd1.out‚ 12.7890625½¿žŸ‚(:“ ‚ÞèÒäÔûö gpu.0.gpu‚90.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚279.79' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚55.958000000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0þ˜ô ‚(:é ‚ÞèÒÒÁù) disk./.usagePercent‚37.41921503051774$ disk./.usageGB‚557.5733451843262 disk.dm-0.in‚11950.11328125 disk.rbd1.in‚0 disk.sda.in‚0) proc.memory.percent‚17.24565029144287 disk.sda1.in‚0 disk.sda3.in‚0 disk.drbd2165.out‚0 disk.dm-1.in‚0 disk.rbd0.in‚0 cpu‚25.585737207918175 disk.drbd2165.in‚0 disk.rbd3.in‚0 disk.sda2.out‚0 disk.dm-1.out‚ 212.6171875 network.recv‚ 1.371315e+06 disk.sda1.out‚0 disk.rbd3.out‚0 disk.rbd0.out‚0# proc.memory.rssMB‚ 2825.52734375 network.sent‚ 6.070501e+06$ disk.nvme0c0n1.in‚11950.11328125 disk.rbd1.out‚ 12.7890625% proc.memory.availableMB‚ 2264.8125 proc.cpu.threads‚17! disk.dm-0.out‚59502.900390625 disk.sda2.in‚0 disk.sda.out‚ 212.6171875$ memory_percent‚86.17668151855469" disk.nvme0n1.in‚11950.11328125 disk.sda3.out‚ 212.6171875& disk.nvme0c0n1.out‚59502.900390625$ disk.nvme0n1.out‚59502.900390625f} £”¡‚(:‰ ‘ÞèÒÿÄŠ÷ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚39.0 gpu.0.powerWatts‚116.331' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.2662 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0§œ¶¢‚(:õ ‘ÞèÒêì˜ù disk.sda1.out‚0 disk.rbd3.in‚0) proc.memory.availableMB‚ 2264.98828125( disk.nvme0c0n1.out‚60479.11572265625 disk.sda2.in‚0 disk.rbd1.out‚ 12.9140625 disk.sda.in‚0 disk.dm-1.out‚ 212.81640625 disk.sda1.in‚0 disk.sda.out‚ 212.81640625# proc.memory.rssMB‚ 2825.52734375 network.recv‚ 1.37485e+06 disk.rbd0.in‚0 disk.sda3.in‚0$ memory_percent‚86.17560863494873$ disk./.usageGB‚557.5769538879395$ disk.nvme0c0n1.in‚11950.11328125" disk.nvme0n1.in‚11950.11328125& disk.nvme0n1.out‚60479.11572265625 disk.dm-0.in‚11950.11328125# disk.dm-0.out‚60479.11572265625 disk.drbd2165.out‚0 disk.rbd3.out‚0) proc.memory.percent‚17.24565029144287) disk./.usagePercent‚37.41945721364513 disk.dm-1.in‚0 disk.sda3.out‚ 212.81640625 disk.drbd2165.in‚0 proc.cpu.threads‚17 disk.sda2.out‚0 disk.rbd1.in‚0 disk.rbd0.out‚0 cpu‚25.585172425590795 network.sent‚ 6.075513e+06ô{+1”£‚(:‰  ÞèÒ÷Éìö gpu.0.gpu‚91.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚285.532' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚57.1064 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0X¸½5ó¤‚(:è  ÞèÒ¸Žù disk.rbd0.out‚0* disk./.usagePercent‚37.419774406980046 disk.rbd3.in‚0 disk.sda2.in‚0 disk.dm-1.out‚213.0625$ disk./.usageGB‚557.5816802978516 disk.sda3.in‚0( proc.memory.availableMB‚ 2265.0390625) proc.memory.percent‚17.24565029144287 disk.sda1.out‚0$ disk.nvme0c0n1.in‚12084.92578125" disk.nvme0n1.in‚12084.92578125 disk.sda2.out‚0# memory_percent‚86.1752986907959 disk.rbd3.out‚0& disk.nvme0n1.out‚60491.73583984375 cpu‚25.584660663248474 proc.cpu.threads‚17 network.recv‚ 1.37821e+06 disk.drbd2165.out‚0 disk.dm-1.in‚0# proc.memory.rssMB‚ 2825.52734375 disk.dm-0.in‚12084.92578125 disk.sda1.in‚0( disk.nvme0c0n1.out‚60491.73583984375 disk.rbd1.out‚ 12.9140625 disk.sda.out‚213.0625 network.sent‚ 6.080504e+06# disk.dm-0.out‚60491.73583984375 disk.sda3.out‚213.0625 disk.drbd2165.in‚0 disk.rbd1.in‚0 disk.sda.in‚0 disk.rbd0.in‚0¾/ZŸ¥‚(:” ¯ÞèÒëæ×ö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚41.0 gpu.0.powerWatts‚292.642' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚58.528400000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÙÞ¨¦‚(:ö ¯ÞèÒÈàìø) proc.memory.availableMB‚ 2265.03515625 disk.sda1.out‚0$ disk.nvme0c0n1.in‚12084.92578125 disk.rbd3.in‚0 disk.sda.out‚ 213.30859375 disk.dm-1.in‚0 disk.dm-1.out‚ 213.30859375 disk.rbd0.out‚0 disk.drbd2165.in‚0 disk.rbd1.out‚ 12.9765625 disk.rbd0.in‚0 network.sent‚ 6.08555e+06" disk.nvme0n1.in‚12084.92578125 disk.sda1.in‚0# proc.memory.rssMB‚ 2825.52734375 disk.rbd3.out‚0( disk.nvme0c0n1.out‚60501.26416015625$ disk./.usageGB‚557.5872497558594 disk.sda2.out‚0 disk.dm-0.in‚12084.92578125) proc.memory.percent‚17.24565029144287 cpu‚25.584151492105942 proc.cpu.threads‚17& disk.nvme0n1.out‚60501.26416015625 disk.sda2.in‚0 disk.sda.in‚0# disk.dm-0.out‚60501.26416015625 disk.sda3.out‚ 213.30859375 disk.drbd2165.out‚0$ memory_percent‚86.17532253265381 network.recv‚ 1.381942e+06* disk./.usagePercent‚37.420148177980046 disk.rbd1.in‚0 disk.sda3.in‚0@ø!Š2§- _wandbruntime‚5412  _runtime‚5412``'ÒǨÁ % _timestamp‚1.7842952282528946e+09 _runtime‚5412.470034666 _step‚264 & lr-AdamW/pg1‚0.0006564027726009225 & lr-AdamW/pg2‚0.0006564027726009225 trainer/global_step‚32499!Ù'Ì©Æ _runtime‚5412.470034666 _step‚264 & lr-AdamW/pg1‚0.0006564027726009225 & lr-AdamW/pg2‚0.0006564027726009225 trainer/global_step‚3249 % _timestamp‚1.7842952282528946e+09ˆ mô2ª- _wandbruntime‚5412  _runtime‚5412ëyÆÞî'«èO M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚55881938 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚27729913 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚17582013 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8340293 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚933809538936832 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚55871959 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚32439296 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚28152019 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3942526495232 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚934213700411392 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 _runtime‚5412.474917562 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚27729919 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚934213726584832 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚17581770 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9241477 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚256350368881152 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚27729913 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚934213726584832 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3942494055936 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚243 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3057916909056 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚238 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚937270271930880 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚28142046 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚937271643493888 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _step‚265 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚27729919 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚934213726584832 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚17581770 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9241477 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚256350368881152 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚27729913 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚934213726584832 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3942494055936 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚243 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3057916909056 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚238 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚937270271930880 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚28142046 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚937271643493888 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 GnK÷,AADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _step‚265 IDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3052263649284 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3058841951744 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚252484960702464 _runtime‚5413.855448478 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17587078 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚55898636 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚55888643 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚252485328793600 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3943730918912 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚28160511 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚237 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚232 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚27738119 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9244474 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚256428659189248 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3943698486784 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 % _timestamp‚1.7842952296380553e+09 _step‚266 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚27738119 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚937550569266688 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚55898636 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 trainer/global_step‚3249 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3057496555008 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 400523264 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8342836 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚28150524 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚32432128 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚937141054392694 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3053608249718 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚55888643 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚27738125 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚256429059712512 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736ímúþ&¯øM N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3052263649284 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3058841951744 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚252484960702464 _runtime‚5413.855448478 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17587078 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚55898636 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚55888643 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚252485328793600 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3943730918912 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚28160511 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚237 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚232 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚27738119 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9244474 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚256428659189248 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3943698486784 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 % _timestamp‚1.7842952296380553e+09 _step‚266 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚27738119 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚937550569266688 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚55898636 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 trainer/global_step‚3249 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3057496555008 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 400523264 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8342836 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚28150524 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚32432128 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚937141054392694 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3053608249718 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚55888643 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚27738125 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚256429059712512 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚937549197696512 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192Šî”2°- _wandbruntime‚5413  _runtime‚5413´3m ±þ 1loss/SRqqt1/detection_loss‚1.0337598496859774 /loss/FBt1/detection_loss‚0.7698916433214703 _step‚267 2loss/SRqqt1/assignment_loss‚0.5863977866767965 2loss/SRqqt2/assignment_loss‚0.5337494903729684 0loss/FBt2/assignment_loss‚0.0701121871550755 /loss/FRt2/detection_loss‚1.5217961859913967 $loss/total_loss‚8.74834181518187 $ _timestamp‚1.784295229724809e+09 _runtime‚5413.941729642 1loss/FBt1/assignment_loss‚0.01286214779820708 /loss/FRt1/detection_loss‚0.5416370989841901 1loss/SRqqt2/detection_loss‚0.6726469567765528 /loss/FBt2/detection_loss‚0.4066688248722424 0loss/FRt2/assignment_loss‚0.7876972688568963 epoch‚8 trainer/global_step‚3249 0loss/FRt1/assignment_loss‚1.8111223746900973Ó:9 ²ƒ _step‚267 2loss/SRqqt1/assignment_loss‚0.5863977866767965 2loss/SRqqt2/assignment_loss‚0.5337494903729684 0loss/FBt2/assignment_loss‚0.0701121871550755 /loss/FRt2/detection_loss‚1.5217961859913967 $loss/total_loss‚8.74834181518187 $ _timestamp‚1.784295229724809e+09 _runtime‚5413.941729642 1loss/FBt1/assignment_loss‚0.01286214779820708 /loss/FRt1/detection_loss‚0.5416370989841901 1loss/SRqqt2/detection_loss‚0.6726469567765528 /loss/FBt2/detection_loss‚0.4066688248722424 0loss/FRt2/assignment_loss‚0.7876972688568963 epoch‚8 trainer/global_step‚3249 0loss/FRt1/assignment_loss‚1.8111223746900973 1loss/SRqqt1/detection_loss‚1.0337598496859774 /loss/FBt1/detection_loss‚0.7698916433214703‹ÎÖÀ¦”³‚(:‰ ¾ÞèÒß—œ÷ gpu.0.gpu‚89.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚278.198' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.6396 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0å@1X÷´‚(:ì ¾ÞèÒ©‰àø disk.drbd2165.out‚0" disk.dm-0.out‚60512.8447265625) proc.memory.percent‚17.24565029144287 disk.sda1.out‚0 disk.sda.out‚ 213.546875 disk.dm-1.in‚0 disk.rbd0.out‚0 disk.dm-0.in‚12084.92578125# proc.memory.rssMB‚ 2825.52734375 cpu‚25.583552729253178 disk.sda3.in‚0 disk.sda2.in‚0 disk.rbd1.out‚ 12.9765625 proc.cpu.threads‚17$ disk./.usageGB‚557.5919876098633 disk.sda3.out‚ 213.546875 network.recv‚ 1.385215e+06 disk.drbd2165.in‚0' disk.nvme0c0n1.out‚60512.8447265625 disk.rbd1.in‚0 disk.dm-1.out‚ 213.546875# memory_percent‚86.1767053604126) proc.memory.availableMB‚ 2264.80859375 network.sent‚ 6.090552e+06 disk.rbd3.in‚0$ disk.nvme0c0n1.in‚12084.92578125 disk.sda.in‚0 disk.rbd0.in‚0) disk./.usagePercent‚37.42046613933757 disk.sda2.out‚0 disk.sda1.in‚0 disk.rbd3.out‚0" disk.nvme0n1.in‚12084.92578125% disk.nvme0n1.out‚60512.8447265625cä Ÿµ‚(:” ÍÞèÒÓ“©÷ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚274.182' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚54.836400000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¸<¢÷¶‚(:ì ÍÞèÒéÌÞø! disk.dm-0.out‚60523.607421875 disk.rbd1.in‚0 disk.sda1.in‚0 disk.drbd2165.out‚0 proc.cpu.threads‚17 disk.sda3.in‚0 disk.drbd2165.in‚0 disk.sda.in‚0 disk.dm-1.in‚0 disk.rbd0.out‚0# proc.memory.rssMB‚ 2825.52734375 disk.rbd3.out‚0 cpu‚25.58304781280589$ disk./.usageGB‚557.5974998474121 disk.sda2.out‚0$ memory_percent‚86.17672920227051( proc.memory.availableMB‚ 2264.8046875) proc.memory.percent‚17.24565029144287* disk./.usagePercent‚37.420836070224546& disk.nvme0c0n1.out‚60523.607421875$ disk.nvme0n1.out‚60523.607421875 disk.dm-0.in‚12084.92578125 disk.sda1.out‚0 disk.rbd3.in‚0 disk.sda3.out‚ 213.8203125 network.sent‚ 6.153435e+06 network.recv‚ 1.389907e+06" disk.nvme0n1.in‚12084.92578125 disk.sda.out‚ 213.8203125 disk.dm-1.out‚ 213.8203125$ disk.nvme0c0n1.in‚12084.92578125 disk.sda2.in‚0 disk.rbd1.out‚ 13.0390625 disk.rbd0.in‚0ê‹pØž·‚(:“ ÜÞèÒÜÐéö gpu.0.gpu‚90.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚279.255' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚55.85099999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Œºú¸‚(:ï ÜÞèÒÿ²Üø network.sent‚ 6.158685e+06) disk./.usagePercent‚37.42111179033893# disk.dm-0.out‚60534.35888671875 disk.nvme0n1.in‚ 12124.703125 cpu‚25.582500187803447 disk.rbd1.in‚0 disk.rbd1.out‚ 13.0390625 disk.dm-1.out‚ 213.98828125 disk.sda3.in‚0 disk.rbd3.in‚0 disk.sda.in‚0) proc.memory.percent‚17.24565029144287 disk.sda1.out‚0 disk.drbd2165.out‚0 disk.rbd3.out‚0 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.sda3.out‚ 213.98828125" disk.nvme0c0n1.in‚ 12124.703125 proc.cpu.threads‚17$ disk./.usageGB‚557.6016082763672 disk.sda2.in‚0) proc.memory.availableMB‚ 2264.99609375 disk.dm-0.in‚ 12124.703125 disk.sda2.out‚0 disk.sda.out‚ 213.98828125$ memory_percent‚86.17556095123291# proc.memory.rssMB‚ 2825.52734375 disk.rbd0.out‚0 network.recv‚ 1.39329e+06 disk.sda1.in‚0 disk.drbd2165.in‚0( disk.nvme0c0n1.out‚60534.35888671875& disk.nvme0n1.out‚60534.35888671875³ÂG±”¹‚(:‰ ëÞèÒ™éô÷ gpu.0.gpu‚91.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚336.753' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚67.3506 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0•a… ûº‚(:ð ëÞèÒ‘ÑÍø disk.dm-0.in‚12197.89453125( disk.nvme0c0n1.out‚60543.57080078125& disk.nvme0n1.out‚60543.57080078125 disk.rbd1.out‚ 13.1015625 disk.sda1.out‚0$ disk./.usageGB‚557.6067352294922 disk.rbd0.out‚0# disk.dm-0.out‚60543.57080078125 disk.rbd3.in‚0$ disk.nvme0c0n1.in‚12197.89453125) proc.memory.percent‚17.24565029144287" disk.nvme0n1.in‚12197.89453125 disk.drbd2165.out‚0 network.sent‚ 6.163687e+06 disk.sda2.in‚0 disk.sda.out‚ 214.328125 disk.dm-1.out‚ 214.328125) proc.memory.availableMB‚ 2264.52734375 disk.sda3.in‚0 disk.sda3.out‚ 214.328125) disk./.usagePercent‚37.42145586446494 disk.rbd0.in‚0 disk.sda1.in‚0 disk.rbd3.out‚0 disk.sda2.out‚0 disk.rbd1.in‚0 disk.sda.in‚0 cpu‚25.582002026454337 proc.cpu.threads‚17 disk.drbd2165.in‚0 network.recv‚ 1.397008e+06 disk.dm-1.in‚0$ memory_percent‚86.17842197418213# proc.memory.rssMB‚ 2825.52734375d3T’»‚(:‡ úÞèÒšÜðö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚267.71' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.542 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0mø¾‚(:ã úÞèÒÅìßø$ disk./.usageGB‚557.6120491027832 disk.dm-1.in‚0 disk.rbd0.out‚0 proc.cpu.threads‚17 disk.dm-0.out‚60556.31640625 disk.rbd3.in‚0" disk.nvme0n1.in‚12197.89453125 disk.sda.out‚ 214.46875$ memory_percent‚86.17029190063477* disk./.usagePercent‚37.421812482960135 disk.drbd2165.out‚0 disk.sda.in‚0 disk.sda1.in‚0 disk.sda2.in‚0 disk.rbd1.out‚ 13.1015625' proc.memory.availableMB‚ 2265.859375 network.sent‚ 6.168683e+06 disk.sda3.out‚ 214.46875 disk.rbd3.out‚0$ disk.nvme0c0n1.in‚12197.89453125) proc.memory.percent‚17.24565029144287 network.recv‚ 1.400332e+06 disk.rbd0.in‚0 disk.dm-0.in‚12197.89453125 disk.drbd2165.in‚0 disk.sda2.out‚0 disk.rbd1.in‚0 disk.dm-1.out‚ 214.46875 disk.sda1.out‚0 disk.sda3.in‚0% disk.nvme0c0n1.out‚60556.31640625# disk.nvme0n1.out‚60556.31640625# proc.memory.rssMB‚ 2825.52734375 cpu‚25.581367692620375âÈZ”½‚(:‰ ‰ßèÒ€Íþö gpu.0.gpu‚36.0 gpu.0.memory‚12, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚332.701' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚66.5402 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0_ƒø þ¾‚(:ó ‰ßèÒ¢¼¢ù network.sent‚ 6.173681e+06 disk.sda1.out‚0( disk.nvme0c0n1.out‚60563.07080078125 disk.sda2.in‚0 disk.drbd2165.out‚0# disk./.usageGB‚557.617790222168 disk.sda.in‚0 disk.sda.out‚ 214.73828125 disk.rbd0.in‚0 disk.sda1.in‚0 disk.sda3.in‚0# proc.memory.rssMB‚ 2825.52734375 cpu‚25.58086934275246) disk./.usagePercent‚37.42219777429917" disk.nvme0n1.in‚12197.89453125& disk.nvme0n1.out‚60563.07080078125 disk.dm-1.in‚0 disk.rbd0.out‚0) proc.memory.availableMB‚ 2265.61328125 disk.sda2.out‚0# disk.dm-0.out‚60563.07080078125) proc.memory.percent‚17.24565029144287$ disk.nvme0c0n1.in‚12197.89453125 disk.dm-1.out‚ 214.73828125 disk.drbd2165.in‚0 disk.rbd3.in‚0 disk.rbd3.out‚0 network.recv‚ 1.403665e+06 disk.sda3.out‚ 214.73828125# memory_percent‚86.1717939376831 proc.cpu.threads‚17 disk.rbd1.in‚0 disk.rbd1.out‚ 13.2265625 disk.dm-0.in‚12197.89453125-Öòh2¿- _wandbruntime‚5491  _runtime‚5491~UÂ_ÇÀÁ & lr-AdamW/pg1‚0.0006551703662081623 & lr-AdamW/pg2‚0.0006551703662081623 trainer/global_step‚3299 % _timestamp‚1.7842953069986262e+09 _runtime‚5491.216591398 _step‚268tï¿ÌÁÆ & lr-AdamW/pg1‚0.0006551703662081623 & lr-AdamW/pg2‚0.0006551703662081623 trainer/global_step‚3299 % _timestamp‚1.7842953069986262e+09 _runtime‚5491.216591398 _step‚268Œ^;ck2Â- _wandbruntime‚5491  _runtime‚5491¸4¿ó^ÃéO ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚56717990 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚256282162769920 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚947703627845632 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚950801205141778 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 VGDeviceStatsMonitor.on_train_bµ¡Ü‘#atch_start/requested_bytes.small_pool.peak‚ 1679318926 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚260284221921792 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚948113763090432 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚17847327 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚950802575170068 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚28577771 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚28140213 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚951217995930112 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3098921896468 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚17847059 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚948113763090432 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4002461779456 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚260284624549376 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚947703653273600 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 % _timestamp‚1.7842953069994063e+09 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚951216624367104 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9379879 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8467443 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 40024272430083E Ðô'ÄîO PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚28567798 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚948113736916992 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3097577296146 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚56708011 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3098921896468 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚17847059 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚948113763090432 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4002461779456 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚260284624549376 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚947703653273600 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 % _timestamp‚1.7842953069994063e+09 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚951216624367104 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9379879 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8467443 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4002427243008 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚56717990 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚256282162769920 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚947703627845632 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚950801205141778 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚260284221921792 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚948113763090432 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚17847327 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚950802575170068 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚28577771 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚28140213 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚951217995930112 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17852383 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚951496921702912 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3105157882368 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚28576276 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚28148419 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚28148425 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4003647435264 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚28586263 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚951496921702912 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚262 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8469991 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚948391737647104 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚260363296612864 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚56724695 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚948391737647104 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8469986 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚951495550132736 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3105157882368 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚56734688 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 % _timestamp‚1.7842953083860478e+09 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚948391763820544 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9382654 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4003612905984 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚947981509623808 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚951081380414602 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚948391763820544 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚56724695 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚28148419 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _runtime‚5492.603443611 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚28576276 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3099845362826 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚28586263 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚948391737647104 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8469986 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚951495550132736 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3105157882368 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚56734688 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 % _timestamp‚1.7842953083860478e+09 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚948391763820544 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9382654 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4003612905984 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚947981509623808 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚951081380414602 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚948391763820544 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚56724695 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚28148419 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _runtime‚5492.603443611 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 J=DeviceStatsMonitor.on_train_batch_end/active.smaƒçõÑll_pool.freed‚28576276 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3099845362826 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚28586263 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17852383 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚951496921702912 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3105157882368 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚28576276 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚28148419 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚28148425 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4003647435264 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚28586263 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚951496921702912 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚262 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8469991 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚948391737647104 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚260363296612864 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚56724695 EDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚965162976803328 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚962013799596032 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4061656471040 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 % _timestamp‚1.7842953855760403e+09 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚3349 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3149203380736 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚961597716754432 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3149203380736 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚260156115062784 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚269 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚962013773422592 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚28993550 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚28550513 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚18112427 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8594593 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3143814409254 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚28993550 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚57554042 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚964742901191976 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚260156483153920 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4061693104640 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0$ûC³ÚîO % _timestamp‚1.7842953855760403e+09 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚3349 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3149203380736 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚961597716754432 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3149203380736 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.larg‚¹Age_pool.freed‚260156115062784 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚269 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚962013773422592 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚28993550 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚28550513 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚18112427 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8594593 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3143814409254 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚28993550 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚57554042 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚964742901191976 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚260156483153920 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4061693104640 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚965162976803328 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚962013799596032 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4061656471040 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0‘çŠÅ†2Û- _wandbruntime‚5571  _runtime‚5571m¸4 ø&ÜòM S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚965441902568960 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3150128416256 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚57560747 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚254 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚260233969561600 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚961875623960576 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚28558719 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚260233601470464 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚18117713 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚57560747 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚28558725 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚965021706436510 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4062884601344 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9520577 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚962291800326144 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚57570740 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3146082475934 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820s ^ý&Ý÷M GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚57560747 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚28558725 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚965021706436510 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4062884601344 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9520577 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚962291800326144 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚57570740 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3146082475934 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚965441902568960 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3150128416256 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚57560747 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚254 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚260233969561600 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚961875623960576 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚28558719 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚260233601470464 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚18117713 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0’{nY2Þ- _wandbruntime‚5571  _runtime‚5571ò½ ß„ % _timestamp‚1.7842953870491812e+09 0loss/FRt1/assignment_loss‚1.6704482945764703 2loss/FBt1/assignment_loss‚0.011638673200682856 /loss/FRt1/detection_loss‚0.5068302848514794 /loss/FBt1/detection_loss‚0.7133920489012131 .loss/FBt2/detection_loss‚0.525053493295084 0loss/FRt2/assignment_loss‚0.7480887918576707 /loss/FRt2/detection_loss‚1.5585686624516935 1loss/SRqqt2/detection_loss‚0.6517746064536145 %loss/total_loss‚8.323906036259377 epoch‚8 trainer/global_step‚3349 _runtime‚5571.266119356 2loss/SRqqt1/assignment_loss‚0.5573242526158946 2loss/SRqqt2/assignment_loss‚0.5034235570791161 _step‚275 4loss/FBt2/assignment_loss‚0.00037295347941778594 1loss/SRqqt1/detection_loss‚0.8769904174970399`êUà‰ /loss/FRt2/detection_loss‚1.5585686624516935 1loss/SRqqt2/detection_loss‚0.6517746064536145 %loss/total_loss‚8.323906036259377 epoch‚8 trainer/global_step‚3349 _runtime‚5571.266119356 2loss/SRqqt1/assignment_loss‚0.5573242526158946 2loss/SRqqt2/assignment_loss‚0.5034235570791161 _step‚275 4loss/FBt2/assignment_loss‚0.00037295347941778594 1loss/SRqqt1/detection_loss‚0.8769904174970399 % _timestamp‚1.7842953870491812e+09 0loss/FRt1/assignment_loss‚1.6704482945764703 2loss/FBt1/assignment_loss‚0.011638673200682856 /loss/FRt1/detection_loss‚0.5068302848514794 /loss/FBt1/detection_loss‚0.7133920489012131 .loss/FBt2/detection_loss‚0.525053493295084 0loss/FRt2/assignment_loss‚0.7480887918576707“‰}”á‚(:‰ ãßèÒ¼Õÿ÷ gpu.0.gpu‚99.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚255.341' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚51.0682 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0:6ùâ‚(:î ãßèÒÜÕ”ù disk.sda.out‚ 216.5390625 disk.dm-1.out‚ 216.5390625 disk.sda3.out‚ 216.5390625$ memory_percent‚92.10424423217773' proc.memory.availableMB‚ 1293.640625) proc.memory.percent‚17.24565029144287 disk.drbd2165.in‚0 disk.sda2.out‚0 cpu‚25.577742142399753 disk.sda1.out‚0$ disk./.usageGB‚557.6483726501465 disk.dm-0.in‚12319.67578125" disk.dm-0.out‚60629.4443359375' disk.nvme0c0n1.out‚60629.4443359375" disk.nvme0n1.in‚12319.67578125 network.recv‚ 1.424991e+06 disk.drbd2165.out‚0% disk.nvme0n1.out‚60629.4443359375 proc.cpu.threads‚17 disk.sda.in‚0 disk.sda1.in‚0 disk.rbd3.in‚0# proc.memory.rssMB‚ 2825.52734375 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda3.in‚0$ disk.nvme0c0n1.in‚12319.67578125 disk.rbd1.in‚0 disk.rbd1.out‚ 13.4140625) disk./.usagePercent‚37.42425018670117 disk.rbd3.out‚0 network.sent‚ 6.260714e+06 disk.sda2.in‚0Ь.VŸã‚(:” òßèÒ¥Ïüö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚290.081' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚58.016200000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0OeÃöä‚(:ë òßèÒÔŽ“ù cpu‚25.57717854954824' disk./.usagePercent‚37.424577876345 disk.sda1.in‚0" disk.nvme0n1.in‚12319.67578125 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda3.in‚0 disk.rbd3.in‚0$ memory_percent‚92.10519790649414$ disk./.usageGB‚557.6532554626465 disk.sda3.out‚ 216.765625( disk.nvme0c0n1.out‚60638.43408203125 disk.sda2.in‚0 disk.sda.out‚ 216.765625# disk.dm-0.out‚60638.43408203125 disk.rbd3.out‚0& disk.nvme0n1.out‚60638.43408203125 network.sent‚ 6.323285e+06 network.recv‚ 1.429495e+06 disk.sda.in‚0 disk.dm-1.out‚ 216.765625 disk.rbd1.out‚ 13.4140625 disk.dm-0.in‚12319.67578125 disk.drbd2165.out‚0 disk.rbd1.in‚0 disk.sda1.out‚0$ disk.nvme0c0n1.in‚12319.67578125 disk.sda2.out‚0' proc.memory.availableMB‚ 1293.484375 disk.dm-1.in‚0 disk.drbd2165.in‚0# proc.memory.rssMB‚ 2825.52734375) proc.memory.percent‚17.24565029144287´QŸå‚(:” àèÒ‘‡÷ gpu.0.gpu‚90.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚285.446' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚57.089200000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0™­»üæ‚(:ñ àèҢ”ù) disk./.usagePercent‚37.42496470372924$ disk./.usageGB‚557.6590194702148 disk.sda2.out‚0 disk.dm-1.in‚0 disk.sda1.out‚0 disk.sda3.out‚ 217.1484375 disk.dm-1.out‚ 217.1484375( disk.nvme0c0n1.out‚60646.43017578125' proc.memory.availableMB‚ 1293.484375 network.sent‚ 6.328828e+06 disk.sda3.in‚0 disk.drbd2165.in‚0 disk.sda2.in‚0 disk.sda.in‚0 disk.sda.out‚ 217.1484375 disk.rbd0.out‚0& disk.nvme0n1.out‚60646.43017578125) proc.memory.percent‚17.24565029144287 network.recv‚ 1.432737e+06 disk.rbd0.in‚0$ disk.nvme0c0n1.in‚12319.67578125 cpu‚25.576706666499387 proc.cpu.threads‚17 disk.rbd1.in‚0 disk.drbd2165.out‚0 disk.rbd3.in‚0 disk.rbd3.out‚0# proc.memory.rssMB‚ 2825.52734375 disk.rbd1.out‚ 13.4765625# disk.dm-0.out‚60646.43017578125 disk.sda1.in‚0 disk.dm-0.in‚12319.67578125" disk.nvme0n1.in‚12319.67578125$ memory_percent‚92.10519790649414ƒÉ™r”ç‚(:‰ àèÒ£¹Ôö gpu.0.gpu‚99.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚39.0 gpu.0.powerWatts‚219.251' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚43.8502 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Rìpyè‚(:ö àèÒÄ­«ø disk.rbd1.in‚0 disk.dm-1.out‚ 217.26953125 disk.rbd3.out‚0& disk.nvme0n1.out‚60657.99267578125 network.recv‚ 1.435712e+06 disk.rbd0.in‚0 disk.sda3.out‚ 217.26953125 disk.rbd3.in‚0" disk.nvme0n1.in‚12319.67578125 disk.sda2.in‚0$ memory_percent‚92.10517406463623 cpu‚25.576156701535407) disk./.usagePercent‚37.42529290538813 disk.sda.in‚0) proc.memory.availableMB‚ 1293.48828125) proc.memory.percent‚17.24565029144287 network.sent‚ 6.333833e+06 disk.rbd0.out‚0 disk.dm-0.in‚12319.67578125 disk.drbd2165.in‚0$ disk.nvme0c0n1.in‚12319.67578125$ disk./.usageGB‚557.6639099121094 disk.rbd1.out‚ 13.4765625 disk.dm-1.in‚0# proc.memory.rssMB‚ 2825.52734375 disk.sda1.in‚0# disk.dm-0.out‚60657.99267578125 disk.sda.out‚ 217.26953125 disk.sda3.in‚0 disk.drbd2165.out‚0( disk.nvme0c0n1.out‚60657.99267578125 disk.sda1.out‚0 disk.sda2.out‚0 proc.cpu.threads‚172Ϲ'”é‚(:‰ ŸàèÒÑËïö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚256.992' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚51.3984 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0†ÔÜUûê‚(:ð ŸàèÒؽ÷ø network.recv‚ 1.439189e+06) disk./.usagePercent‚37.42565438802648$ disk./.usageGB‚557.6692962646484 disk.sda2.in‚0 disk.rbd1.in‚0 disk.sda3.in‚0# disk.nvme0c0n1.in‚ 12399.2734375 disk.sda1.in‚0# proc.memory.rssMB‚ 2825.52734375 cpu‚25.575684478069856 disk.sda.in‚0 disk.sda.out‚ 217.5078125 disk.rbd0.out‚0$ memory_percent‚92.10517406463623) proc.memory.availableMB‚ 1293.48828125 disk.dm-0.in‚ 12399.2734375 disk.rbd3.out‚0( disk.nvme0c0n1.out‚60667.77294921875! disk.nvme0n1.in‚ 12399.2734375& disk.nvme0n1.out‚60667.77294921875 network.sent‚ 6.338833e+06 disk.sda2.out‚0 disk.dm-1.in‚0 proc.cpu.threads‚17 disk.dm-1.out‚ 217.5078125 disk.rbd0.in‚0# disk.dm-0.out‚60667.77294921875 disk.sda1.out‚0 disk.sda3.out‚ 217.5078125 disk.drbd2165.out‚0) proc.memory.percent‚17.24565029144287 disk.rbd3.in‚0 disk.rbd1.out‚ 13.5390625 disk.drbd2165.in‚0H2ë- _wandbruntime‚5648  _runtime‚5648a§…ìÀ & lr-AdamW/pg1‚0.0006526460469483727 & lr-AdamW/pg2‚0.0006526460469483727 trainer/global_step‚3399 $ _timestamp‚1.784295463uÀ74330034e+09 _runtime‚5648.547200287 _step‚276”\ÔÒËíÅ & lr-AdamW/pg1‚0.0006526460469483727 & lr-AdamW/pg2‚0.0006526460469483727 trainer/global_step‚3399 $ _timestamp‚1.784295464330034e+09 _runtime‚5648.547200287 _step‚276”Ôæ²Á2î- _wandbruntime‚5648  _runtime‚5648â÷±Úï'ïéO JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚267 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚268151572306432 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚975913809928192 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚979110700802560 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚18377707 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚18377440 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36633600 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚979109329239552 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚264030435446784 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚978683227213884 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚975491831091200 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 _runtime‚5648.548534887 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚28960813 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚268151167581696 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4120768768512 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 % _timestamp‚1.7842954643312337e+09 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8721743 _step‚277 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚29419302 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 trainer/global_step‚3399 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3196864700928 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚975913809928192 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3195519311360 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚28960813 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚28960819 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3195519311360 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚58390094 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3196864700928 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚29429275 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚975913836101632 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9655697 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚978681857185594 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9655959 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚58380115 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚979110700802560 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3196864700928 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚29429275 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚975913836101632 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9655697 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚978681857185594 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9655959 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚58380115 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚979110700802560 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚979109329239552 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚264030435446784 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚978683227213884 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚975491831091200 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 _runtime‚5648.548534887 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚28960813 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚268151167581696 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4120768768512 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 % _timestamp‚1.7842954643312337e+09 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8721743 _step‚277 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚29419302 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 trainer/global_step‚3399 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3196864700928 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚975913809928192 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3195519311360 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚28960813 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚28960819 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3195519311360 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚58390094 IDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3197789743616 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚975769687441408 trainer/global_step‚3399 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚28969019 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚978960662430016 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚28969019 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3196444346880 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚978962032458418 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚58406792 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4121947540992 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚263 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚268230237486592 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚29427780 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404717568 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚29427780 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 _step‚278 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚58406792 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚58396799 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚976191836831744 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9658391 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚976191810658304 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚58396799 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3196444346880 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚264107921854464 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚5649.938024428 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3197789743616 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚979389626575360 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚268229832769024 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3192319589042 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚975769712869376 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚979388255005184 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8724291 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚264108289945600 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚29437767 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚18382677 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9658654 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚28969025 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36626432 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3190974988608 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚28969025 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚18382945 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216;Þ‹ˆ†óøM PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚28969025 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚18382945 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3197789743616 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚975769687441408 trainer/globaƒÌ úxl_step‚3399 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚28969019 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚978960662430016 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚28969019 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3196444346880 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚978962032458418 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚58406792 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4121947540992 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚263 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚268230237486592 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚29427780 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404717568 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚29427780 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 _step‚278 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚58406792 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚58396799 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚976191836831744 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9658391 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚976191810658304 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚58396799 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3196444346880 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚264107921854464 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚5649.938024428 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3197789743616 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚979389626575360 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚268229832769024 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3192319589042 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚975769712869376 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚979388255005184 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8724291 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚264108289945600 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚29437767 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚18382677 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9658654 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚28969025 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36626432 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3190974988608–ãê(2ô- _wandbruntime‚5650  _runtime‚5650)ÔÜ¥õ€ 0loss/FRt2/assignment_loss‚0.7609349770041612 2loss/SRqqt2/assignment_loss‚0.5302809204002605 1loss/FBt2/assignment_loss‚0.01702857922873431 _runtime‚5650.022085382 _step‚279 1loss/FBt1/assignment_loss‚0.12280535159915361 /loss/FRt1/detection_loss‚0.5820415456922275 epoch‚8 % _timestamp‚1.7842954658051455e+09 1loss/SRqqt1/detection_loss‚0.9150195784794003 1loss/SRqqt2/detection_loss‚0.5948614830268416 .loss/FBt1/detection_loss‚0.599806159085251 /loss/FBt2/detection_loss‚0.4453710686361444 trainer/global_step‚3399 0loss/FRt1/assignment_loss‚1.7286576452580364 2loss/SRqqt1/assignment_loss‚0.5690878881063807 /loss/FRt2/detection_loss‚1.5037581242325428 %loss/total_loss‚8.369653320749135½ý†² ö… _runtime‚5650.022085382 _step‚279 1loss/FBt1/assignment_loss‚0.12280535159915361 /loss/FRt1/detection_loss‚0.5820415456922275 epoch‚8 % _timestamp‚1.7842954658051455e+09 1loss/SRqqt1/detection_loss‚0.9150195784794003 1loss/SRqqt2/detection_loss‚0.5948614830268416 .loss/FBt1/detection_loss‚0.599806159085251 /loss/FBt2/detection_loss‚0.4453710686361444 trainer/global_step‚3399 0loss/FRt1/assignment_loss‚1.7286576452580364 2loss/SRqqt1/assignment_loss‚0.5690878881063807 /loss/FRt2/detection_loss‚1.5037581242325428 %loss/total_loss‚8.369653320749135 0loss/FRt2/assignment_loss‚0.7609349770041612 2loss/SRqqt2/assignment_loss‚0.5302809204002605 1loss/FBt2/assignment_loss‚0.01702857922873431—?éþ÷‚(:ó ®àèÒ¡þ­ù disk.dm-1.out‚ 218.62890625 disk.sda3.in‚0 disk.rbd3.in‚0 disk.rbd3.out‚0# disk.nvme0c0n1.in‚ 12399.2734375( disk.nvme0c0n1.out‚60681.07958984375! disk.nvme0n1.in‚ 12399.2734375 disk.sda2.in‚0 disk.rbd1.in‚0 disk.drbd2165.in‚0 cpu‚25.575216400528443 disk.rbd0.out‚0 disk.sda.out‚ 218.62890625 disk.drbd2165.out‚0 disk.sda2.out‚0 network.sent‚ 6.343879e+06 disk.sda.in‚0 disk.dm-0.in‚ 12399.2734375 disk.sda1.in‚0 disk.sda3.out‚ 218.62890625& disk.nvme0n1.out‚60681.07958984375 disk.rbd1.out‚ 13.5390625$ memory_percent‚92.10543632507324$ disk./.usageGB‚557.6800079345703 disk.dm-1.in‚0 disk.rbd0.in‚0# disk.dm-0.out‚60681.07958984375 disk.sda1.out‚0( proc.memory.availableMB‚ 1293.4453125# proc.memory.rssMB‚ 2825.52734375 network.recv‚ 1.442336e+06* disk./.usagePercent‚37.426373257182625) proc.memory.percent‚17.24565029144287 proc.cpu.threads‚17mâ·ƒ”ø‚(:‰ ®àèÒ©™Ë‰ gpu.0.gpu‚66.0 gpu.0.memory‚20, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚111.482' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.2964 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0f=Ñå”ù‚(:‰ ½àèÒŽÞ†÷ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚280.902' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚56.1804 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0+´Ì™÷ú‚(:ì ½àèÒ˜“³ù disk.rbd0.out‚0 disk.drbd2165.in‚0% disk.nvme0n1.out‚60697.0576171875# proc.memory.rssMB‚ 2825.52734375 disk.sda.in‚0 disk.rbd3.in‚0# disk.nvme0c0n1.in‚ 12399.2734375' disk.nvme0c0n1.out‚60697.0576171875 cpu‚25.57466649308772 network.sent‚ 6.406331e+06 disk.dm-1.out‚ 220.20703125 disk.sda1.out‚0 disk.sda2.in‚0 disk.rbd1.in‚0$ memory_percent‚92.10553169250488 proc.cpu.threads‚17 disk.dm-0.in‚ 12399.2734375 network.recv‚ 1.447154e+06 disk.sda.out‚ 220.20703125 disk.rbd0.in‚0" disk.dm-0.out‚60697.0576171875 disk.sda3.out‚ 220.20703125 disk.rbd3.out‚0' disk./.usagePercent‚37.426439563134 disk.sda3.in‚0 disk.drbd2165.out‚0( proc.memory.availableMB‚ 1293.4296875$ disk./.usageGB‚557.6809959411621 disk.dm-1.in‚0! disk.nvme0n1.in‚ 12399.2734375 disk.sda2.out‚0) proc.memory.percent‚17.24565029144287 disk.sda1.in‚0 disk.rbd1.out‚ 13.6015625ŽéíŸû‚(:” ÌàèÒ›ëÐö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚269.838' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚53.967600000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0&wlðùü‚(:î ÌàèÒ¢×¾ø disk.sda3.out‚ 222.41796875 disk.dm-1.in‚0 network.recv‚ 1.450516e+06 disk.drbd2165.in‚0 disk.rbd3.out‚0' disk.nvme0c0n1.out‚60725.5908203125 disk.sda2.in‚0 disk.rbd1.in‚0( proc.memory.availableMB‚ 1293.4296875 disk.dm-1.out‚ 222.41796875 cpu‚25.57421607397425$ disk./.usageGB‚557.6893615722656 disk.sda.in‚0 disk.rbd0.in‚0 network.sent‚ 6.411909e+06! disk.nvme0n1.in‚ 12399.2734375% disk.nvme0n1.out‚60725.5908203125 disk.rbd0.out‚0" disk.dm-0.out‚60725.5908203125# proc.memory.rssMB‚ 2825.52734375) disk./.usagePercent‚37.42700098765659) proc.memory.percent‚17.24565029144287 disk.sda3.in‚0 disk.drbd2165.out‚0 disk.rbd3.in‚0# disk.nvme0c0n1.in‚ 12399.2734375 disk.rbd1.out‚ 13.6015625$ memory_percent‚92.10553169250488 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.out‚0 disk.sda.out‚ 222.41796875 disk.dm-0.in‚ 12399.2734375 proc.cpu.threads‚17ªhªŸý‚(:” ÛàèÒˆÂèö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚252.395' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚50.479000000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Wg7ÿþ‚(:ô ÛàèÒ÷ÞÝø network.sent‚ 6.416915e+06# disk.dm-0.out‚60740.12255859375( disk.nvme0c0n1.out‚60740.12255859375" disk.nvme0n1.in‚12504.31640625 cpu‚25.57362613022142 disk.rbd3.in‚0 disk.sda3.in‚0 disk.rbd3.out‚0$ disk.nvme0c0n1.in‚12504.31640625 disk.sda2.out‚0$ memory_percent‚92.10550785064697) disk./.usagePercent‚37.42721936208329 disk.rbd0.in‚0 disk.sda3.out‚ 224.66015625) proc.memory.availableMB‚ 1293.43359375 network.recv‚ 1.45431e+06 disk.dm-1.in‚0 disk.dm-0.in‚12504.31640625 disk.drbd2165.out‚0& disk.nvme0n1.out‚60740.12255859375 disk.sda2.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 13.7265625$ disk./.usageGB‚557.6926155090332 disk.rbd0.out‚0# proc.memory.rssMB‚ 2825.52734375 proc.cpu.threads‚17 disk.sda.in‚0 disk.sda.out‚ 224.66015625 disk.dm-1.out‚ 224.66015625 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd2165.in‚0) proc.memory.percent‚17.24565029144287RéÁ‹žÿ‚(:“ êàèÒž¬Ûö gpu.0.gpu‚93.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚46.0 gpu.0.powerWatts‚336.465' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚67.29299999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ox €‚(:õ êàèÒ—ÄÆø network.sent‚ 6.421962e+06 disk.sda3.in‚0) proc.memory.percent‚17.24565029144287 network.recv‚ 1.45764e+06 disk.rbd1.out‚ 13.7265625 disk.sda.in‚0# disk.dm-0.out‚60752.20361328125 proc.cpu.threads‚17* disk./.usagePercent‚37.427578284646295$ disk./.usageGB‚557.6979637145996 disk.dm-1.out‚ 225.87109375 disk.rbd0.out‚0 disk.drbd2165.in‚0 disk.drbd2165.out‚0$ memory_percent‚92.10550785064697 cpu‚25.57317402118373 disk.rbd0.in‚0 disk.dm-0.in‚12504.31640625 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd3.in‚0 disk.rbd3.out‚0) proc.memory.availableMB‚ 1293.43359375# proc.memory.rssMB‚ 2825.52734375 disk.sda.out‚ 225.87109375 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-1.in‚0 disk.sda3.out‚ 225.87109375 disk.rbd1.in‚0$ disk.nvme0c0n1.in‚12504.31640625( disk.nvme0c0n1.out‚60752.20361328125" disk.nvme0n1.in‚12504.31640625& disk.nvme0n1.out‚60752.20361328125f¢Aø2- _wandbruntime‚5727  _runtime‚5727§¶"3Ç‚Á _step‚280 & lr-AdamW/pg1‚0.0006513542962136524 & lr-AdamW/pg2‚0.0006513542962136524 trainer/global_step‚3449 % _timestamp‚1.7842955431394775e+09 _runtime‚5727.356589723Ô?”ÌƒÆ _step‚280 & lr-AdamW/pg1‚0.0006513542962136524 & lr-AdamW/pg2‚0.0006513542962136524 trainer/global_step‚3449 % _timestamp‚1.7842955431394775e+09 _runtime‚5727.356589723˜ú;ê'2„- _wandbruntime‚5727  _runtime‚5727±Æï'…éO JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚269 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _step‚281 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚993055681675776 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3243180631552 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4179839755776 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚993057053238784 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚29371113 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚989385894572032 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚989813846433792 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9793438 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚29371113 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚264 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4179874292224 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚29371119 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚18642067 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3237633235792 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚989813872607232 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚267904755830784 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3241835241984 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820ÇD-â†îO TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚989813872607232 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚267905123921920 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚29845054 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7842955431405041e+09 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚272084998214144 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚993055681675776 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3243180631552 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4179839755776 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚993057053238784 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚29371113 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚989385894572032 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 E;DevWS¤_iceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚989813846433792 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9793438 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚29371113 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚264 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4179874292224 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚29371119 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚18642067 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3237633235792 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚989813872607232 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚267904755830784 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3241835241984 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚269 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _step‚281 I!S2‡- _wandbruntime‚5728  _runtime‚5728+ù&ˆóM XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3242760277504 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚992900988451924 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3237212101716 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4181024975360 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚29853532 % _timestamp‚1.7842955445265915e+09 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚59242844 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚267982610329600 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 _step‚282 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚272163669834240 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚29853532 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4181059504640 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚990091873337344 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3244105674240 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚59232851 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 trainer/global_step‚3449 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚29379325 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚18647347 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚29379319 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚29379325 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚993334607441408 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚993335979011584 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚992902358480326 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3244105674240 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚29863519 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚59232851 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚272163267213824 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3242760277504 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 _runtime‚5728.744120138 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚990091847163904 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚990091847163904 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚29379319 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9795911 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3238556702150 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚989663776350208 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚993335979011584 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9796169 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚59242844¤â-jþ&‰øM \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3244105674240 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚59232851 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 trainer/global_step‚3449 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚29379325 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚18647347 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚29379319 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚29379325 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚993334607441408 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚993335979011584 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚992902358480326 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3244105674240 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚29863519 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚59232851 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚272163267213824 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3242760277504 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 _runtime‚5728.744120138 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚990091847163904 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚990091847163904 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚29379319 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9795911 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3238556702150 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚989663776350208 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚993335979011584 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9796169 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚59242844 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3242760277504 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚992900988451924 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3237212101716 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4181024975360 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚29853532 % _timestamp‚1.7842955445265915e+09 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚59242844 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚267982610329600 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 _step‚282 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚272163669834240 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚29853532 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4181059504640 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚990091873337344 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175š¼Õÿ2Š- _wandbruntime‚5728  _runtime‚5728½`R`‹ 0loss/FRt2/assignment_loss‚0.8714537548850168 2loss/SRqqt2/assignment_loss‚0.5602093165395093 1loss/SRqqt2/detection_loss‚0.6166537954188126 %loss/total_loss‚8.711567931768329 epoch‚8 $ _timestamp‚1.784295544612752e+09 _runtime‚ 5728.82968577 2loss/SRqqt1/assignment_loss‚0.5347468793870778 2loss/FBt2/assignment_loss‚0.021719148541333454 /loss/FRt1/detection_loss‚0.5727878415328747 _step‚283 2loss/FBt1/assignment_loss‚0.005902642108958744 1loss/SRqqt1/detection_loss‚0.9359187928974511 0loss/FRt1/assignment_loss‚1.7535858286686263 /loss/FRt2/detection_loss‚1.4890355920410312 /loss/FBt1/detection_loss‚0.8260800291318446 /loss/FBt2/detection_loss‚0.5234743106157932 trainer/global_step‚3449*Ížx Œ† 0loss/FRt2/assignment_loss‚0.8714537548850168 2loss/SRqqt2/assignment_loss‚0.5602093165395093 1loss/SRqqt2/detection_loss‚0.6166537954188126 %loss/total_loss‚8.711567931768329 epoch‚8 $ _timestamp‚1.784295544612752e+09 _runtime‚ 5728.82968577 2loss/SRqqt1/assignment_loss‚0.5347468793870778 2loss/FBt2/assignment_loss‚0.021719148541333454 /loss/FRt1/detection_loss‚0.5727878415328747 _step‚283 2loss/FBt1/assignment_loss‚0.005902642108958744 1loss/SRqqt1/detection_loss‚0.9359187928974511 0loss/FRt1/assignment_loss‚1.7535858286686263 /loss/FRt2/detection_loss‚1.4890355920410312 /loss/FBt1/detection_loss‚0.8260800291318446 /loss/FBt2/detection_loss‚0.5234743106157932 trainer/global_step‚3449›èJ•ÂŸ‚(:” ùàèÒÇÉÅö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚297.863' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚59.572599999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 'ÔüŽ‚(:ñ ùàèÒ‚êºø) disk./.usagePercent‚37.42791749462916 disk.sda.out‚ 226.55078125" disk.dm-0.out‚60763.4658203125 disk.sda1.in‚0 disk.rbd1.out‚ 13.7890625( proc.memory.availableMB‚ 1293.4453125$ disk./.usageGB‚557.7030181884766 disk.sda.in‚0 disk.dm-1.out‚ 226.55078125 disk.rbd0.in‚0 disk.rbd3.out‚0 disk.sda2.out‚0) proc.memory.percent‚17.24565029144287 network.sent‚ 6.426975e+06 disk.dm-1.in‚0 disk.rbd0.out‚0 disk.dm-0.in‚12504.31640625 disk.sda3.out‚ 226.55078125' disk.nvme0c0n1.out‚60763.4658203125 network.recv‚ 1.461075e+06 disk.drbd2165.out‚0% disk.nvme0n1.out‚60763.4658203125# proc.memory.rssMB‚ 2825.52734375 disk.sda1.out‚0 disk.sda3.in‚0 disk.drbd2165.in‚0 disk.rbd3.in‚0$ disk.nvme0c0n1.in‚12504.31640625 disk.sda2.in‚0$ memory_percent‚92.10543632507324 proc.cpu.threads‚17" disk.nvme0n1.in‚12504.31640625 disk.rbd1.in‚0 cpu‚25.57263652766392Ç<ž‚(:“ ˆáèÒªóÞö gpu.0.gpu‚91.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚46.0 gpu.0.powerWatts‚345.056' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚69.01119999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0üuÂàú‚(:ï ˆáèÒäˆÐø network.recv‚ 1.465541e+06 disk.dm-0.in‚12504.31640625 disk.rbd3.out‚0 proc.memory.rssMB‚ 2825.53125 disk.sda1.in‚0 disk.sda1.out‚0) proc.memory.percent‚17.24567413330078# disk./.usageGB‚557.704761505127 disk.rbd0.in‚0 disk.drbd2165.out‚0& disk.nvme0n1.out‚60776.88818359375 disk.sda2.in‚0# disk.dm-0.out‚60776.88818359375 disk.sda3.out‚ 227.97265625 disk.drbd2165.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 13.7890625 cpu‚25.572272605064416 disk.sda3.in‚0 disk.rbd3.in‚0 network.sent‚ 6.489744e+06* disk./.usagePercent‚37.428034490072314 disk.dm-1.in‚0& proc.memory.availableMB‚ 1293.65625 disk.sda.in‚0 disk.dm-1.out‚ 227.97265625# memory_percent‚92.1041488647461 proc.cpu.threads‚17 disk.sda.out‚ 227.97265625 disk.rbd0.out‚0$ disk.nvme0c0n1.in‚12504.31640625( disk.nvme0c0n1.out‚60776.88818359375" disk.nvme0n1.in‚12504.31640625 disk.sda2.out‚0»eNwŸ‘‚(:” —áèÒƇêö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚258.298' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚51.659600000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ù¡ñó’‚(:è —áèÒ°øÓø disk.rbd0.in‚0 disk.rbd0.out‚0# disk.dm-0.out‚60785.35205078125 disk.sda1.in‚0 disk.drbd2165.in‚0$ memory_percent‚92.10309982299805' proc.memory.availableMB‚ 1293.828125 disk.sda1.out‚0 disk.rbd3.in‚0" disk.nvme0c0n1.in‚ 12573.296875 disk.nvme0n1.in‚ 12573.296875 proc.cpu.threads‚17$ disk./.usageGB‚557.7097816467285 disk.sda.in‚0 disk.sda3.out‚ 228.3359375 disk.sda2.out‚0 disk.dm-1.out‚ 228.3359375 disk.rbd1.in‚0 disk.sda3.in‚0 disk.drbd2165.out‚0 disk.rbd3.out‚0( disk.nvme0c0n1.out‚60785.35205078125 proc.memory.rssMB‚ 2825.53125 network.recv‚ 1.469393e+06 disk.dm-1.in‚0 disk.rbd1.out‚ 13.8515625 network.sent‚ 6.494998e+06) disk./.usagePercent‚37.42837139598738 disk.dm-0.in‚ 12573.296875& disk.nvme0n1.out‚60785.35205078125 disk.sda2.in‚0) proc.memory.percent‚17.24567413330078 cpu‚25.571739048133132 disk.sda.out‚ 228.3359375iÈH”“‚(:‰ ¦áèÒ”´íö gpu.0.gpu‚63.0 gpu.0.memory‚20, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚336.034' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚67.2068 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¿ò|ûõ”‚(:ê ¦áèÒžµØø network.sent‚ 6.500006e+06) disk./.usagePercent‚37.42836678785176 disk.sda1.in‚0 disk.drbd2165.in‚0 disk.drbd2165.out‚0$ memory_percent‚92.10476875305176 proc.memory.rssMB‚ 2825.53125# disk.dm-0.out‚60799.14794921875 disk.sda3.out‚ 228.63671875 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.rbd0.in‚0 disk.rbd0.out‚0$ disk./.usageGB‚557.7097129821777 disk.dm-0.in‚ 12573.296875( disk.nvme0c0n1.out‚60799.14794921875 disk.rbd1.out‚ 13.8515625 disk.sda.in‚0 cpu‚25.5712944296517( proc.memory.availableMB‚ 1293.5546875 disk.sda3.in‚0 disk.nvme0n1.in‚ 12573.296875& disk.nvme0n1.out‚60799.14794921875) proc.memory.percent‚17.24567413330078" disk.nvme0c0n1.in‚ 12573.296875 disk.sda2.out‚0 disk.sda.out‚ 228.63671875 disk.dm-1.in‚0 disk.sda1.out‚0 disk.sda2.in‚0 disk.rbd1.in‚0 disk.dm-1.out‚ 228.63671875 proc.cpu.threads‚17 network.recv‚ 1.472991e+06à™–Â•‚(:‰ µáèÒ’÷ö gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWa­ÇÒtts‚266.372' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.2744 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÀZLìô–‚(:é µáèÒð¢•ù) disk./.usagePercent‚37.42903317546338# disk.dm-0.out‚60817.21728515625 disk.sda1.out‚0 disk.nvme0n1.in‚ 12573.296875 disk.rbd0.in‚0 proc.memory.rssMB‚ 2825.53125 disk.rbd3.out‚0" disk.nvme0c0n1.in‚ 12573.296875 disk.sda.in‚0 disk.sda.out‚ 231.3046875 disk.sda1.in‚0& disk.nvme0n1.out‚60817.21728515625 disk.sda2.out‚0 disk.rbd1.out‚ 13.9140625 cpu‚25.57076205604063 proc.cpu.threads‚17 network.sent‚ 6.505042e+06 disk.sda3.in‚0 disk.rbd1.in‚0 disk.dm-1.out‚ 231.3046875) proc.memory.availableMB‚ 1293.79296875 disk.drbd2165.out‚0 disk.dm-1.in‚0 disk.dm-0.in‚ 12573.296875( disk.nvme0c0n1.out‚60817.21728515625) proc.memory.percent‚17.24567413330078 disk.rbd0.out‚0 disk.rbd3.in‚0 network.recv‚ 1.476409e+06 disk.sda3.out‚ 231.3046875 disk.drbd2165.in‚0$ disk./.usageGB‚557.7196426391602 disk.sda2.in‚0$ memory_percent‚92.10331439971924ݽøÌ”—‚(:‰ ÄáèÒÁùåö gpu.0.gpu‚90.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚220.076' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚44.0152 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Øûz ö˜‚(:ë ÄáèÒñ‚Øø) proc.memory.percent‚17.24567413330078 disk.rbd0.out‚0 disk.drbd2165.in‚0 disk.rbd3.in‚0$ memory_percent‚92.10505485534668 cpu‚25.570327002896523 disk.rbd1.in‚0 disk.dm-0.in‚ 12573.296875 disk.rbd3.out‚0 disk.nvme0n1.in‚ 12573.296875 proc.cpu.threads‚17 network.sent‚ 6.510039e+06 disk.sda.in‚0( disk./.usagePercent‚37.4294422755031$ disk./.usageGB‚557.7257385253906 disk.dm-1.in‚0 disk.dm-1.out‚ 234.08984375 disk.sda3.out‚ 234.08984375 disk.drbd2165.out‚0( proc.memory.availableMB‚ 1293.5078125 network.recv‚ 1.479477e+06 disk.sda.out‚ 234.08984375 disk.sda1.out‚0 disk.sda2.out‚0# disk.dm-0.out‚60842.21337890625 disk.sda1.in‚0 disk.sda2.in‚0 disk.rbd0.in‚0( disk.nvme0c0n1.out‚60842.21337890625 disk.rbd1.out‚ 13.9140625 disk.sda3.in‚0" disk.nvme0c0n1.in‚ 12573.296875& disk.nvme0n1.out‚60842.21337890625 proc.memory.rssMB‚ 2825.53125'8íä2™- _wandbruntime‚5805  _runtime‚5805äGµ¦ÇšÁ & lr-AdamW/pg1‚0.0006500429282762612 & lr-AdamW/pg2‚0.0006500429282762612 trainer/global_step‚3499 % _timestamp‚1.7842956217116916e+09 _runtime‚5805.928897157 _step‚284¡y±Ì›Æ & lr-AdamW/pg1‚0.0006500429282762612 & lr-AdamW/pg2‚0.0006500429282762612 trainer/global_step‚3499 % _timestamp‚1.7842956217116916e+09 _runtime‚5805.928897157 _step‚284œ»¡F;2œ- _wandbruntime‚5805  _runtime‚5805÷ljKú'ôO WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚30270806 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1003713882939392 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3282525748578 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1007003405675008 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚29781413 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3288151172608 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3289496562176 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚251 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚30280779 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1006563879257700 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 $ _timestamp‚1.784295621712756e+09 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚60062198 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚30280779 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8976043 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1007002034112000 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8976048 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚276018382182912 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1006562509229410 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚18906960 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9930917 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1003713909112832 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1003279983480832 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1007002034112000 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3288151172608 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3283870348900 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚60052219 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9931168 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4238937876992 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚29781419 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚271779076214784 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚3499 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 $ _timestamp‚1.784295621712756e+09 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚60062198 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚30280779 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8976043 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1007002034112000 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8976048 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚276018382182912 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1006562509229410 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚18906960 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9930917 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1003713909112832 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1003279983480832 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1007002034112000 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3288151172608 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3283870348900 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚60052219 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9931168 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4238937876992 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚29781419 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚271779076214784 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚3499 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1007003405675008 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚29781413 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648oÁU2Ÿ- _wandbruntime‚5807  _runtime‚5807{ͤ:# ÿM K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚276097056458240 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1003991883669504 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1003991883669504 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1006842684502234 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚29789619 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9933638 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1003557865259008 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚18912224 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8978586 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3289076208128 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚249 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚60068903 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚244 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 trainer/global_step‚3499 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 QDeviceStatsMonitor.on ´}Ë_train_batch_end/allocation.all.allocated‚60078896 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4240091215360 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4240125744640 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1003557890686976 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1003991909842944 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚276096653837824FHhá '¡„N YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1007282331447808 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1007280959877632 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚60078896 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1006841314473832 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚271856562622464 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚29789625 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9933638 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1003557865259008 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚18912224 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8978586 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3289076208128 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚249 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚60068903 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚244 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 trainer/global_step‚3499 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚60078896 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4240091215360 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4240125744640 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1003557890686976 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1003991909842944 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚276096653837824 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚276097056458240 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1003991883669504 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1003991883669504 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1006842684502234 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚29789619 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 EDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚30706531 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚258 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚19171874 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _step‚289 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚30191719 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4298409257472 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9103193 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1017613919444992 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚253 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1020949758111232 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1017613945618432 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1017174097817600 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1020504205279608 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1020502835251318 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 % _timestamp‚1.7842957004579105e+09 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚60898250 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚19172132 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 _runtime‚5884.675239238 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3330107462008 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚30191713 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚30696558 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9103198 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚30706531 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3328762861686 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 trainer/global_step‚3549 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3334467103232 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1017613919444992 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚279952173947392 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1020948386548224 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995yðƒ«À´üO G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10068681 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚275653396598784 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚275653764689920 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1020948386548224 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚60888271 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3335812492800 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚30706531 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚258 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚19171874 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _step‚289 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚30191719 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 G?DeviceStatsMonitor.on_train_batch_sta'´HaBrt/segment.small_pool.peak‚820 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4298409257472 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9103193 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1017613919444992 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚253 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1020949758111232 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1017613945618432 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1017174097817600 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1020504205279608 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1020502835251318 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 % _timestamp‚1.7842957004579105e+09 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚60898250 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚19172132 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 _runtime‚5884.675239238 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3330107462008 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚30191713 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚30696558 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9103198 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚30706531 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3328762861686 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 trainer/global_step‚3549 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3334467103232 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1017613919444992 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚279952173947392 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 IDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚19177177 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚60914948 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚30199919 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚60904955 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3335392138752 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚280030451901952 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1017891920175104 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 _step‚290 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚280030854522368 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚30715023 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚245 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 % _timestamp‚1.7842957018440824e+09 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3336737535488 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9105741 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1020781640495740 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚60904955 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚250 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3336737535488 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚275731251097600 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚19177427 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1017891946348544 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9105736 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10071686 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1020783010524142 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚30199919 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 trainer/global_step‚3549 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4299603424768 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚30705036 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚60914948 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚30715023 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1017891920175104 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1021228683884032 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1017451979595776 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1017451954167808 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚30199925 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚280030854522368 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚30715023 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚245 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 % _timestamp‚1.7842957018440824e+09 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3336737535488 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9105741 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1020781640495740 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚60904955 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚250 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3336737535488 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚275731251097600 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚19177427 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1017891946348544 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9105736 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10071686 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1020783010524142 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚30199919 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 trainer/global_step‚3549 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4299603424768 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚30705036 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚60914948 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚30715023 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1017891920175104 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1021228683884032 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1017451979595776 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1017451954167808 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚30199925 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚19177177 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚60914948 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚30199919 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚60904955 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3335392138752 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚280030451901952 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1¢60Z2¸- _wandbruntime‚5886  _runtime‚5886ªœÄU ¹ƒ 0loss/FRt2/assignment_loss‚0.7174607610579619 2loss/SRqqt1/assignment_loss‚0.6451578706186842 2loss/SRqqt2/assignment_loss‚0.5302470547259611 3loss/FBt1/assignment_loss‚0.0002317428487510356 1loss/SRqqt1/detection_loss‚1.0135553106112027 1loss/SRqqt2/detection_loss‚0.6274484548494634 epoch‚8 % _timestamp‚1.7842957019304333e+09 3loss/FBt2/assignment_loss‚6.842849981088997e-05 /loss/FRt2/detection_loss‚1.5184558358657914 /loss/FBt1/detection_loss‚0.7664737886858776 /loss/FBt2/detection_loss‚0.6604727514488145 _runtime‚5886.147363477 _step‚291 trainer/global_step‚3549 .loss/FRt1/assignment_loss‚1.79436313671668 /loss/FRt1/detection_loss‚0.5221939788221364 %loss/total_loss‚8.796129114751135[Ó]ºˆ 0loss/FRt2/assignment_loss‚0.7174607610579619 2loss/SRqqt1/assignment_loss‚0.6451578706186842 2loss/SRqqt2/assignment_loss‚0.5302470547259611 3loss/FBt1/assignment_loss‚0.0002317428487510356 1loss/SRqqt1/detection_loss‚1.0135553106112027 1loss/SRqqt2/detection_loss‚0.6274484548494634 epoch‚8 % _timestamp‚1.7842957019304333e+09 3loss/FBt2/assignment_loss‚6.842849981088997e-05 /loss/FRt2/detection_loss‚1.5184558358657914 /loss/FBt1/detection_loss‚0.7664737886858776 /loss/FBt2/detection_loss‚0.6604727514488145 _runtime‚5886.147363477 _step‚291 trainer/global_step‚3549 .loss/FRt1/assignment_loss‚1.79436313671668 /loss/FRt1/detection_loss‚0.5221939788221364 %loss/total_loss‚8.796129114751135£Ó8Î “»‚(:ˆ žâèÒ¬ìÊö gpu.0.gpu‚90.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚289.215' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚57.843 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0£kp~õ¼‚(:ê žâèÒ„œ¨ø* disk./.usagePercent‚37.431967021805775 disk.sda.in‚0 disk.sda3.out‚ 240.60546875 disk.drbd2165.in‚0' disk.nvme0c0n1.out‚60926.9150390625 disk.rbd1.out‚ 14.1640625 disk.sda1.out‚0" disk.nvme0c0n1.in‚ 12730.609375( proc.memory.availableMB‚ 1292.5078125$ disk./.usageGB‚557.7633590698242 disk.dm-1.in‚0 disk.drbd2165.out‚0 disk.rbd3.in‚0) proc.memory.percent‚17.24567413330078 network.sent‚ 6.597098e+06 disk.nvme0n1.in‚ 12730.609375 cpu‚25.567486688078965 disk.sda1.in‚0 disk.sda3.in‚0 proc.memory.rssMB‚ 2825.53125 disk.rbd1.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.dm-0.in‚ 12730.609375" disk.dm-0.out‚60926.9150390625 disk.rbd3.out‚0 proc.cpu.threads‚17 network.recv‚ 1.500475e+06 disk.sda2.out‚0% disk.nvme0n1.out‚60926.9150390625$ memory_percent‚92.11115837097168 disk.sda2.in‚0 disk.sda.out‚ 240.60546875 disk.dm-1.out‚ 240.60546875vþqŸ½‚(:” ­âèÒ­´Š÷ gpu.0.gpu‚93.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚250.801' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚50.160199999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¥:(:ã ­âèÒŸÄù disk.drbd2165.out‚0 disk.rbd3.in‚0$ memory_percent‚92.11134910583496( proc.memory.availableMB‚ 1292.4765625 disk.rbd0.out‚0 network.sent‚ 6.659904e+06( disk.nvme0c0n1.out‚60932.74755859375 proc.cpu.threads‚17 disk.sda1.out‚0 disk.rbd3.out‚0 disk.sda.out‚ 241.078125 disk.rbd0.in‚0# disk.dm-0.out‚60932.74755859375& disk.nvme0n1.out‚60932.74755859375 disk.dm-1.in‚0 disk.sda3.in‚0 network.recv‚ 1.505202e+06 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 241.078125 disk.sda1.in‚0" disk.nvme0c0n1.in‚ 12730.609375) proc.memory.percent‚17.24567413330078 cpu‚25.5670523330967) disk./.usagePercent‚37.43228779924618# disk./.usageGB‚557.768138885498 disk.nvme0n1.in‚ 12730.609375 disk.rbd1.in‚0 disk.rbd1.out‚ 14.2265625 proc.memory.rssMB‚ 2825.53125 disk.sda.in‚0 disk.dm-1.out‚ 241.078125 disk.dm-0.in‚ 12730.609375 disk.drbd2165.in‚0@ÙÇ¿‚(:“ ¼âèÒÍÁÈö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚@c‡×291.631' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚58.32619999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0&.(ÛåÀ‚(:Ú ¼âèÒ„æùø disk.dm-0.in‚ 12730.609375 disk.sda1.in‚0 proc.memory.rssMB‚ 2825.53125) proc.memory.percent‚17.24567413330078 disk.rbd0.in‚0! disk.dm-0.out‚60941.388671875 disk.drbd2165.out‚0 disk.rbd1.out‚ 14.2265625 disk.sda.out‚241.8125 disk.drbd2165.in‚0# disk./.usageGB‚557.770923614502 disk.nvme0n1.in‚ 12730.609375 disk.sda2.out‚0 network.sent‚ 6.664906e+06 disk.dm-1.out‚241.8125 disk.rbd0.out‚0 disk.sda1.out‚0" disk.nvme0c0n1.in‚ 12730.609375) disk./.usagePercent‚37.43247468474618 disk.sda3.out‚241.8125$ memory_percent‚92.11137294769287 disk.dm-1.in‚0 disk.sda3.in‚0 disk.sda2.in‚0 disk.rbd1.in‚0) proc.memory.availableMB‚ 1292.47265625 cpu‚25.566552238826283 proc.cpu.threads‚17 network.recv‚ 1.508496e+06 disk.sda.in‚0 disk.rbd3.in‚0 disk.rbd3.out‚0& disk.nvme0c0n1.out‚60941.388671875$ disk.nvme0n1.out‚60941.388671875Û£]¤žÁ‚(:“ ËâèÒäÕöö gpu.0.gpu‚92.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚326.369' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚65.27380000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0w27Pç‚(:Ü ËâèÒŽ´¶ù disk.dm-0.in‚ 12772.21875 disk.sda1.in‚0 disk.drbd2165.out‚0 disk.rbd1.out‚ 14.2890625 network.recv‚ 1.511916e+06# disk./.usageGB‚557.771354675293 disk.rbd0.out‚0 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.dm-1.in‚0 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.sda2.in‚0) proc.memory.availableMB‚ 1292.18359375 disk.sda3.in‚0 disk.sda3.out‚ 243.26171875 disk.rbd1.in‚0) disk./.usagePercent‚37.43250361359755! disk.nvme0c0n1.in‚ 12772.21875% disk.nvme0c0n1.out‚60949.11328125 cpu‚25.566170887922002 network.sent‚6.67e+06 disk.sda.in‚0 disk.sda.out‚ 243.26171875 disk.dm-1.out‚ 243.26171875 disk.rbd0.in‚0 disk.dm-0.out‚60949.11328125$ memory_percent‚92.11313724517822 disk.nvme0n1.in‚ 12772.21875 disk.sda2.out‚0# disk.nvme0n1.out‚60949.11328125 proc.memory.rssMB‚ 2825.53125) proc.memory.percent‚17.24567413330078 proc.cpu.threads‚17­èƔÂ(:‰ ÚâèÒ¥­üö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚281.671' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚56.3342 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 u¬»âÄ‚(:× ÚâèÒ–ý³ù disk.sda.in‚0 disk.sda.out‚244.875 disk.sda1.in‚0 disk.rbd3.in‚0" disk.dm-0.out‚60966.1923828125 disk.sda3.in‚0 proc.memory.rssMB‚ 2825.53125 network.recv‚ 1.51523e+06 disk.rbd1.out‚ 14.2890625 disk.dm-1.in‚0 disk.drbd2165.in‚0 disk.nvme0n1.in‚ 12772.21875) proc.memory.percent‚17.24567413330078 cpu‚25.565628674024158 proc.cpu.threads‚17 network.sent‚ 6.675028e+06 disk.sda3.out‚244.875 disk.drbd2165.out‚0 disk.rbd3.out‚0! disk.nvme0c0n1.in‚ 12772.21875' disk.nvme0c0n1.out‚60966.1923828125$ memory_percent‚92.10488796234131 disk.rbd0.out‚0 disk.dm-1.out‚244.875 disk.sda2.in‚0 disk.rbd1.in‚0 disk.dm-0.in‚ 12772.21875) proc.memory.availableMB‚ 1293.53515625% disk.nvme0n1.out‚60966.1923828125) disk./.usagePercent‚37.43275526900371 disk.sda2.out‚0 disk.sda1.out‚0$ disk./.usageGB‚557.7751045227051 disk.rbd0.in‚0«ÐçY2Å- _wandbruntime‚5963  _runtime‚5963,ËÓÕÇÆÁ % _timestamp‚1.7842957792258837e+09 _runtime‚5963.443046613 _step‚292 & lr-AdamW/pg1‚0.0006473616789548813 & lr-AdamW/pg2‚0.0006473616789548813 trainer/global_step‚3599q¯®ÌÇÆ & lr-AdamW/pg1‚0.0006473616789548813 & lr-AdamW/pg2‚0.0006473616789548813 trainer/global_step‚3599 % _timestamp‚1.7842957792258837e+09 _runtime‚5963.443046613 _step‚292¤)$«'2È- _wandbruntime‚5963  _runtime‚5963ÿ“ï ý'É÷O PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚30602019 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚31122310 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚19437088 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1034444531301516 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚30602013 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1031513982124032 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1034894738984448 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4358081594880 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3374999974794 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 trainer/global_step‚3599 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚61734302 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚31122310 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9230343 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1031068161298432 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚283886201205248 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1034896110547456 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚254 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚279527716982784 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1034894738984448 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1031513982124032 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 _step‚293 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10206745 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1034443161273226 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1031513955950592 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4358116131328 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3380783033856 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3376344575116 % _timestamp‚1.7842957792269554e+09 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚31132283 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚61724323 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚19437342 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1034896110547456 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚283885798577664 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1031513955950592 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚30602019 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚279528085073920 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3382128423424 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648v'(ÊüO YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1031068161298432 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚283886201205248 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1034896110547456 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚254 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚279527716982784 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1034894738984448 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1031513982124032 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 _step‚293 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10206745 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1034443161273226 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1031513955950592 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4358116131328 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3380783033856 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3376344575116 % _timestamp‚1.7842957792269554e+09 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚31132283 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚61724323 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚19437342 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1034896110547456 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚283885798577664 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1031513955950592 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚30602019 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚279528085073920 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3382128423424 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚30602019 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚31122310 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚19437088 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1034444531301516 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚30602013 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1031513982124032 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1034894738984448 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4358081594880 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3374999974794 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 trainer/global_step‚3599 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚61734302 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚31122310 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9230343 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0¥m`2Ë- _wandbruntime‚5964  _runtime‚5964"%6¿ÌN PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚61741007 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚30610225 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚251 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚283964880018944 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1034723336546050 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1031791956680704 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1034721966517648 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚61741007 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3383053466112 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1031791982854144 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚279605571481600 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚30610225 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1031346043076608 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3377268041474 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1035175036320256 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1031791982854144 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚246 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3383053466112 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚61751000 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚19442586 _runtime‚5964.828688278 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1035175036320256 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1031791956680704 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9232886 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚30610219 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1035173664750080 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 N>DeviceStatsMonitor.on_train_batch_end/requestŸ©ûÿH ed_bytes.all.peak‚ 47499471119 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10209449 _step‚294 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚19442335 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚31140775 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10209695 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 % _timestamp‚1.7842957806113179e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚283964477398528 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚30610225 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚251 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚283964880018944 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1034723336546050 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1031791956680704 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1034721966517648 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚61741007 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3383053466112 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1031791982854144 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚279605571481600 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚30610225 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1031346043076608 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3377268041474 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1035175036320256 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1031791982854144 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚246 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3383053466112 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚61751000 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚19442586 _runtime‚5964.828688278 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1035175036320256 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1031791956680704 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9232886 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚30610219 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1035173664750080 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10209449 _step‚294 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚19442335 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚31140775 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10209695 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0¦ñù¹Í2Î- _wandbruntime‚5964  _runtime‚5964”ÒÍ^Ï 1loss/SRqqt1/detection_loss‚1.0391472499616419 /loss/FBt1/detection_loss‚0.6693219044021904 0loss/FRt1/assignment_loss‚1.7642081183624445 /loss/FBt2/detection_loss‚0.5059063059570313 epoch‚8 trainer/global_step‚3599 _runtime‚5964.915378769 _step‚295 % _timestamp‚1.7842957806984499e+09 /loss/FRt1/detection_loss‚0.5432306704681509 1loss/SRqqt2/detection_loss‚0.6559243848351957 %loss/total_loss‚8.813685872755956 0loss/FRt2/assignment_loss‚0.8485835131603418 2loss/SRqqt1/assignment_loss‚0.5503125412327575 2loss/SRqqt2/assignment_loss‚0.5809446249065147 1loss/FBt1/assignment_loss‚0.01658857651184156 2loss/FBt2/assignment_loss‚0.013971868222578523 .loss/FRt2/detection_loss‚1.625546114735267‹ó¡i І .loss/FRt2/detection_loss‚1.625546114735267 1loss/SRqqt1/detection_loss‚1.0391472499616419 /loss/FBt1/detection_loss‚0.6693219044021904 0loss/FRt1/assignment_loss‚1.7642081183624445 /loss/FBt2/detection_loss‚0.5059063059570313 epoch‚8 trainer/global_step‚3599 _runtime‚5964.915378769 _step‚295 % _timestamp‚1.7842957806984499e+09 /loss/FRt1/detection_loss‚0.5432306704681509 1loss/SRqqt2/detection_loss‚0.6559243848351957 %loss/total_loss‚8.813685872755956 0loss/FRt2/assignment_loss‚0.8485835131603418 2loss/SRqqt1/assignment_loss‚0.5503125412327575 2loss/SRqqt2/assignment_loss‚0.5809446249065147 1loss/FBt1/assignment_loss‚0.01658857651184156 2loss/FBt2/assignment_loss‚0.013971868222578523§º,ä’Ñ‚(:‡ éâèÒ–çéø gpu.0.gpu‚1.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚37.0 gpu.0.powerWatts‚109.347' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.8694 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ˆ?ƒôÒ‚(:é éâèÒÕ¡Îù$ disk./.usageGB‚557.7788352966309 disk.rbd0.in‚0 disk.sda1.in‚0 disk.drbd2165.out‚0' disk.nvme0c0n1.out‚60973.2392578125 disk.nvme0n1.in‚ 12772.21875 cpu‚25.56759003075884 network.sent‚ 6.680111e+06 disk.sda2.out‚0' proc.memory.availableMB‚ 1314.484375* disk./.usagePercent‚37.433005644372194 disk.dm-1.in‚0 disk.dm-0.in‚ 12772.21875 disk.sda3.in‚0 disk.sda2.in‚0# proc.memory.rssMB‚ 2804.87890625 disk.rbd0.out‚0" disk.dm-0.out‚60973.2392578125 disk.rbd3.in‚0 disk.rbd1.out‚ 14.3515625$ memory_percent‚91.97702407836914 disk.rbd1.in‚0 disk.dm-1.out‚ 245.40234375 disk.sda3.out‚ 245.40234375 disk.rbd3.out‚0% disk.nvme0n1.out‚60973.2392578125 proc.cpu.threads‚17 network.recv‚ 1.518896e+06! disk.nvme0c0n1.in‚ 12772.21875* proc.memory.percent‚17.119622230529785 disk.sda.in‚0 disk.sda.out‚ 245.40234375 disk.sda1.out‚0 disk.drbd2165.in‚0××)”Ó‚(:‰ øâèÒäõàö gpu.0.gpu‚98.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚332.266' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚66.4532 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0+C{ÆâÔ‚(:× øâèÒÊ Æø disk.rbd3.out‚0 proc.cpu.threads‚17 network.sent‚ 6.742362e+06 disk.rbd0.out‚0 disk.sda3.out‚ 246.0390625 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.rbd3.in‚0 disk.nvme0n1.in‚ 12772.21875* disk./.usagePercent‚37.433116495634515 disk.sda2.in‚0 disk.rbd1.out‚ 14.3515625 disk.rbd0.in‚0 disk.nvme0n1.out‚ 60982.84375 disk.sda2.out‚0 disk.rbd1.in‚0' proc.memory.availableMB‚ 1262.640625# proc.memory.rssMB‚ 2856.24609375 disk.sda.out‚ 246.0390625 disk.dm-0.in‚ 12772.21875 disk.sda1.in‚0 disk.sda3.in‚0 disk.sda1.out‚0$ memory_percent‚92.29345321655273 cpu‚25.574129865035705 network.recv‚ 1.523159e+06 disk.sda.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 246.0390625 disk.dm-0.out‚ 60982.84375! disk.nvme0c0n1.in‚ 12772.21875" disk.nvme0c0n1.out‚ 60982.84375) proc.memory.percent‚17.43314266204834$ disk./.usageGB‚557.7804870605469—ˆ.2Õ- _wandbruntime‚5991  _runtime‚5991ç{åÐÎ=ÖÈ{ 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9937853217124939 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8894179463386536 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2783927619457245 6 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.284855455160141 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9964998960494995 +jet/accuracy_1_of_1‚0.30790627002716064 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9814814329147339 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.29796844720840454 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9003564119338989 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6454166769981384 jet/accuracy_2_of_5‚1 _runtime‚5991.424245264 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9936025738716125 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.4904840588569641 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 particle/accuracy_1_of_4‚1 +particle/specificity‚0.9102353453636169 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.28005290031433105 jet/accuracy_4_of_5‚0 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.31155040860176086 8"Purity/*FRt*SRqqt0FBt/event_purity‚0.273665189743042 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5575000047683716 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8865454792976379 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03311171755194664 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 7 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.1661464422941208 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9962305426597595 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 6 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.640354573726654 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.29641464352607727 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.08642559498548508 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.7163999676704407 *jet/accuracy_2_of_4‚0.7828972935676575 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 jet/accuracy_5_of_5‚0 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.5937231779098511 5 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.28696209192276 particle/accuracy_1_of_1‚1 $ _timestamp‚1.784295807206391e+09 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9814814329147339 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.5407934188842773 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9750880002975464 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.5896425247192383 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.32499998807907104 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.045032840222120285 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.5425496101379395 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9920265674591064 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 7 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.1661464422941208 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9962305426597595 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.2657370865345001 particle/accuracy_2_of_4‚1 6validation_average_jet_accuracy‚0.3485647737979889 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.49257874488830566 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7624430060386658 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 particle/accuracy_1_of_5‚1 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 6 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.579668402671814 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6367136240005493 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9965144395828247 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2870015799999237 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.30968567728996277 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 *jet/accuracy_3_of_4‚0.0942603051662445 8"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.655950129032135 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.8182917833328247 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 +jet/accuracy_3_of_3‚0.07572398334741592 *jet/accuracy_1_of_4‚0.9627400636672974 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.49162882566452026 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.5893091559410095 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.26055723428726196 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7488681077957153 5 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.27142333984375 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7268518209457397 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.3231310248374939 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.22509552538394928 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.8125 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9940198063850403 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 (loss/val_total_loss‚8.27924184283561 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 .particle/accuracy_4_of_4‚0.956527590751648 trainer/global_step‚3599 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9971977472305298 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.26939865946769714 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.4977515637874603 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.29581141471862793 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9949904680252075 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.2777668833732605 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.17353126406669617 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.4664824903011322 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9177461266517639 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.21512256562709808 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8958427906036377 , Purity/0FRt2SRqqt*FBt/FBt_purity‚0.96875 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 'particle/f_score‚0.6491354703903198 +"Purity/2FRt*SRqqt2FBt/event_purity‚0.75 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.3935185372829437 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 7 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.1770598441362381 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 particle/accuracy_3_of_5‚1 epoch‚8 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.10185185074806213 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.34200653433799744 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5018739104270935 8"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.657690167427063 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 *jet/accuracy_1_of_3‚0.9038254022598267 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.08614759892225266 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6063212752342224 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 particle/accuracy_3_of_4‚1 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 jet/accuracy_3_of_5‚1 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5752815008163452 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4532703757286072 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6159922480583191 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7268518209457397 particle/accuracy_1_of_3‚1 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2861701250076294 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 +jet/accuracy_4_of_4‚0.01628045178949833 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.2864583134651184 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.8571428656578064 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.10185185074806213 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6842636466026306 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8644770383834839 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.037462443113327026 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 9"Purity/2FRt*SRqqt1FBt/event_purity‚0.1274999976158142 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_3_of_3‚1 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.8571428656578064 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9490740299224854 (particle/accuracy‚0.8518378138542175 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 7"Purity/1FRt0SRqqt0FBt/event_purity‚0.28696209192276 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.27524444460868835 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7337583899497986 8"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.596236526966095 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.6554280519485474 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.7163999676704407 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3029889166355133 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.5002619028091431 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8958427906036377 particle/accuracy_5_of_5‚0 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.4999527633190155 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.5570265054702759 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.5601423978805542 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9875440001487732 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 jet/accuracy_1_of_5‚1 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_2_of_3‚1 _step‚296 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8933485150337219 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.28060251474380493 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.5375266075134277 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.18760690093040466 , Purity/0FRt2SRqqt1FBt/FBt_purity‚0.96875 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.11538461595773697 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.4977515637874603 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.2977624833583832 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 +particle/sensitivity‚0.6382915377616882 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4981018304824829 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 +jet/accuracy_2_of_3‚0.48069483041763306 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9893404841423035 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5685795545578003 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.17858797311782837 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.5468273758888245 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 particle/accuracy_2_of_5‚1 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.17432861030101776 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9920704960823059 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6426359415054321 /"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.421875 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.5979971289634705 particle/accuracy_2_of_2‚1 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9857958555221558 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9947816729545593 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9484599828720093 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8999621868133545 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.4967537820339203 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.5893699526786804 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.08454369753599167 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.2764943540096283 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.11538461595773697 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5440324544906616 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9922739863395691 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5044757723808289 *jet/accuracy_1_of_2‚0.7602090835571289 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.19304782152175903 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.04481087997555733 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.2606423795223236 +jet/accuracy_2_of_2‚0.12450866401195526 6 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.942307710647583 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6119791269302368 particle/accuracy_4_of_5‚1 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.3350122570991516 particle/accuracy_1_of_2‚1 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9843757152557373 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.5859483480453491 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.07188767194747925 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.5208333134651184 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9894220232963562„Й]×Í{ , Purity/0FRt2Sá\rà¼=Rqqt1FBt/FBt_purity‚0.96875 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.11538461595773697 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.4977515637874603 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.2977624833583832 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 +particle/sensitivity‚0.6382915377616882 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4981018304824829 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 +jet/accuracy_2_of_3‚0.48069483041763306 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9893404841423035 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5685795545578003 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.17858797311782837 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.5468273758888245 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 particle/accuracy_2_of_5‚1 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.17432861030101776 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9920704960823059 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6426359415054321 /"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.421875 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.5979971289634705 particle/accuracy_2_of_2‚1 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9857958555221558 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9947816729545593 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9484599828720093 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8999621868133545 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.4967537820339203 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.5893699526786804 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.08454369753599167 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.2764943540096283 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.11538461595773697 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5440324544906616 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9922739863395691 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5044757723808289 *jet/accuracy_1_of_2‚0.7602090835571289 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.19304782152175903 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.04481087997555733 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.2606423795223236 +jet/accuracy_2_of_2‚0.12450866401195526 6 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.942307710647583 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6119791269302368 particle/accuracy_4_of_5‚1 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.3350122570991516 particle/accuracy_1_of_2‚1 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9843757152557373 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.5859483480453491 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.07188767194747925 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.5208333134651184 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9894220232963562 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9937853217124939 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8894179463386536 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2783927619457245 6 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.284855455160141 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9964998960494995 +jet/accuracy_1_of_1‚0.30790627002716064 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9814814329147339 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.29796844720840454 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9003564119338989 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6454166769981384 jet/accuracy_2_of_5‚1 _runtime‚5991.424245264 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9936025738716125 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.4904840588569641 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 particle/accuracy_1_of_4‚1 +particle/specificity‚0.9102353453636169 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.28005290031433105 jet/accuracy_4_of_5‚0 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.31155040860176086 8"Purity/*FRt*SRqqt0FBt/event_purity‚0.273665189743042 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5575000047683716 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8865454792976379 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03311171755194664 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 7 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.1661464422941208 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9962305426597595 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 6 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.640354573726654 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.29641464352607727 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.08642559498548508 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.7163999676704407 *jet/accuracy_2_of_4‚0.7828972935676575 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 jet/accuracy_5_of_5‚0 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.5937231779098511 5 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.28696209192276 particle/accuracy_1_of_1‚1 $ _timestamp‚1.784295807206391e+09 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9814814329147339 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.5407934188842773 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9750880002975464 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.5896425247192383 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.32499998807907104 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.045032840222120285 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.5425496101379395 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9920265674591064 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 7 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.1661464422941208 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9962305426597595 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.2657370865345001 particle/accuracy_2_of_4‚1 6validation_average_jet_accuracy‚0.3485647737979889 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.49257874488830566 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7624430060386658 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 particle/accuracy_1_of_5‚1 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 6 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.579668402671814 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6367136240005493 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9965144395828247 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2870015799999237 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.30968567728996277 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 *jet/accuracy_3_of_4‚0.0942603051662445 8"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.655950129032135 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.8182917833328247 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 +jet/accuracy_3_of_3‚0.07572398334741592 *jet/accuracy_1_of_4‚0.9627400636672974 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.49162882566452026 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.5893091559410095 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.26055723428726196 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7488681077957153 5 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.27142333984375 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7268518209457397 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.3231310248374939 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.22509552538394928 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.8125 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9940198063850403 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 (loss/val_total_loss‚8.27924184283561 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 .particle/accuracy_4_of_4‚0.956527590751648 trainer/global_step‚3599 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9971977472305298 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.26939865946769714 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.4977515637874603 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.29581141471862793 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9949904680252075 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.2777668833732605 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.17353126406669617 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.4664824903011322 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9177461266517639 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.21512256562709808 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8958427906036377 , Purity/0FRt2SRqqt*FBt/FBt_purity‚0.96875 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 'particle/f_score‚0.6491354703903198 +"Purity/2FRt*SRqqt2FBt/event_purity‚0.75 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.3935185372829437 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 7 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.1770598441362381 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 particle/accuracy_3_of_5‚1 epoch‚8 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.10185185074806213 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.34200653433799744 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5018739104270935 8"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.657690167427063 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 *jet/accuracy_1_of_3‚0.9038254022598267 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.08614759892225266 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6063212752342224 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 particle/accuracy_3_of_4‚1 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 jet/accuracy_3_of_5‚1 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5752815008163452 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4532703757286072 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6159922480583191 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7268518209457397 particle/accuracy_1_of_3‚1 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2861701250076294 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 +jet/accuracy_4_of_4‚0.01628045178949833 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.2864583134651184 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.8571428656578064 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.10185185074806213 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6842636466026306 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8644770383834839 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.037462443113327026 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 9"Purity/2FRt*SRqqt1FBt/event_purity‚0.1274999976158142 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_3_of_3‚1 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.8571428656578064 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9490740299224854 (particle/accuracy‚0.8518378138542175 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 7"Purity/1FRt0SRqqt0FBt/event_purity‚0.28696209192276 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.27524444460868835 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7337583899497986 8"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.596236526966095 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.6554280519485474 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.7163999676704407 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3029889166355133 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.5002619028091431 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8958427906036377 particle/accuracy_5_of_5‚0 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.4999527633190155 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.5570265054702759 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.5601423978805542 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9875440001487732 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 jet/accuracy_1_of_5‚1 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_2_of_3‚1 _step‚296 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8933485150337219 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.28060251474380493 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.5375266075134277 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.18760690093040466¨¸5 žØ‚(:“ ‡ãèÒ ¿Ýö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚40.0 gpu.0.powerWatts‚268.84' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚53.767999999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0CK~ØúÙ‚(:ï ‡ãèÒ¼Æûø network.recv‚ 1.527115e+06* disk./.usagePercent‚37.433401688027665 disk.sda.out‚ 246.28515625 disk.dm-1.in‚0 disk.dm-0.in‚12847.77734375 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.rbd3.out‚0 disk.rbd1.out‚ 14.41796875) proc.memory.percent‚17.24567413330078 disk.rbd0.in‚0 disk.rbd0.out‚0& disk.nvme0c0n1.out‚61745.630859375 disk.sda2.out‚0 disk.sda1.in‚0 disk.drbd2165.out‚0$ disk.nvme0c0n1.in‚12847.77734375$ disk.nvme0n1.out‚61745.630859375$ memory_percent‚82.19349384307861) proc.memory.availableMB‚ 2917.41796875 disk.dm-1.out‚ 246.28515625 cpu‚25.576390399282616 network.sent‚ 6.747929e+06$ disk./.usageGB‚557.7847366333008 disk.sda.in‚0! disk.dm-0.out‚61745.630859375 disk.sda3.out‚ 246.28515625 disk.sda3.in‚0 disk.rbd3.in‚0" disk.nvme0n1.in‚12847.77734375 proc.memory.rssMB‚ 2825.53125 proc.cpu.threads‚17 disk.rbd1.in‚0 disk.sda2.in‚0ò<2•Ú‚(:Š –ãèÒãåäø gpu.0.gpu‚100.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚344.641' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚68.9282 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0æ¿®õÛ‚(:ê –ãèÒÈ´“ú disk.sda1.out‚0 disk.drbd2165.out‚0% disk.nvme0n1.out‚63058.7294921875 proc.memory.rssMB‚ 2825.53125 disk.sda.out‚ 248.328125 disk.rbd3.out‚0 cpu‚25.57594201176562 disk.sda2.in‚0 disk.dm-1.out‚ 248.328125 disk.sda3.in‚0) disk./.usagePercent‚37.43383510878313 disk.sda2.out‚0 disk.rbd1.in‚0 disk.dm-1.in‚0 disk.rbd0.out‚0 proc.cpu.threads‚17 network.recv‚ 1.531495e+06 disk.rbd0.in‚0 disk.dm-0.in‚12847.77734375 disk.sda1.in‚0$ disk.nvme0c0n1.in‚12847.77734375) proc.memory.availableMB‚ 2917.33984375) proc.memory.percent‚17.24567413330078 network.sent‚ 6.805632e+06$ disk./.usageGB‚557.7911949157715 disk.sda.in‚0 disk.rbd3.in‚0' disk.nvme0c0n1.out‚63058.7294921875" disk.nvme0n1.in‚12847.77734375 disk.rbd1.out‚ 14.41796875" disk.dm-0.out‚63058.7294921875 disk.sda3.out‚ 248.328125 disk.drbd2165.in‚0$ memory_percent‚82.19397068023682î™ÔFžÜ‚(:“ ¥ãèÒŒèÞö gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚272.619' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚54.52380000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0APRùÝ‚(:î ¥ãèÒ–×Üø disk.sda3.out‚ 251.09765625' disk.nvme0c0n1.out‚63073.6591796875 disk.sda2.out‚0 disk.sda.out‚ 251.09765625( proc.memory.availableMB‚ 2917.5859375 disk.rbd0.out‚0 disk.sda1.in‚0$ disk.nvme0c0n1.in‚12847.77734375 disk.rbd1.in‚0 disk.rbd3.out‚0 disk.sda2.in‚0 disk.rbd1.out‚ 14.49609375 cpu‚25.575435752840036 disk.sda3.in‚0 disk.drbd2165.in‚0 disk.rbd0.in‚0) proc.memory.percent‚17.24567413330078 disk.sda1.out‚0 disk.drbd2165.out‚0" disk.nvme0n1.in‚12847.77734375$ memory_percent‚82.19246864318848 proc.cpu.threads‚17 network.recv‚ 1.53501e+06$ disk./.usageGB‚557.8008728027344% disk.nvme0n1.out‚63073.6591796875 disk.sda.in‚0 proc.memory.rssMB‚ 2825.53125 disk.dm-1.in‚0" disk.dm-0.out‚63073.6591796875 disk.rbd3.in‚0 disk.dm-0.in‚12847.77734375 disk.dm-1.out‚ 251.09765625 network.sent‚ 6.810103e+06( disk./.usagePercent‚37.4344845998975}gD³ŸÞ‚(:” ´ãèÒ˜Îö gpu.0.gpu‚100.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚339.941' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚67.98819999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0«®€þß‚(:ó ´ãèÒß“Äø disk.sda.in‚0) proc.memory.availableMB‚ 2917.57421875 proc.memory.rssMB‚ 2825.53125 disk.rbd0.in‚0 disk.sda.out‚ 253.73046875) proc.memory.percent‚17.24567413330078" disk.nvme0n1.in‚12847.77734375 cpu‚25.574962587701446) disk./.usagePercent‚37.43444619876736 disk.drbd2165.out‚0# memory_percent‚82.1925401687622 proc.cpu.threads‚17$ disk./.usageGB‚557.8003005981445 disk.rbd0.out‚0 disk.dm-0.in‚12847.77734375 disk.sda3.out‚ 253.73046875$ disk.nvme0c0n1.in‚12847.77734375 network.recv‚ 1.538299e+06# disk.dm-0.out‚63099.76318359375 disk.sda1.in‚0 disk.sda3.in‚0& disk.nvme0n1.out‚63099.76318359375 network.sent‚ 6.815641e+06 disk.dm-1.out‚ 253.73046875 disk.sda1.out‚0 disk.drbd2165.in‚0( disk.nvme0c0n1.out‚63099.76318359375 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd1.out‚ 14.49609375 disk.dm-1.in‚0 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.rbd1.in‚0»“e·”à‚(:‰ ÃãèÒçã‘÷ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚271.478' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.2956 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0“ž0ïúá‚(:ï ÃãèÒɉ¶ù cpu‚25.574482220998746$ disk.nvme0c0n1.in‚12900.48828125 proc.cpu.threads‚17 disk.dm-1.out‚ 255.3671875 disk.sda1.in‚0 disk.sda3.in‚0" disk.nvme0n1.in‚12900.48828125 disk.sda2.out‚0# memory_percent‚82.1953296661377$ disk./.usageGB‚557.8033561706543 disk.dm-1.in‚0 disk.drbd2165.in‚0 disk.rbd3.in‚0 network.sent‚ 6.820791e+06 disk.sda.in‚0# disk.dm-0.out‚63107.55419921875) disk./.usagePercent‚37.43465126080229 disk.rbd1.in‚0 disk.rbd1.out‚ 14.55859375 disk.rbd0.out‚0 disk.drbd2165.out‚0( disk.nvme0c0n1.out‚63107.55419921875 network.recv‚ 1.542043e+06 disk.sda.out‚ 255.3671875 disk.rbd0.in‚0 disk.sda1.out‚0 disk.sda3.out‚ 255.3671875 disk.rbd3.out‚0 disk.sda2.in‚0) proc.memory.percent‚17.24567413330078 disk.dm-0.in‚12900.48828125& disk.nvme0n1.out‚63107.55419921875( proc.memory.availableMB‚ 2917.1171875 proc.memory.rssMB‚ 2825.53125˜Áˆ•2â- _wandbruntime‚6074  _runtime‚6074}î—¿ÆãÀ & lr-AdamW/pg2‚0.0006459919697825261 trainer/global_step‚3649 $ _timestamp‚1.784295890100493e+09 _runtime‚6074.317707751 _step‚297 & lr-AdamW/pg1‚0.0006459919697825261 ©òËäÅ $ _timestamp‚1.784295890100493e+09 _runtime‚6074.317707751 _step‚297 & lr-AdamW/pg1‚0.0006459919697825261 & lr-AdamW/pg2‚0.0006459919697825261 trainer/global_step‚3649©câ2å- _wandbruntime‚6074  _runtime‚6074ËCÊû#æ÷O ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚287875838832128 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3427066219344 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚19746063 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚31144325 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9366416 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1048788184567808 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚238 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚19745820 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4425723755008 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3434286968320 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1049241973891072 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚31144325 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1049241973891072 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚31607049 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1052215225359184 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚62751368 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1052674915469824 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1049242000064512 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3432941578752 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚31144319 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9366411 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3432941578752 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚31617022 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4425689218560 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1049242000064512 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3428410819666 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1052216595387474 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _step‚298 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1052676287032832 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1052674915469824 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚283450517704704 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 OIDeviceStatsMonitor.on_train_batch_start/inactive_splõ¾ðit.large_pool.current‚5 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 trainer/global_step‚3649 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1052676287032832 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚287876241459712 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚31607049 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3432941578752 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚31144319 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9366411 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3432941578752 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚31617022 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4425689218560 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1049242000064512 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3428410819666 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1052216595387474 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _step‚298 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1052676287032832 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1052674915469824 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚283450517704704 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 trainer/global_step‚3649 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1052676287032832 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚287876241459712 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚31607049 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚19746063 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚31144325 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9366416 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1048788184567808 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚238 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚19745820 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4425723755008 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3434286968320 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1049241973891072 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚31144325 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1049241973891072 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚31607049 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1052215225359184 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚62751368 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1052674915469824 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1049242000064512 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 IDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1052953841235456 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚283528004112384 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1049066066345984 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 $ _timestamp‚1.784295891485974e+09 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚246 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9368959 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1052955212805632 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1049520000794624 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10382407 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚62778045 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚19751366 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚19751120 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚241 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 _step‚299 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚31615527 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚287954531546624 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3429334286024 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1049519974621184 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3435212011008 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3433866614272 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚3649 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1052494030603606 _runtime‚6075.703441457 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚31152525 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚283527636021248 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚62778045 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚62768052 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1052955212805632 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚31152525 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚31152525 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1052953841235456 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚283528004112384 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1049066066345984 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 $ _timestamp‚1.784295891485974e+09 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚246 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9368959 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1052955212805632 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1049520000794624 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10382407 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚62778045 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚19751366 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚19751120 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚241 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 _step‚299 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚31615527 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚287954531546624 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3429334286024 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1049519974621184 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3435212011008 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3433866614272 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚3649 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1052494030603606 _runtime‚6075.703441457 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚31152525 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚283527636021248 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚62778045 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚62768052 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1052955212805632 QDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1066621267906048 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1066155551381092 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3474647932774 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1063142010396672 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚31554619 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1066622639469056 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚63597399 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1063142036570112 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9493561 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10516927 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3480602898944 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1066622639469056 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚32032801 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1062682248048640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚31554619 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1066156921409382 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚20010216 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚63587420 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1063142036570112 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9493566 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36633600 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚6152.963048954 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚32042774 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1066621267906048 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1066155551381092 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3474647932774 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1063142010396672 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚31554619 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1066622639469056 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚63597399 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1063142036570112 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9493561 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10516927 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3480602898944 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1066622639469056 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚32032801 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1062682248048640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚31554619 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1066156921409382 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚20010216 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚63587420 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1063142036570112 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9493566 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36633600 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚6152.963048954 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚32042774 IDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3480182544896 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9496109 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9496104 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1063420011126784 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚266 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚32051266 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚20015564 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10519189 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1063420037300224 [MDeviceStatsMonitor.on_traÉ2ÌŽšin_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1066901565241856 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚31562825 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3481527941632 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10519455 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚20015293 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 _step‚303 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚63604104 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1066901565241856 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404717568 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚31562825 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚31562831 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3480182544896 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚63614097 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1063420037300224 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚31562831 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚32051266 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1066900193671680 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4486497617408– ¯" '‚†N REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚20015293 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 _step‚303 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚63604104 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1066901565241856 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404717568 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚31562825 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚31562831 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3480182544896 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚63614097 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1063420037300224 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚31562831 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚32051266 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1066900193671680 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4486497617408 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚32041279 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 % _timestamp‚1.7842959703192523e+09 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚32041279 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3475571399132 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 trainer/global_step‚3699 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚63614097 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚63604104 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚291888454022656 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3480182544896 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9496109 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9496104 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1063420011126784 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚266 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚32051266 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚20015564 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10519189 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1063420037300224 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1066901565241856 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚31562825 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3481527941632 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10519455 BVŬú‚(:ï èäèÒà²ãø# disk.nvme0n1.out‚64266.90234375$ memory_percent‚88.13374042510986 proc.cpu.threads‚17 disk.rbd1.in‚0# proc.memory.rssMB‚ 2825.53515625 disk.rbd1.out‚ 14.93359375 disk.dm-1.out‚ 262.23046875 disk.sda1.in‚0 disk.sda3.in‚0 disk.rbd3.in‚0" disk.nvme0n1.in‚13035.31640625 disk.sda2.in‚0 network.sent‚ 7.027882e+06 disk.rbd0.out‚0 disk.drbd2165.in‚0) proc.memory.percent‚17.24569797515869 disk.dm-1.in‚0 disk.rbd3.out‚0 disk.sda2.out‚0$ disk./.usageGB‚559.0364303588867 disk.sda.in‚0 disk.sda.out‚ 262.23046875 disk.dm-0.out‚64266.90234375 network.recv‚ 1.582501e+06 disk.rbd0.in‚0 disk.drbd2165.out‚0) proc.memory.availableMB‚ 1944.16796875 cpu‚25.569520047722758* disk./.usagePercent‚37.517403904192726 disk.dm-0.in‚13035.31640625$ disk.nvme0c0n1.in‚13035.31640625 disk.sda1.out‚0 disk.sda3.out‚ 262.23046875% disk.nvme0c0n1.out‚64266.90234375«Mçk2- _wandbruntime‚6231  _runtime‚6231£QSçÄ‘¾ _runtime‚ 6231.70893079 _step‚305 & lr-AdamW/pg1‚0.0006431948222781383 & lr-AdamW/pg2‚0.0006431948222781383 trainer/global_step‚3749 # _timestamp‚1.78429604749085e+09$«wÓÉ’Ã & lr-AdamW/pg1‚0.0006431948222781383 & lr-AdamW/pg2‚0.0006431948222781383 trainer/global_step‚3749 # _timestamp‚1.78429604749085e+09 _runtime‚ 6231.70893079 _step‚305±ß:~!2“- _wandbruntime‚6231  _runtime‚6231›É‘dø”öO trainer/global_step‚3749 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚20273476 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 406821888 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚295743034912768 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 _runtime‚6231.709048222 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1077042073075712 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3520885045882 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1077042046902272 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚32468526 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚31964925 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1076576336957440 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚31964919 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚291199158472704 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1080567620342272 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4544244531200 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1077042073075712 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚20273754 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚31964919 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 $ _timestamp‚1.784296047491645e+09 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1080568991905280 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9620716 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1077042046902272 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚32458553 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚32458553 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚64433451 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1080567620342272 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10652765 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚38730752 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1080097247431290 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 _step‚306 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1076576362385408 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10653038 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3525573440000 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚295743441734656 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.curren'½t‚ 1719664640 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚291198790381568 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4544283261952 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚273 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3526918829568 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚32468526 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚31964925 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3525573440000 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9620711 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3519540445560 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1080095877403000 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0‘y(•ûO VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1080095877403000 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚3749 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚20273476 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 406821888 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚295743034912768 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 _runtime‚6231.709048222 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1077042073075712 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3520885045882 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1077042046902272 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚32468526 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚31964925 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1076576336957440 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚31964919 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚291199158472704 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1080567620342272 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4544244531200 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1077042073075712 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚20273754 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚31964919 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 $ _timestamp‚1.784296047491645e+09 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1080568991905280 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9620716 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1077042046902272 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚32458553 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚32458553 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚64433451 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1080567620342272 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10652765 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚38730752 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1080097247431290 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 _step‚306 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1076576362385408 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10653038 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3525573440000 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚295743441734656 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚291198790381568 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4544283261952 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚273 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3526918829568 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚32468526 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚31964925 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3525573440000 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9620711 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3519540445560²ã´Ú2–- _wandbruntime‚6233  _runtime‚6233H Ü’'—N H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3520463911806 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚31973125 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3521808512240 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3526498475520 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚295822100510720 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚64450149 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚31973125 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 _runtime‚6233.095683714 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚31973131 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1080846546107904 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3526498475520 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4545455630336 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚64440156 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3527843872256 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚20278735 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9623259 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚32477018 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚32477018 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 trainer/global_step‚3749 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 % _timestamp‚1.7842960488783436e+09 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4545416906752 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 406814720 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚32467031 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚31973131 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚38723584 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚64440156 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1077320047632384 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1080847917678080 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚32467031 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚20279011 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚295821693696000 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9623254 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1077320073805824 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3527843872256 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚64450149 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚271 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0ÿ¥Ã.Ÿ!˜†N E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚31973125 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3521808512240 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3526498475520 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚295822100510720 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚64450149 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚31973125 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 _runtime‚6233.095683714 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚31973131 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1080846546107904 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3526498475520 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4545455630336 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚64440156 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3527843872256 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚20278735 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9623259 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚32477018 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚32477018 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 trainer/global_step‚3749 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 % _timestamp‚1.7842960488783436e+09 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4545416906752 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 406814720 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚32467031 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚31973131 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚38723584 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚64440156 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1077320047632384 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1080847917678080 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚32467031 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚20279011 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚295821693696000 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9623254 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚107732007380582sŸÒöm4 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3527843872256 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚64450149 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚271 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3520463911806 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0³˜a2™- _wandbruntime‚6233  _runtime‚6233f·¦š€ 0loss/FRt2/assignment_loss‚0.7703458580208749 2loss/SRqqt1/assignment_loss‚0.5755601377787649 1loss/FBt1/assignment_loss‚0.04546826941774958 % _timestamp‚1.7842960489632363e+09 2loss/SRqqt2/assignment_loss‚0.5313627606441781 2loss/FBt2/assignment_loss‚0.034160208545533026 .loss/FRt2/detection_loss‚1.475998907990429 epoch‚9 _runtime‚6233.180175267 0loss/FRt1/assignment_loss‚1.7602336900083269 /loss/FRt1/detection_loss‚0.5352483205129267 0loss/SRqqt1/detection_loss‚1.005213566036124 1loss/SRqqt2/detection_loss‚0.6290117789463667 %loss/total_loss‚8.319333577904672 trainer/global_step‚3749 _step‚308 /loss/FBt1/detection_loss‚0.3805023181913911 /loss/FBt2/detection_loss‚0.5762277618120077‚ð? ›… 0loss/FRt1/assignment_loss‚1.7602336900083269 /loss/FRt1/detection_loss‚0.5352483205129267 0loss/SRqqt1/detection_loss‚1.005213566036124 1loss/SRqqt2/detection_loss‚0.6290117789463667 %loss/total_loss‚8.319333577904672 trainer/global_step‚3749 _step‚308 /loss/FBt1/detection_loss‚0.3805023181913911 /loss/FBt2/detection_loss‚0.5762277618120077 0loss/FRt2/assignment_loss‚0.7703458580208749 2loss/SRqqt1/assignment_loss‚0.5755601377787649 1loss/FBt1/assignment_loss‚0.04546826941774958 % _timestamp‚1.7842960489632363e+09 2loss/SRqqt2/assignment_loss‚0.5313627606441781 2loss/FBt2/assignment_loss‚0.034160208545533026 .loss/FRt2/detection_loss‚1.475998907990429 epoch‚9 _runtime‚6233.180175267´ðSIJ”œ‚(:‰ ÷äèÒ¡éÍö gpu.0.gpu‚93.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚341.292' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚68.2584 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0y7ðý‚(:ò ÷äèÒ—áÂø disk.dm-0.in‚13035.31640625% disk.nvme0n1.out‚64406.6318359375 disk.sda2.in‚0$ memory_percent‚88.13469409942627) disk./.usagePercent‚37.51437507905513) proc.memory.percent‚17.24569797515869 cpu‚25.56915622728885 disk.sda1.out‚0 disk.sda3.in‚0 disk.sda3.out‚ 262.49609375) proc.memory.availableMB‚ 1944.01171875 disk.drbd2165.in‚0 disk.rbd1.out‚ 14.99609375# proc.memory.rssMB‚ 2825.53515625 proc.cpu.threads‚17 network.recv‚ 1.58609e+06 disk.dm-1.in‚0 disk.rbd3.out‚0$ disk.nvme0c0n1.in‚13035.31640625 disk.rbd1.in‚0 network.sent‚ 7.032872e+06 disk.sda1.in‚0 disk.drbd2165.out‚0 disk.rbd3.in‚0' disk.nvme0c0n1.out‚64406.6318359375$ disk./.usageGB‚558.9912986755371 disk.sda.out‚ 262.49609375 disk.rbd0.in‚0" disk.dm-0.out‚64406.6318359375" disk.nvme0n1.in‚13035.31640625 disk.sda.in‚0 disk.dm-1.out‚ 262.49609375 disk.rbd0.out‚0 disk.sda2.out‚0~‘팟ž‚(:” †åèÒÝûãö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚290.567' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚58.113400000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 £1óŸ‚(:è †åèÒµ›ù disk.rbd3.in‚0 disk.sda2.in‚0 disk.sda.in‚0 disk.sda.out‚ 262.65625 disk.drbd2165.out‚0) disk./.usagePercent‚37.51437098293458 disk.rbd0.out‚0 disk.sda1.in‚0 network.recv‚ 1.590713e+06$ disk./.usageGB‚558.9912376403809$ disk.nvme0c0n1.in‚13130.46484375 disk.rbd1.in‚0# proc.memory.rssMB‚ 2825.53515625 cpu‚25.56866642829614 disk.sda3.in‚0% disk.nvme0n1.out‚64415.3505859375' proc.memory.availableMB‚ 1944.046875 disk.rbd3.out‚0 disk.sda2.out‚0 network.sent‚ 7.095905e+06 disk.rbd0.in‚0 disk.sda1.out‚0 disk.sda3.out‚ 262.65625$ memory_percent‚88.13447952270508) proc.memory.percent‚17.24569797515869 disk.drbd2165.in‚0' disk.nvme0c0n1.out‚64415.3505859375 disk.dm-0.in‚13130.46484375" disk.nvme0n1.in‚13130.46484375 disk.rbd1.out‚ 14.99609375 disk.dm-1.in‚0 disk.dm-1.out‚ 262.65625 proc.cpu.threads‚17" disk.dm-0.out‚64415.3505859375ܱ|à“ ‚(:ˆ •åèÒ€Šïö gpu.0.gpu‚11.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚335.402' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚67.0804 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ë6tï¡‚(:ä •åèÒ”·‘ù disk.sda.in‚0 disk.sda3.out‚ 263.00390625 memory_percent‚ 88.134765625" disk.nvme0n1.in‚13130.46484375 disk.drbd2165.in‚0 proc.memory.availableMB‚1944 cpu‚25.568265059431127 network.sent‚ 7.10116e+06 disk.sda2.in‚0 disk.rbd1.out‚ 15.05859375 disk.rbd3.out‚0) disk./.usagePercent‚37.51438173525102 disk.sda.out‚ 263.00390625 disk.sda3.in‚0$ disk.nvme0c0n1.in‚13130.46484375' disk.nvme0c0n1.out‚64418.6533203125 disk.rbd0.in‚0" disk.dm-0.out‚64418.6533203125) proc.memory.percent‚17.24569797515869 network.recv‚ 1.594716e+06# disk./.usageGB‚558.991397857666 disk.dm-1.in‚0 disk.rbd3.in‚0% disk.nvme0n1.out‚64418.6533203125 disk.rbd0.out‚0 disk.dm-0.in‚13130.46484375 disk.sda1.in‚0 disk.rbd1.in‚0 disk.dm-1.out‚ 263.00390625 disk.sda1.out‚0 disk.drbd2165.out‚0# proc.memory.rssMB‚ 2825.53515625 proc.cpu.threads‚17 disk.sda2.out‚0þ ÊŸ¢‚(:” ¤åèÒÚâáö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚274.743' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚54.948600000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0R­·,ö£‚(:ë ¤åèÒ궰ø proc.cpu.threads‚17' disk.nvme0c0n1.out‚64423.5615234375% disk.nvme0n1.out‚64423.5615234375 disk.rbd1.out‚ 15.05859375$ disk./.usageGB‚558.9912071228027 disk.rbd0.in‚0 disk.rbd0.out‚0" disk.nvme0n1.in‚13130.46484375# proc.memory.availableMB‚1944.25 disk.sda.in‚0$ disk.nvme0c0n1.in‚13130.46484375 disk.sda1.in‚0 disk.dm-1.out‚ 263.1015625" disk.dm-0.out‚64423.5615234375 disk.rbd1.in‚0 network.recv‚ 1.597742e+06 disk.sda1.out‚0 disk.sda3.in‚0 disk.drbd2165.out‚0 network.sent‚ 7.10615e+06 disk.sda2.in‚0 cpu‚25.567792145793817* disk./.usagePercent‚37.514368934874305 disk.dm-1.in‚0 disk.sda3.out‚ 263.1015625 disk.drbd2165.in‚0 disk.rbd3.out‚0$ memory_percent‚88.13323974609375 disk.sda.out‚ 263.1015625 disk.dm-0.in‚13130.46484375 disk.rbd3.in‚0 disk.sda2.out‚0# proc.memory.rssMB‚ 2825.53515625) proc.memory.percent‚17.24569797515869QùÑ¥•¤‚(:Š ³åèÒðíÝö gpu.0.gpu‚100.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚336.569' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚67.3138 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0=æ´ö¥‚(:ë ³åèÒÕݾø disk.rbd1.out‚ 15.12109375 disk.rbd0.out‚0 disk.dm-0.in‚13130.46484375" disk.nvme0n1.in‚13130.46484375 network.sent‚ 7.111147e+06) disk./.usagePercent‚37.51438096722842 disk.sda.in‚0 disk.rbd0.in‚0 disk.sda3.in‚0$ memory_percent‚88.13474178314209 network.recv‚ 1.601189e+06 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.rbd3.in‚0$ disk.nvme0c0n1.in‚13130.46484375 cpu‚25.567353641463797 disk.dm-1.out‚ 263.3984375 disk.sda3.out‚ 263.3984375% disk.nvme0c0n1.out‚64426.91015625) proc.memory.availableMB‚ 1944.00390625) proc.memory.percent‚17.24569797515869 disk.sda.out‚ 263.3984375 disk.sda1.in‚0 disk.drbd2165.out‚0 disk.rbd3.out‚0 disk.rbd1.in‚0$ disk./.usageGB‚558.9913864135742 disk.dm-1.in‚0 disk.dm-0.out‚64426.91015625 disk.sda2.in‚0 disk.sda2.out‚0# proc.memory.rssMB‚ 2825.53515625 proc.cpu.threads‚17# disk.nvme0n1.out‚64426.91015625]§^2¦- _wandbruntime‚6310  _runtime‚6310t€gǧÁ & lr-AdamW/pg2‚0.0006417675636016629 trainer/global_step‚3799 % _timestamp‚1.7842961262344143e+09 _runtime‚6310.451652766 _step‚309 & lr-AdamW/pg1‚0.0006417675636016629]¶ŠüÌ¨Æ & lr-AdamW/pg1‚0.0006417675636016629 & lr-AdamW/pg2‚0.0006417675636016629 trainer/global_step‚3799 % _timestamp‚1.7842961262344143e+09 _runtime‚6310.451652766 _step‚309µ¸ ‘å2©- _wandbruntime‚6310  _runtime‚6310w…øuý'ª÷O P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚32375225 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9747861 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚299676655056384 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1094513972778496 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1094037573453198 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3565777558668 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚20537985 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3567122158990 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3571889370624 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1094513972778496 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1090942109581312 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9747866 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1094515344341504 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚32884305 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚295073478856704 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚32375219 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚38730752 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3573234760192 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 _runtime‚6310.455944529 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1090942109581312 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚299676248234496 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚295073110765568 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1094513972778496 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1094037573453198 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3565777558668 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚20537985 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3567122158990 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3571889370624 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1094513972778496 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1090942109581312 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9747866 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1094515344341504 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚32884305 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚295073478856704 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚32375219 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚38730752 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3573234760192 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 _runtime‚6310.455944529 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1090942109581312 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚299676248234496 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚295073110765568 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚65286201 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 _step‚311 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3572814406144 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9750409 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1094794270114304 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚295150597173248 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚32902770 UCDeviceStatsMonitor.on_tÌx}>!rain_batch_end/active_bytes.small_pool.freed‚ 3572814406144 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1094315008669330 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1094794270114304 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚32383431 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚299755315075584 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7842961276204033e+09 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚65286201 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚20543176 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9750404 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1090748307644416 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3574159802880 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1090748333072384 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3568045625348 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1091220110311424 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10792772 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4604349811200 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚38723584 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4604311087616 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1094792898544128 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚261 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚266 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚299754908260864 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚295150965264384 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1094792898544128 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚261 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚266 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚299754908260864 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚295150965264384 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚65286201 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 _step‚311 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3572814406144 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9750409 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1094794270114304 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚295150597173248 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚32902770 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3572814406144 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1094315008669330 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1094794270114304 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚32383431 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚299755315075584 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7842961276204033e+09 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚65286201 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚20543176 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9750404 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1090748307644416 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434·€îÃb2¯- _wandbruntime‚6311  _runtime‚6311y.+°‚ 0loss/FRt1/assignment_loss‚1.7857541826887917 1loss/SRqqt2/detection_loss‚0.6144069565595338 %loss/total_loss‚8.626231668358834 epoch‚9 .loss/FBt2/detection_loss‚0.522324511150439 2loss/SRqqt1/assignment_loss‚0.6087439530430772 2loss/FBt1/assignment_loss‚0.008365099322407492 /loss/FRt2/detection_loss‚1.5265967842117323 /loss/FBt1/detection_loss‚0.8047685164555505 trainer/global_step‚3799 % _timestamp‚1.7842961277068293e+09 _runtime‚6311.923752435 3loss/FBt2/assignment_loss‚0.0026691834720015237 0loss/FRt2/assignment_loss‚0.7925355412177185 2loss/SRqqt2/assignment_loss‚0.5165355337073189 /loss/FRt1/detection_loss‚0.5209019641777475 0loss/SRqqt1/detection_loss‚0.922629442352515 _step‚312’ã°« ±‡ /loss/FRt1/detection_loss‚0.5209019641777475 0loss/SRqqt1/detection_loss‚0.922629442352515 _step‚312 0loss/FRt1/assignment_loss‚1.7857541826887917 1loss/SRqqt2/detection_loss‚0.6144069565595338 %loss/total_loss‚8.626231668358834 epoch‚9 .loss/FBt2/detection_loss‚0.522324511150439 2loss/SRqqt1/assignment_loss‚0.6087439530430772 2loss/FBt1/assignment_loss‚0.008365099322407492 /loss/FRt2/detection_loss‚1.5265967842117323 /loss/FBt1/detection_loss‚0.8047685164555505 trainer/global_step‚3799 % _timestamp‚1.7842961277068293e+09 _runtime‚6311.923752435 3loss/FBt2/assignment_loss‚0.0026691834720015237 0loss/FRt2/assignment_loss‚0.7925355412177185 2loss/SRqqt2/assignment_loss‚0.5165355337073189¸ÿ©Fž²‚(:“ ÂåèÒßÈÇ÷ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚284.135' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚56.82699999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0š.J³‚(:ø ÂåèÒཚø network.recv‚ 1.604448e+06 disk.sda3.out‚ 264.55859375" disk.nvme0n1.in‚13195.07421875 disk.sda2.in‚0) proc.memory.availableMB‚ 1944.82421875 disk.rbd0.in‚0# disk.dm-0.out‚64433.10595703125& disk.nvme0n1.out‚64433.10595703125 disk.sda.out‚ 264.55859375 disk.sda1.out‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 264.55859375 disk.sda3.in‚0$ disk.nvme0c0n1.in‚13195.07421875 disk.sda2.out‚0 disk.sda.in‚0$ memory_percent‚88.12973499298096 proc.cpu.threads‚17 disk.dm-0.in‚13195.07421875 disk.sda1.in‚0 disk.drbd2165.out‚0 disk.rbd3.in‚0 disk.rbd1.in‚0 cpu‚25.566920503922884 network.sent‚ 7.116188e+06* disk./.usagePercent‚37.011534808586646 disk.rbd1.out‚ 15.12109375$ disk./.usageGB‚551.4986152648926 disk.rbd3.out‚0( disk.nvme0c0n1.out‚64433.10595703125# proc.memory.rssMB‚ 2825.53515625) proc.memory.percent‚17.24569797515869 disk.rbd0.out‚0 disk.drbd2165.in‚0|Ëð ”´‚(:‰ ÑåèÒ±à÷ gpu.0.gpu‚90.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚292.107' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚58.4214 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0l¼öÎøµ‚(:í ÑåèÒœ¼›ø disk.dm-1.in‚0 disk.dm-0.in‚13195.20703125! disk.dm-0.out‚64438.541015625 disk.rbd3.out‚0$ disk.nvme0c0n1.in‚13195.20703125# proc.memory.rssMB‚ 2825.53515625 proc.cpu.threads‚17 disk.sda2.out‚0 disk.drbd2165.out‚0 network.recv‚ 1.609362e+06) disk./.usagePercent‚37.01177750372911$ disk./.usageGB‚551.5022315979004" disk.nvme0n1.in‚13195.20703125$ disk.nvme0n1.out‚64438.541015625 disk.rbd1.in‚0) proc.memory.percent‚17.24569797515869& disk.nvme0c0n1.out‚64438.541015625 disk.dm-1.out‚ 265.44140625 disk.sda3.out‚ 265.44140625 disk.sda1.out‚0 disk.rbd3.in‚0$ memory_percent‚88.12828063964844 disk.sda.out‚ 265.44140625 disk.sda3.in‚0 cpu‚25.566526346124355 network.sent‚ 7.179236e+06 disk.rbd1.out‚ 15.18359375 disk.drbd2165.in‚0% proc.memory.availableMB‚ 1945.0625 disk.sda2.in‚0 disk.sda.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda1.in‚06#Nž¶‚(:“ àåèÒ¡ãÏö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚294.864' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚58.97279999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¿…æ™ù·‚(:î àåèÒÐÃÐù$ disk./.usageGB‚551.5063018798828 disk.drbd2165.out‚0 disk.sda.in‚0 disk.sda.out‚ 265.81640625% proc.memory.availableMB‚ 1945.3125 disk.rbd3.out‚0" disk.nvme0n1.in‚13195.20703125 disk.sda2.in‚0) proc.memory.percent‚17.24569797515869 cpu‚25.566161581443104 disk.sda1.out‚0 disk.sda3.out‚ 265.81640625 disk.drbd2165.in‚0 disk.rbd3.in‚0% disk.nvme0n1.out‚64452.2255859375 network.sent‚ 7.18424e+06 disk.rbd0.out‚0 disk.dm-0.in‚13195.20703125 disk.sda1.in‚0' disk.nvme0c0n1.out‚64452.2255859375 disk.sda2.out‚0 disk.rbd1.out‚ 15.18359375 disk.dm-1.in‚0 network.recv‚ 1.61274e+06 disk.sda3.in‚0 disk.rbd1.in‚0$ memory_percent‚88.12675476074219" disk.dm-0.out‚64452.2255859375$ disk.nvme0c0n1.in‚13195.20703125) disk./.usagePercent‚37.01205066376814 disk.rbd0.in‚0 disk.dm-1.out‚ 265.81640625# proc.memory.rssMB‚ 2825.53515625 proc.cpu.threads‚171Õà“¸‚(:ˆ ïåèÒ¹‘öö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚280.815' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚56.163 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Gó-—ù¹‚(:î ïåèÒ¸¡€ù$ disk.nvme0n1.out‚64455.556640625 disk.rbd0.out‚0 disk.dm-0.in‚13195.20703125 disk.rbd3.out‚0 disk.sda.out‚ 266.20703125 disk.sda1.out‚0" disk.nvme0n1.in‚13195.20703125 disk.sda2.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 15.24609375! disk.dm-0.out‚64455.556640625 disk.drbd2165.in‚0 disk.drbd2165.out‚0$ disk.nvme0c0n1.in‚13195.20703125) proc.memory.percent‚17.24569797515869# disk./.usageGB‚551.506233215332 disk.sda.in‚0 disk.dm-1.out‚ 266.20703125 disk.sda3.in‚0# proc.memory.rssMB‚ 2825.53515625 network.sent‚ 7.189288e+06 network.recv‚ 1.61627e+06 disk.sda2.out‚0 disk.sda1.in‚0 disk.sda3.out‚ 266.20703125 cpu‚25.565697919976763 disk.dm-1.in‚0& disk.nvme0c0n1.out‚64455.556640625$ memory_percent‚88.12685012817383 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.rbd3.in‚0' proc.memory.availableMB‚ 1945.296875* disk./.usagePercent‚37.012046055632524“2V瓺‚(:ˆ þåèÒìÖÌö gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚38.0 gpu.0.powerWatts‚110.125' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.025 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0:Ækð»‚(:å þåèÒø network.recv‚ 1.619617e+06 disk.rbd0.out‚0$ memory_percent‚88.12830448150635 cpu‚25.565198903485513( disk.nvme0c0n1.out‚64460.88232421875 disk.sda2.out‚0 disk.sda1.out‚0! disk.nvme0c0n1.in‚ 13266.84375 disk.nvme0n1.in‚ 13266.84375& disk.nvme0n1.out‚64460.88232421875 disk.sda.in‚0 disk.sda1.in‚0) proc.memory.percent‚17.24569797515869) disk./.usagePercent‚37.01204451958732 disk.dm-0.in‚ 13266.84375 disk.rbd3.out‚0) proc.memory.availableMB‚ 1945.05859375 proc.cpu.threads‚17 disk.rbd1.out‚ 15.24609375 disk.sda.out‚ 266.40625 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.sda3.in‚0 disk.drbd2165.in‚0# proc.memory.rssMB‚ 2825.53515625 network.sent‚ 7.194454e+06$ disk./.usageGB‚551.5062103271484 disk.rbd1.in‚0 disk.dm-1.out‚ 266.40625 disk.drbd2165.out‚0 disk.sda3.out‚ 266.40625 disk.rbd3.in‚0 disk.sda2.in‚0# disk.dm-0.out‚64460.88232421875¨*H2¼- _wandbruntime‚6389  _runtime‚6389S“—]ǽÁ & lr-AdamW/pg1‚0.0006403213034604379 & lr-AdamW/pg2‚0.0006403213034604379 trainer/global_step‚3849 % _timestamp‚1.7842962047898724e+09 _runtime‚6389.007059161 _step‚313«~ÍuÌ¾Æ _runtime‚6389.007059161 _step‚313 & lr-AdamW/pg1‚0.0006403213034604379 & lr-AdamW/pg2‚0.0006403213034604379 trainer/global_step‚3849 % _timestamp‚1.7842962047898724e+09¹Ü]–2¿- _wandbruntime‚6389  _runtime‚6389Ä<³\À÷O LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚20802406 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1108460325214720 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1107976529446816 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚32785525 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1108460325214720 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚32785519 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3613359272098 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10927390 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0¿'£àä G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚32785519 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚33320030 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10927136 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚33310057 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚33320030 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚298947431149568 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1104364540203008 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3612014671776 % _timestamp‚1.7842962047909694e+09 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1104842119913472 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1104842146086912 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3618205301248 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚33310057 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3618205301248 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4662140256768 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1104842146086912 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚259 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚20802147 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚303609939497472 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1104842119913472 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3619550690816 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9875011 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚254 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6ئõ(ÁüO E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚66095576 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1108461696777728 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3619550690816 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚303609536869888 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚298947799240704 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1107977899475106 _runtime‚6389.008315563 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1108461696777728 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1108460325214720 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1107976529446816 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚32785525 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1108460325214720 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚32785519 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3613359272098 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10927390 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚32785519 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚33320030 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10927136 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚33310057 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚33320030 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚298947431149568 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1104364540203008 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3612014671776 % _timestamp‚1.7842962047909694e+09 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1104842119913472 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1104842146086912 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3618205301248 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚33310057 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3618205301248 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4662140256768 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1104842146086912 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚259 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚20802147 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚303609939497472 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1104842119913472 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3619550690816 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9875011 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚254 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚20802406 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚66122253 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3620475733504 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3620475733504 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚32793725 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚299025285648384 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚299024917557248 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36626432 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 % _timestamp‚1.7842962061774042e+09 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚32793731 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚20807419 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚303688207490560 trainer/global_step‚3849 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚33318535 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1105120120643584 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚33328522 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10929865 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 _step‚315 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1105120146817024 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚66122253 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9877554 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1104642421981184 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚255 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚66112260 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4663289933312 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1108255334691238 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1104642396553216 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚66112260 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚20807674 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚66112260 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4663289933312 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1108255334691238 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1104642396553216 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚66112260 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚20807674 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚66122253 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3620475733504 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3620475733504 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚32793725 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚299025285648384 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚299024917557248 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36626432 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 % _timestamp‚1.7842962061774042e+09 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚32793731 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚20807419 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚303688207490560 trainer/global_step‚3849 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚33318535 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1105120120643584 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚33328522 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10929865 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 _step‚315 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1105120146817024 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚66122253 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9877554 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1104642421981184 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚255 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1122406677650944 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚66941607 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 $ _timestamp‚1.784296283669244e+09 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚21066094 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3664521231872 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚307543650118144 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚307543245393408 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10002161 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚66931628 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4721530493440 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3665866621440 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚33195825 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚302821751533568 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1118742182592512 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1118742182592512 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3665866621440 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1118258603683840 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚33735809 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚33745782 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1121918225497014 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚3899 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚33195825 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3664521231872 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1122406677650944 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚269 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1121916855468724 _step‚318 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1118742156419072 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚33735809 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1118258629111808 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚302822119624704 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1122408049213952 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚264 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚33195819 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4721493859840 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820dSâx“"ÙýO YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4721493859840 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10002166 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11063933 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 _runtime‚6467.886568419 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36633600 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1122406677650944 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚66941607 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 $ _timestamp‚1.784296283669244e+09 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚21066094 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3664521231872 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚307543650118144 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚307543245393408 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10002161 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚66931628 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4721530493440 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3665866621440 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚33195825 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚302821751533568 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1118742182592512 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1118742182592512 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3665866621440 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1118258603683840 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚33735809 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚33745782 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1121918225497014 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚3899 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚33195825 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3664521231872 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1122406677650944 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚269 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1121916855468724 _step‚318 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 ]HDeviceStatsMonitor.on_train_batÃEPpch_start/allocated_bytes.large_pool.freed‚1118742156419072 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚33735809 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1118258629111808 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚302822119624704 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1122408049213952 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚264 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚33195819 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820¾7…2Ú- _wandbruntime‚6469  _runtime‚6469èûáM 'ÛƒN REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1122685603416576 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚66958305 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3665446267392 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚302899237941248 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1119020183322624 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚66948312 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚254 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 trainer/global_step‚3899 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚259 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚66958305 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21071375 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10004709 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4722718860800 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚319 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚33204025 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3666791664128 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1119020183322624 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚66948312 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1122686974986752 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1122197030741548 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚33754274 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4722682234368 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚33204031 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3660519851564 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21071375 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10004709 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4722718860800 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚319 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚33204025 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3666791664128 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1119020183322624 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚66948312 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1122686974986752 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1122197030741548 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚33754274 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4722682234368 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚33204031 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3660519851564 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3665446267392 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚302899237941248 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1119020183322624 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚66948312 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚254 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 trainer/global_step‚3899 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚259 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚66958305 EDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1136353030087168 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1136354401650176 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1132642192924672 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚21330562 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404724736 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3710837162496 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1135858551518922 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚33606125 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4780920078848 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚34161561 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚21330819 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚257 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1136354401650176 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚34171534 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1136353030087168 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3712182552064 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 trainer/global_step‚3949 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _runtime‚6546.432967459 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚34161561 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚33606119 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚311476991996416 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3710837162496 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚67777659 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11201503 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1132152718020608 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36633600 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3712182552064 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1132642192924672 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚33606125 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1132152692592640 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648µwä²(ïþO E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚33606125 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4780920078848 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚34161561 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚21330819 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚257 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1136354401650176 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚34171534 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1136353030087168 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3712182552064 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 trainer/global_step‚3949 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _runtime‚6546.432967459 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚34161561 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚33606119 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚311476991996416 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3710837162496 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚67777659 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11201503 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1132152718020608 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36633600 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3712182552064 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1132642192924672 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚33606125 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1132152692592640 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚33606119 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10129316 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1136353030087168 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1136354401650176 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1132642192924672 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚21330562 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404724736 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3710837162496 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1135858551518922ˆõ¥2ð- _wandbruntime‚6547  _runtime‚6547I–fô 'ñƒN LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚67794357 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚34170039 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚33614325 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1136137356763456 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10131859 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚311555674041856 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚34170039 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3713107594752 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3711762198016 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3706756964672 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3705412364238 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10131854 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚67794357 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚33614331 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 _step‚323 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1132920219828224 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚21336104 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21335853 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11204245 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1136135986735054 trainer/global_step‚3949 _runtime‚6547.819128392«á’²JòˆN QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3711762198016 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3706756964672 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3705412364238 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10131854 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚67794357 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚33614331 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 _step‚323 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1132920219828224 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚21336104 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21335853 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11204245 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1136135986735054 trainer/global_step‚3949 _runtime‚6547.819128392 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚67794357 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚34170039 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚33614325 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1136137356763456 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10131859 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚311555674041856 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚34170039 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3713107594752 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1150299382523392 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1146046781501440 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚34587313 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚21595357 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 _runtime‚6625.161410173 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1150300754086400 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10256461 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11338634 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1149798877540830 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3757153093120 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚315410777048576 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1150299382523392 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3758498482688 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚34016419 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1146542255603712 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚68613711 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚21595095 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3752070611422 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 trainer/global_step‚3999 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚34597286 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 _step‚326 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1150300754086400 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚310570392301568 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404724736 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3758498482688 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚262 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11338891 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚257 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚34587313 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚34016425 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4840384747008 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1146046806929408 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚315411181773312 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4840421380608 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1146542255603712 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚310570760392704¸AE˜…þO L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚34016419 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1146542255603712 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚68613711 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚21595095 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3752070611422 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 trainer/global_step‚3999 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚34597286 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 _step‚326 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1150300754086400 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚310570392301568 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404724736 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3758498482688 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚262 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11338891 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚257 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚34587313 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚34016425 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4840384747008 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1146046806929408 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚315411181773312 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4840421380608 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1146542255603712 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚310570760392704 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36633600 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 % _timestamp‚1.7842964409440975e+09 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚34016419 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚34597286 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚68603732 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1146542229430272 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚Õ·ñ4l  47545124352 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1150299382523392 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1146046781501440 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚34587313 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚21595357 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 _runtime‚6625.161410173 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1150300754086400 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10256461 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11338634 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1149798877540830 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3757153093120 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚315410777048576 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1150299382523392 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3758498482688ÆÑ)’ù2†- _wandbruntime‚6626  _runtime‚6626µqæö '‡ƒN XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚21600623 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10259009 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3751649477346 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚68630409 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36626432 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚68620416 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3752994077780 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚248 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚310647878709248 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1150579679859200 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4841571253760 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21600375 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1150579679859200 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3758078128640 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1146820256333824 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚34024631 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7842964423290308e+09 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚34595791 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚34024625 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚310648246800384 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚243 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚315489854680576 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3759423525376 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404717568 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚34605778 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3759423525376 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚34605778 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11341371 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1146324688707584 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚68620416 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚315489449963008 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1150578308289024 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1146820230160384 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚68630409 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚34595791 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10259004 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11341614 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1146324663279616 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4841607880192 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434ÎØ('ˆˆN XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚21600623 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10259009 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3751649477346 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚68630409 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36626432 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚68620416 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3752994077780 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚248 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚310647878709248 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1150579679859200 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4841571253760 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21600375 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1150579679859200 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3758078128640 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1146820256333824 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚34024631 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7842964423290308e+09 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚34595791 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚34024625 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚310648246800384 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚243 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚315489854680576 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3759423525376 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404717568 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚34605778 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3759423525376 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚34605778 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11341371 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1146324688707584 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚68620416 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚315489449963008 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1150578308289024 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1146820230160384 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚68630409 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚34595791 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10259004 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11341614 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1146324663279616 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4841607880192 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434Ç4…B2‰- _wandbruntime‚6626  _runtime‚6626ù#U%Š " _timestamp‚1.7842964424162e+09 2loss/SRqqt1/assignment_loss‚0.5537235219968546 trainer/global_step‚3999 _step‚328 3loss/SRqqt2/assignment_loss‚0.47760865112356615 /loss/FBt1/detection_loss‚0.7980989895494921 _runtime‚6626.633154211 0loss/FRt2/assignment_loss‚0.7499668032145329 2loss/FBt2/assignment_loss‚0.027654963682693517 1loss/SRqqt2/detection_loss‚0.7153454318949654 0loss/FBt2/detection_loss‚0.49646928887009006 %loss/total_loss‚8.475729077733513 /loss/FRt1/assignment_loss‚1.694719738557563 2loss/FBt1/assignment_loss‚0.006791062739916194 /loss/FRt1/detection_loss‚0.5540296979882174 /loss/FRt2/detection_loss‚1.5833122942471605 1loss/SRqqt1/detection_loss‚0.8180086338684613 epoch‚9ž—^5 ‹† " _timestamp‚1.7842964424162e+09 2loss/SRqqt1/assignment_loss‚0.5537235219968546 trainer/global_step‚3999 _step‚328 3loss/SRqqt2/assignment_loss‚0.47760865112356615 /loss/FBt1/detection_loss‚0.7980989895494921 _runtime‚6626.633154211 0loss/FRt2/assignment_loss‚0.7499668032145329 2loss/FBt2/assignment_loss‚0.027654963682693517 1loss/SRqqt2/detection_loss‚0.7153454318949654 0loss/FBt2/detection_loss‚0.49646928887009006 %loss/total_loss‚8.475729077733513 /loss/FRt1/assignment_loss‚1.694719738557563 2loss/FBt1/assignment_loss‚0.006791062739916194 /loss/FRt1/detection_loss‚0.5540296979882174 /loss/FRt2/detection_loss‚1.5833122942471605 1loss/SRqqt1/detection_loss‚0.8180086338684613 epoch‚9Èåˆ×”Œ‚(:‰ ýçèÒåó»ø gpu.0.gpu‚97.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚46.0 gpu.0.powerWatts‚333.229' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚66.6458 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‰²;‚(:õ ýçèÒâ”áù network.recv‚ 1.682124e+06) disk./.usagePercent‚37.01809525765909 disk.sda2.out‚0 disk.rbd0.out‚0 disk.sda3.in‚0 disk.rbd3.in‚0$ memory_percent‚87.92860507965088! proc.memory.rssMB‚ 2805.828125 network.sent‚ 7.49021e+06$ disk./.usageGB‚551.5963706970215 disk.rbd0.in‚0 disk.dm-0.in‚13582.38671875 disk.rbd3.out‚0* proc.memory.percent‚17.125415802001953 disk.rbd1.in‚0 disk.dm-1.in‚0# disk.dm-0.out‚64626.76611328125 cpu‚25.559493633200525 disk.sda2.in‚0 disk.rbd1.out‚ 15.80859375 disk.dm-1.out‚ 271.36328125) proc.memory.availableMB‚ 1977.77734375& disk.nvme0n1.out‚64626.76611328125 disk.sda3.out‚ 271.36328125 disk.sda1.in‚0 disk.drbd2165.in‚0 disk.drbd2165.out‚0$ disk.nvme0c0n1.in‚13582.38671875 proc.cpu.threads‚17" disk.nvme0n1.in‚13582.38671875 disk.sda.in‚0 disk.sda1.out‚0( disk.nvme0c0n1.out‚64626.76611328125 disk.sda.out‚ 271.36328125?}HüŸŽ‚(:” ŒèèÒ¾Õãö gpu.0.gpu‚100.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚328.649' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚65.72980000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ž½»ðñ‚(:æ ŒèèÒËŒÒø cpu‚25.56547978057647 disk.sda2.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0! disk.nvme0c0n1.in‚ 13582.40625 network.sent‚ 7.552725e+06) disk./.usagePercent‚37.01854147879127$ disk./.usageGB‚551.6030197143555 disk.dm-0.in‚ 13582.40625 disk.drbd2165.in‚0 disk.rbd3.out‚0 disk.rbd1.in‚0) proc.memory.availableMB‚ 1977.69921875% disk.nvme0n1.out‚64637.7236328125 disk.sda2.out‚0 disk.rbd1.out‚ 15.80859375 disk.sda.in‚0 disk.sda.out‚ 271.7421875 disk.sda3.in‚0 disk.sda3.out‚ 271.7421875 disk.rbd3.in‚0 disk.nvme0n1.in‚ 13582.40625! proc.memory.rssMB‚ 2805.828125* proc.memory.percent‚17.125415802001953 disk.sda1.in‚0 disk.sda1.out‚0 network.recv‚ 1.686985e+06 disk.dm-1.in‚0 disk.dm-1.out‚ 271.7421875 proc.cpu.threads‚17" disk.dm-0.out‚64637.7236328125$ memory_percent‚87.92908191680908 disk.drbd2165.out‚0' disk.nvme0c0n1.out‚64637.7236328125K/Y‰2- _wandbruntime‚6653  _runtime‚6653Ÿ·:jU‘ºz 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6438866257667542 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6646307110786438 (particle/accuracy‚0.8533451557159424 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.22546795010566711 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.10564686357975006 +jet/accuracy_1_of_1‚0.31211933493614197 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.2550812363624573 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6602698564529419 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7165558934211731 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.8571428656578064 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.10106713324785233 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 particle/accuracy_2_of_5‚1 particle/accuracy_5_of_5‚1 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5735049843788147 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9855788350105286 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5534561276435852 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.20237943530082703 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9230769276618958 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.8125 % _timestamp‚1.7842964688601334e+09 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.10084304213523865 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9989176988601685 ("Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.2464267909526825 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5147492289543152 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.27406421303749084 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7431517243385315 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5517676472663879 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.5522007346153259 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9973499774932861 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9943124651908875 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.5499304533004761 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9490740299224854 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.48271769285202026 7"Purity/0FRt*SRqqt2FBt/event_purity‚0.91717928647995 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 8"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.541368305683136 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.2664880156517029 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.10185185074806213 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.2086639404296875 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.46990740299224854 particle/accuracy_1_of_1‚1 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 6 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.106284961104393 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7128143906593323 )jet/accuracy_1_of_4‚0.969353973865509 particle/accuracy_3_of_3‚1 . Purity/2FRt0SRqqt1FBt/FRt_purity‚ 0.6484375 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.2510177195072174 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.12265437096357346 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 +jet/accuracy_3_of_3‚0.07205024361610413 trainer/global_step‚3999 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.5913459062576294 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.41155797243118286 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.12046954780817032 +"Purity/2FRt*SRqqt2FBt/event_purity‚0.75 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9825049638748169 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4763512909412384 & Purity/1FRt1SRqqt2FBt/FBt_purity‚1 *jet/accuracy_1_of_3‚0.9056432843208313 (loss/val_total_loss‚8.54274617082814 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.5987155437469482 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9950499534606934 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6457609534263611 7"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.71783846616745 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 9"Purity‘òë(/2FRt2SRqqt0FBt/SRqqt_purity‚0.7775208353996277 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.27264466881752014 ("Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 6validation_average_jet_accuracy‚0.3532482981681824 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8333333134651184 *jet/accuracy_2_of_3‚0.5157615542411804 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9992441534996033 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.20890460908412933 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.15416666865348816 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3098028898239136 ."Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.40625 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.24972547590732574 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6490515470504761 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6405903100967407 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6534193754196167 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9894220232963562 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.49633553624153137 particle/accuracy_1_of_4‚1 epoch‚9 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9744219779968262 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6434960961341858 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9872110486030579 _step‚329 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt0SRqqt0FBt/event_purity‚0.0530613474547863 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8330309987068176 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6526750922203064 *particle/specificity‚0.912312924861908 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.6189253926277161 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9858131408691406 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.08994698524475098 particle/accuracy_1_of_2‚1 particle/accuracy_2_of_3‚1 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.1538461595773697 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8761975169181824 8"Purity/0FRt2SRqqt*FBt/event_purity‚0.266854852437973 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.07099030911922455 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.24697616696357727 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9993098974227905 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 +jet/accuracy_3_of_4‚0.09882503747940063 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9950039386749268 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8426114916801453 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9989176988601685 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7698791027069092 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.26674771308898926 jet/accuracy_4_of_5‚0 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.2064545899629593 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8343537449836731 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 *jet/accuracy_1_of_2‚0.7642673254013062 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5538626909255981 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5554166436195374 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2717617452144623 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5842108726501465 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.21819861233234406 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.5865319967269897 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5534561276435852 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.2875623106956482 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5248671174049377 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6473684906959534 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.2550812363624573 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8426114916801453 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2281591296195984 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9907875061035156 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2851789593696594 particle/accuracy_4_of_5‚1 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.2308899611234665 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.5449487566947937 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9383158683776855 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.028035424649715424 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.08994698524475098 +jet/accuracy_4_of_4‚0.01628045178949833 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 _runtime‚6653.077884141 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.23504000902175903 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.03723840042948723 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 /particle/accuracy_3_of_4‚0.9968254566192627 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.8571428656578064 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3273499011993408 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.04360930994153023 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 /particle/accuracy_4_of_4‚0.9418520331382751 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5930823683738708 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.6231171488761902 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.22141392529010773 8"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.657841682434082 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.7775208353996277 particle/accuracy_2_of_2‚1 particle/accuracy_2_of_4‚1 particle/accuracy_1_of_5‚1 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9932670593261719 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.2383299171924591 & Purity/1FRt1SRqqt*FBt/FBt_purity‚1 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 jet/accuracy_2_of_5‚1 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3138848841190338 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5625954866409302 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.19209732115268707 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 +jet/accuracy_2_of_2‚0.13006246089935303 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.5984832048416138 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7387437224388123 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 particle/accuracy_1_of_3‚1 ."Purity/2FRt0SRqqt1FBt/event_purity‚0.34375 particle/accuracy_3_of_5‚1 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8521090149879456 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 *jet/accuracy_2_of_4‚0.7602911591529846 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6192900538444519 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.10185185074806213 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.4097222089767456 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.1538461595773697 jet/accuracy_3_of_5‚1 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8526076078414917 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9952685236930847 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.032827336341142654 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.6615086793899536 jet/accuracy_1_of_5‚1 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 6 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.996185839176178 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9031957983970642 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8333333134651184 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.26345202326774597 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9981656074523926 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 'particle/f_score‚0.6512658596038818 jet/accuracy_5_of_5‚0 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.33084872364997864 6 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.992041826248169 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5812678933143616 6 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.997619092464447 +particle/sensitivity‚0.6377257704734802{NE=’¿z 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5538626909255981 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5554166436195374 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2717617452144623 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5842108726501465 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.21819861233234406 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.5865319967269897 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5534561276435852 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.2875623106956482 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5248671174049377 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6473684906959534 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.2550812363624573 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8426114916801453 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2281591296195984 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9907875061035156 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2851789593696594 particle/accuracy_4_of_5‚1 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.2308899611234665 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.5449487566947937 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9383158683776855 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.028035424649715424 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.08994698524475098 +jet/accuracy_4_of_4‚0.01628045178949833 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 _runtime‚6653.077884141 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.23504000902175903 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.03723840042948723 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 /particle/accuracy_3_of_4‚0.9968254566192627 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.8571428656578064 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3273499011993408 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.04360930994153023 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 /particle/accuracy_4_of_4‚0.9418520331382751 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5930823683738708 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.6231171488761902 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.22141392529010773 8"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.657841682434082 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.7775208353996277 particle/accuracy_2_of_2‚1 particle/accuracy_2_of_4‚1 particle/accuracy_1_of_5‚1 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9932670593261719 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.2383299171924591 & Purity/1FRt1SRqqt*FBt/FBt_purity‚1 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 jet/accuracy_2_of_5‚1 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3138848841190338 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5625954866409302 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.19209732115268707 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 +jet/accuracy_2_of_2‚0.13006246089935303 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.5984832048416138 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7387437224388123 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 particle/accuracy_1_of_3‚1 ."Purity/2FRt0SRqqt1FBt/event_purity‚0.34375 particle/accuracy_3_of_5‚1 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8521090149879456 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 *jet/accuracy_2_of_4‚0.7602911591529846 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6192900538444519 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.10185185074806213 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.4097222089767456 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.1538461595773697 jet/accuracy_3_of_5‚1 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8526076078414917 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9952685236930847 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.032827336341142654 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.6615086793899536 jet/accuracy_1_of_5‚1 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 6 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.996185839176178 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9031957983970642 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8333333134651184 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.26345202326774597 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9981656074523926 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 'particle/f_score‚0.6512658596038818 jet/accuracy_5_of_5‚0 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.33084872364997864 6 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.992041826248169 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5812678933143616 6 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.997619092464447 +particle/sensitivity‚0.6377257704734802 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6438866257667542 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6646307110786438 (particle/accuracy‚0.8533451557159424 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.22546795010566711 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.10564686357975006 +jet/accuracy_1_of_1‚0.31211933493614197 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.2550812363624573 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6602698564529419 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7165558934211731 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.8571428656578064 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.10106713324785233 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 particle/accuracy_2_of_5‚1 particle/accuracy_5_of_5‚1 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5735049843788147 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9855788350105286 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5534561276435852 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.20237943530082703 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9230769276618958 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.8125 % _timestamp‚1.7842964688601334e+09 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.10084304213523865 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9989176988601685 ("Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.2464267909526825 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5147492289543152 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.27406421303749084 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7431517243385315 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5517676472663879 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.5522007346153259 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9973499774932861 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9943124651908875 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.5499304533004761 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9490740299224854 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.48271769285202026 7"Purity/0FRt*SRqqt2FBt/event_purity‚0.91717928647995 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 8"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.541368305683136 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.2664880156517029 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.10185185074806213 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.2086639404296875 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.46990740299224854 particle/accuracy_1_of_1‚1 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 6 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.106284961104393 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7128143906593323 )jet/accuracy_1_of_4‚0.969353973865509 particle/accuracy_3_of_3‚1 . Purity/2FRt0SRqqt1FBt/FRt_purity‚ 0.6484375 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.2510177195072174 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.12265437096357346 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 +jet/accuracy_3_of_3‚0.07205024361610413 trainer/global_step‚3999 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.5913459062576294 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.41155797243118286 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.12046954780817032 +"Purity/2FRt*SRqqt2FBt/event_purity‚0.75 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9825049638748169 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4763512909412384 & Purity/1FRt1SRqqt2FBt/FBt_purity‚1 *jet/accuracy_1_of_3‚0.9056432843208313 (loss/val_total_loss‚8.54274617082814 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.5987155437469482 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9950499534606934 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6457609534263611 7"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.71783846616745 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.7775208353996277 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.27264466881752014 ("Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 6validation_average_jet_accuracy‚0.3532482981681824 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8333333134651184 *jet/accuracy_2_of_3‚0.5157615542411804 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9992441534996033 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.20890460908412933 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.15416666865348816 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3098028898239136 ."Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.40625 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.24972547590732574 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6490515470504761 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6405903100967407 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6534193754196167 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9894220232963562 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.49633553624153137 particle/accuracy_1_of_4‚1 epoch‚9 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9744219779968262 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6434960961341858 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9872110486030579 _step‚329 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt0SRqqt0FBt/event_purity‚0.0530613474547863 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8330309987068176 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6526750922203064 *particle/specificity‚0.912312924861908 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.6189253926277161 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9858131408691406 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.08994698524475098 particle/accuracy_1_of_2‚1 particle/accuracy_2_of_3‚1 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.1538461595773697 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8761975169181824 8"Purity/0FRt2SRqqt*FBt/event_purity‚0.266854852437973 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.07099030911922455 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.24697616696357727 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9993098974227905 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 +jet/accuracy_3_of_4‚0.09882503747940063 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9950039386749268 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8426114916801453 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9989176988601685 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7698791027069092 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.26674771308898926 jet/accuracy_4_of_5‚0 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.2064545899629593 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8343537449836731 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 *jet/accuracy_1_of_2‚0.7642673254013062É»Û ò”“‚(:‰ ›èèÒ“–ßö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚40.0 gpu.0.powerWatts‚286.551' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚57.3102 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0*#ý”‚(:ò ›èèÒÙÅù network.recv‚ 1.690658e+06 disk.rbd0.out‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd3.out‚0$ memory_percent‚84.03465747833252 cpu‚25.568545912876782 disk.rbd0.in‚0( disk.nvme0c0n1.out‚65538.72216796875 disk.nvme0n1.in‚ 13582.640625) disk./.usagePercent‚37.01871658794469 disk.rbd1.out‚ 15.93359375# disk.dm-0.out‚65538.72216796875) proc.memory.availableMB‚ 2615.76171875# proc.memory.rssMB‚ 2826.58203125 disk.dm-1.out‚ 272.21484375 disk.dm-0.in‚ 13582.640625 disk.drbd2165.in‚0 disk.rbd3.in‚0 proc.cpu.threads‚17 disk.drbd2165.out‚0 disk.sda.out‚ 272.21484375 disk.sda2.out‚0 disk.sda3.out‚ 272.21484375" disk.nvme0c0n1.in‚ 13582.640625& disk.nvme0n1.out‚65538.72216796875* proc.memory.percent‚17.252087593078613 network.sent‚ 7.558274e+06$ disk./.usageGB‚551.6056289672852 disk.rbd1.in‚0 disk.sda.in‚0 disk.dm-1.in‚0 disk.sda3.in‚0 disk.sda2.in‚0./Ò‘•‚(:† ªèèÒÈÞ…ø gpu.0.gpu‚6.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚330.385' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚66.077 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0aæ~Qî–‚(:ã ªèèÒÐͪø" disk.nvme0n1.in‚13674.44140625 disk.sda.out‚272.75 disk.rbd0.in‚0 disk.sda3.in‚0 disk.drbd2165.in‚0' proc.memory.availableMB‚ 2615.234375 proc.cpu.threads‚17 disk.rbd1.in‚0# disk.dm-0.out‚66865.09326171875( disk.nvme0c0n1.out‚66865.09326171875* proc.memory.percent‚17.252087593078613 disk.rbd1.out‚ 15.93359375 disk.sda.in‚0 disk.dm-0.in‚13674.44140625 disk.sda1.out‚0 disk.rbd3.in‚0 disk.sda2.in‚0 disk.dm-1.out‚272.75 disk.rbd3.out‚0$ memory_percent‚84.03787612915039 network.recv‚ 1.694942e+06 disk.rbd0.out‚0 disk.sda1.in‚0$ disk.nvme0c0n1.in‚13674.44140625 cpu‚25.56817492132258 network.sent‚ 7.615639e+06 disk.dm-1.in‚0 disk.sda3.out‚272.75# proc.memory.rssMB‚ 2826.58203125) disk./.usagePercent‚37.01900997257893 disk.drbd2165.out‚0& disk.nvme0n1.out‚66865.09326171875$ disk./.usageGB‚551.6100006103516 disk.sda2.out‚0º·;ÙŸ—‚(:” ¹èèÒ¬¬—÷ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚267.174' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚53.434799999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0~‹]k(:ä ¹èèÒžéõø disk.rbd3.out‚0& disk.nvme0c0n1.out‚66872.689453125 disk.sda2.out‚0 disk.rbd1.in‚0 disk.sda.out‚ 272.953125# disk.nvme0c0n1.in‚ 13674.4609375$ memory_percent‚84.03816223144531# proc.memory.rssMB‚ 2826.58203125 network.recv‚ 1.698785e+06 disk.rbd1.out‚ 15.99609375 proc.cpu.threads‚17) disk./.usagePercent‚37.01945798576385$ disk./.usageGB‚551.6166763305664 disk.dm-0.in‚ 13674.4609375% proc.memory.availableMB‚ 2615.1875* proc.memory.percent‚17.252087593078613 cpu‚25.56771990134121 disk.rbd0.out‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.drbd2165.out‚0$ disk.nvme0n1.out‚66872.689453125 disk.sda.in‚0 disk.dm-1.out‚ 272.953125 disk.sda3.out‚ 272.953125 disk.rbd3.in‚0! disk.nvme0n1.in‚ 13674.4609375 disk.sda2.in‚0 network.sent‚ 7.620256e+06 disk.sda3.in‚0! disk.dm-0.out‚66872.689453125 disk.dm-1.in‚0 disk.rbd0.in‚0 pœ×ž™‚(:“ ÈèèÒ½öÍö gpu.0.gpu‚39.0 gpu.0.memory‚13, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚338.745' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚67.74900000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0³ÚÓQñš‚(:æ ÈèèÒŽ÷±ø network.recv‚ 1.701835e+06$ disk./.usageGB‚551.6239242553711% disk.nvme0c0n1.out‚66882.35546875 disk.rbd1.in‚0# proc.memory.rssMB‚ 2826.58203125 cpu‚25.567350230654764* disk./.usagePercent‚37.019944400078906 disk.rbd3.out‚0 disk.rbd1.out‚ 15.99609375( proc.memory.availableMB‚ 2615.2734375 network.sent‚ 7.625784e+06 disk.sda3.in‚0 disk.drbd2165.out‚0# disk.nvme0n1.out‚66882.35546875$ memory_percent‚84.03763771057129 disk.sda.in‚0 disk.rbd0.out‚0 disk.sda2.in‚0 disk.sda2.out‚0 proc.cpu.threads‚17 disk.dm-1.in‚0 disk.dm-0.out‚66882.35546875 disk.drbd2165.in‚0# disk.nvme0c0n1.in‚ 13674.6953125! disk.nvme0n1.in‚ 13674.6953125 disk.sda.out‚ 273.640625 disk.dm-0.in‚ 13674.6953125* proc.memory.percent‚17.252087593078613 disk.dm-1.out‚ 273.640625 disk.rbd0.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda3.out‚ 273.640625 disk.rbd3.in‚0ì/Ùž›‚(:“ ×èèÒäÝÛö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚262.054' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚52.41079999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0*²±œ‚(:ó ×èèÒ´²¹ø) disk./.usagePercent‚37.02050096045835" disk.nvme0n1.in‚13674.91015625 disk.sda3.out‚ 274.0859375* proc.memory.percent‚17.252087593078613 cpu‚25.5669033606563 network.sent‚ 7.630784e+06( disk.nvme0c0n1.out‚66896.51904296875 disk.sda.out‚ 274.0859375 disk.dm-1.in‚0 disk.dm-0.in‚13674.91015625 disk.sda1.out‚0 disk.rbd3.out‚0) proc.memory.availableMB‚ `·|M2615.27734375 disk.rbd0.out‚0 proc.cpu.threads‚17 network.recv‚ 1.705136e+06 disk.sda2.out‚0 disk.sda.in‚0 disk.sda3.in‚0 disk.rbd3.in‚0& disk.nvme0n1.out‚66896.51904296875 disk.dm-1.out‚ 274.0859375# disk.dm-0.out‚66896.51904296875 disk.sda1.in‚0 disk.drbd2165.in‚0 disk.rbd1.out‚ 16.05859375 disk.rbd0.in‚0# proc.memory.rssMB‚ 2826.58203125$ disk./.usageGB‚551.6322174072266 disk.sda2.in‚0$ memory_percent‚84.03761386871338$ disk.nvme0c0n1.in‚13674.91015625 disk.rbd1.in‚0 disk.drbd2165.out‚0'}딂(:‰ æèèÒÁ™Êö gpu.0.gpu‚99.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚333.229' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚66.6458 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Þæέæž‚(:Û æèèÒø¯ø$ disk./.usageGB‚551.6392593383789 disk.sda2.in‚0 disk.rbd1.in‚0 disk.sda.in‚0* proc.memory.percent‚17.252087593078613 disk.rbd1.out‚ 16.05859375 disk.rbd0.in‚0' proc.memory.availableMB‚ 2615.265625 disk.sda3.out‚274.25 disk.rbd3.in‚0% disk.nvme0n1.out‚66909.5966796875 disk.dm-1.out‚274.25 proc.cpu.threads‚17 network.recv‚ 1.708301e+06 disk.drbd2165.in‚0 disk.rbd3.out‚0 disk.sda.out‚274.25* disk./.usagePercent‚37.020973550366556" disk.nvme0c0n1.in‚ 13764.703125 disk.nvme0n1.in‚ 13764.703125 disk.sda2.out‚0 disk.dm-1.in‚0$ memory_percent‚84.03768539428711 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd2165.out‚0 disk.rbd0.out‚0" disk.dm-0.out‚66909.5966796875 disk.sda3.in‚0# proc.memory.rssMB‚ 2826.58203125 cpu‚25.56653349920327 network.sent‚ 7.635824e+06' disk.nvme0c0n1.out‚66909.5966796875 disk.dm-0.in‚ 13764.703125‰ ¹ß2Ÿ- _wandbruntime‚6736  _runtime‚6736€´\iÇ Á trainer/global_step‚4049 % _timestamp‚1.7842965518568332e+09 _runtime‚6736.074022418 _step‚330 & lr-AdamW/pg1‚0.0006343481242403624 & lr-AdamW/pg2‚0.0006343481242403624&ÒEÌ¡Æ & lr-AdamW/pg2‚0.0006343481242403624 trainer/global_step‚4049 % _timestamp‚1.7842965518568332e+09 _runtime‚6736.074022418 _step‚330 & lr-AdamW/pg1‚0.0006343481242403624ÊÍDÓ2¢- _wandbruntime‚6736  _runtime‚6736'ÉÝý'£÷O PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚34558731 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1167569571598498 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3810657027584 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1163766804770816 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚34558731 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚243 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚319400575949312 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚69630777 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚32439296 _runtime‚6736.078902281 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 # _timestamp‚1.78429655185793e+09 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚35082025 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1168079559008768 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚34558725 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1167570941626788 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3802792255650 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1163766779342848 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10392534 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚314492824932352 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1164270273544192 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11510195 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 400530432 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3804136855972 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1168079559008768 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚69640756 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10392529 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚34558725 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4907783456256 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚4049 _step‚331 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚35082025 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚21902729 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3810657027584 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1168080930571776 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008l|(¤üO VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 400530432 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3804136855972 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1168079559008768 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚69640756 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10392529 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚34558725 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4907783456256 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚4049 _step‚331 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚35082025 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚21902729 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3810657027584 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1168080930571776 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚34558731 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1167569571598498 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3810657027584 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1163766804770816 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚34558731 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚243 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚319400575949312 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚69630777 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚32439296 _runtime‚6736.078902281 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 # _timestamp‚1.78429655185793e+09 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚35082025 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1168079559008768 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚34558725 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1167570941626788 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3802792255650 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1163766779342848 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10392534 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚314492824932352 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1164270273544192 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11510195 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504ËæWNÛ2¥- _wandbruntime‚6737  _runtime‚6737ZôëÄ%¦‚N UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1168359856344576 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10395072 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚222 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1164548274274304 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3810236673536 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21907692 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚35090517 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚319479275887104 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚34566937 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3811582070272 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1167849746871322 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1164548274274304 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚ 6737.45995616 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 400523264 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4908996979200 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3803715721896 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10395077 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚34566931 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1168358484774400 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚35090517 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3805060322330 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 trainer/global_step‚4049 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚69657454 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚69647461 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚227 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11512620 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 % _timestamp‚1.7842965532422962e+09 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11512842 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚69657454 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1164548248100864 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚314570311340032 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚32432128 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4908964547072 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚21907919 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1164044686548992 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 _step‚332 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1167848376842920 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1164044661121024 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21907692 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚35090517 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚319479275887104 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚34566937 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3811582070272 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1167849746871322 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1164548274274304 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚ 6737.45995616 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 400523264 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4908996979200 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3803715721896 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10395077 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚34566931 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1168358484774400 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚35090517 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3805060322330 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 trainer/global_step‚4049 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚69657454 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚69647461 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚227 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11512620 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 % _timestamp‚1.7842965532422962e+09 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11512842 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚69657454 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1164548248100864 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚314570311340032 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚32432128 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4908964547072 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚21907919 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1164044686548992 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 _step‚332 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1167848376842920 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1164044661121024 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 QDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 _runtime‚6815.233635623 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚34969031 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚34969025 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚22167138 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚323335162927616 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚318367145316352 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1177660868251648 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1182025911444992 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1181511267648696 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚34969031 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚70476808 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚261 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 $ _timestamp‚1.784296631016329e+09 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1182027283008000 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚22167399 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚34969025 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚70466829 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1182025911444992 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚35497804 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10519684 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1182025911444992 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1181511267648696 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚34969031 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚70476808 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚261 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 $ _timestamp‚1.784296631016329e+09 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1182027283008000 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚22167399 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.pea$lÁk‚ 49071259648 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚34969025 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚70466829 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1182025911444992 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚35497804 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10519684 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 _runtime‚6815.233635623Ï??pd2»- _wandbruntime‚6816  _runtime‚6816ËàéT '¼ƒN [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1178448284606464 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚70483513 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚35516269 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1178448284606464 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3857898000896 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚34977231 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚323414254326272 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚35506282 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10522222 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1182306208780800 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚255 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1181790072893230 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7842966324021237e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3856552604160 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 trainer/global_step‚4099 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚70493506 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3851297435438 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚323413851705856 _runtime‚6816.619464216 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3857898000896 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11650373 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚22172345 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1177938750029824 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1178448310779904 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚250 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11650123 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1182306208780800 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1182304837210624 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1181788702864828 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚34977231 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4969254511104 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚323413851705856 _runtime‚6816.619464216 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3857898000896 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11650373 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚22172345 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1177938750029824 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1178448310779904 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚250 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11650123 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1182306208780800 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1182304837210624 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1181788702864828 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚34977231 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4969254511104 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚70483513 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚35516269 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1178448284606464 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3857898000896 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚34977231 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚323414254326272 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚35506282 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10522222 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1182306208780800 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚255 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1181790072893230 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7842966324021237e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3856552604160 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 trainer/global_step‚4099 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚70493506 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0У¦Û»2¾- _wandbruntime‚6816  _runtime‚6816Ó¢¿ _runtime‚6816.706098055 0loss/FRt1/assignment_loss‚1.7156949059005357 .loss/FRt1/detection_loss‚0.555630118653751 .loss/FRt2/detection_loss‚1.443937879761063 1loss/SRqqt1/detection_loss‚1.0090151662036921 epoch‚10 2loss/SRqqt2/assignment_loss‚0.5011582126819712 2loss/FBt1/assignment_loss‚0.029516857188392958 $loss/total_loss‚8.80841575114034 0loss/FRt2/assignment_loss‚0.6956165905192528 trainer/global_step‚4099 % _timestamp‚1.7842966324891677e+09 _step‚337 2loss/SRqqt1/assignment_loss‚0.5999935526614919 1loss/FBt2/assignment_loss‚0.01704639372943486 1loss/SRqqt2/detection_loss‚0.6292960240038115 /loss/FBt1/detection_loss‚1.1398870542723065 0loss/FBt2/detection_loss‚0.471622995564636440,¤Š À† 0loss/FRt2/assignment_loss‚0.6956165905192528 trainer/global_step‚4099 % _timestamp‚1.7842966324891677e+09 _step‚337 2loss/SRqqt1/assignment_loss‚0.5999935526614919 1loss/FBt2/assignment_loss‚0.01704639372943486 1loss/SRqqt2/detection_loss‚0.6292960240038115 /loss/FBt1/detection_loss‚1.1398870542723065 0loss/FBt2/detection_loss‚0.47162299556463644 _runtime‚6816.706098055 0loss/FRt1/assignment_loss‚1.7156949059005357 .loss/FRt1/detection_loss‚0.555630118653751 .loss/FRt2/detection_loss‚1.443937879761063 1loss/SRqqt1/detection_loss‚1.0090151662036921 epoch‚10 2loss/SRqqt2/assignment_loss‚0.5011582126819712 2loss/FBt1/assignment_loss‚0.029516857188392958 $loss/total_loss‚8.80841575114034ÑÂ’Á‚(:‡ ÀéèÒàÈ–÷ gpu.0.gpu‚94.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚320.72' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚64.144 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ô<¤õù‚(:î ÀéèÒôˆ”ø disk.drbd2165.in‚0! disk.nvme0n1.in‚ 13955.9765625 disk.rbd1.out‚ 16.24609375! disk.dm-0.out‚67010.115234375 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd3.out‚0 disk.dm-1.out‚ 283.62109375 disk.rbd3.in‚0 disk.sda2.out‚0* disk./.usagePercent‚37.023499832714435 disk.rbd0.out‚0 disk.sda2.in‚0 disk.rbd1.in‚0 disk.sda.out‚ 283.62109375 proc.cpu.threads‚17$ disk.nvme0n1.out‚67010.115234375# proc.memory.rssMB‚ 2826.58203125$ disk./.usageGB‚551.6769027709961 disk.dm-0.in‚ 13955.9765625 disk.sda3.in‚0 disk.sda3.out‚ 283.62109375& disk.nvme0c0n1.out‚67010.115234375 network.sent‚ 7.72327e+06 network.recv‚ 1.729259e+06 disk.rbd0.in‚0 disk.drbd2165.out‚0 disk.sda.in‚0$ memory_percent‚89.96548652648926( proc.memory.availableMB‚ 1644.0546875 cpu‚25.564127545378785 disk.dm-1.in‚0# disk.nvme0c0n1.in‚ 13955.9765625* proc.memory.percent‚17.252087593078613°F¤ÒôÂ(:é ÏéèÒıø$ memory_percent‚89.96551036834717# proc.memory.rssMB‚ 2826.58203125 disk.rbd0.out‚0 disk.dm-0.out‚67022.85546875 disk.sda3.in‚0 disk.rbd3.in‚0 disk.sda.in‚0 disk.drbd2165.out‚0# disk.nvme0n1.out‚67022.85546875 cpu‚25.563732892879482 proc.cpu.threads‚17# disk.nvme0c0n1.in‚ 13956.2109375% disk.nvme0c0n1.out‚67022.85546875) proc.memory.availableMB‚ 1644.05078125* proc.memory.percent‚17.252087593078613 disk.dm-1.in‚0 disk.dm-1.out‚ 284.3671875 disk.dm-0.in‚ 13956.2109375 network.recv‚ 1.734046e+06 disk.drbd2165.in‚0 disk.rbd1.out‚ 16.30859375 disk.sda1.in‚0 disk.sda2.in‚0 disk.rbd1.in‚0 network.sent‚ 7.786224e+06$ disk./.usageGB‚551.6838188171387 disk.sda.out‚ 284.3671875 disk.sda3.out‚ 284.3671875 disk.rbd3.out‚0! disk.nvme0n1.in‚ 13956.2109375) disk./.usagePercent‚37.02396397437401 disk.rbd0.in‚0 disk.sda1.out‚0 disk.sda2.out‚0ÉÆ^1’Ä‚(:‡ ÏéèÒÅ›ø gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚260.99' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.198 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 Í¡bžÅ‚(:“ ÞéèÒÁ÷Òö gpu.0.gpu‚99.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚332.989' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚66.59779999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0gÅyWÆ‚(:ç ÞéèÒ’—Çø disk.rbd1.out‚ 16.30859375 disk.sda.in‚0 disk.sda.out‚ 284.578125 network.recv‚ 1.73704e+06# disk./.usageGB‚551.694263458252 disk.dm-1.out‚ 284.578125 disk.dm-0.in‚13956.42578125 disk.sda3.in‚0 disk.drbd2165.out‚0 disk.rbd0.in‚0 disk.sda2.in‚0# proc.memory.rssMB‚ öw•¨›2826.58203125 disk.rbd0.out‚0 disk.sda1.in‚0$ disk.nvme0c0n1.in‚13956.42578125 disk.dm-0.out‚67034.01171875 disk.drbd2165.in‚0 disk.rbd3.out‚0 disk.rbd3.in‚0# disk.nvme0n1.out‚67034.01171875) proc.memory.availableMB‚ 1644.50390625* proc.memory.percent‚17.252087593078613 proc.cpu.threads‚17) disk./.usagePercent‚37.02466492300277 disk.dm-1.in‚0 disk.sda1.out‚0% disk.nvme0c0n1.out‚67034.01171875" disk.nvme0n1.in‚13956.42578125 disk.sda2.out‚0 cpu‚25.563408655909058 disk.rbd1.in‚0$ memory_percent‚89.96274471282959 network.sent‚ 7.791972e+06 disk.sda3.out‚ 284.578125$I4E”Ç‚(:‰ íéèÒàÅ¥÷ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚265.067' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.0134 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0RXpûÈ‚(:ð íéèÒ佋ù" disk.dm-0.out‚67044.7724609375 disk.sda3.out‚ 284.84765625 disk.rbd3.in‚0 disk.rbd3.out‚0# disk.nvme0c0n1.in‚ 14029.4921875' disk.nvme0c0n1.out‚67044.7724609375 disk.rbd0.out‚0 network.recv‚ 1.740661e+06) disk./.usagePercent‚37.02521329114111 disk.sda3.in‚0$ memory_percent‚89.96305465698242' proc.memory.availableMB‚ 1644.453125 disk.dm-0.in‚ 14029.4921875* proc.memory.percent‚17.252087593078613 cpu‚25.562940092048603 disk.rbd1.out‚ 16.37109375 disk.sda.in‚0 disk.sda1.in‚0 disk.dm-1.in‚0 disk.sda2.in‚0 disk.rbd1.in‚0 disk.sda1.out‚0% disk.nvme0n1.out‚67044.7724609375 disk.sda2.out‚0 disk.sda.out‚ 284.84765625 disk.drbd2165.out‚0 proc.cpu.threads‚17 network.sent‚ 7.796431e+06$ disk./.usageGB‚551.7024345397949 disk.drbd2165.in‚0! disk.nvme0n1.in‚ 14029.4921875# proc.memory.rssMB‚ 2826.58203125 disk.dm-1.out‚ 284.84765625 disk.rbd0.in‚0[÷ö•É‚(:… üéèÒ‹¡›÷ gpu.0.gpu‚90.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚234.9' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚46.98 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0­äsûÊ‚(:ð üéèÒï¾ø network.sent‚ 7.801467e+06 network.recv‚ 1.744268e+06 disk.sda.out‚ 284.99609375 disk.sda3.out‚ 284.99609375 disk.sda2.out‚0 proc.cpu.threads‚17 disk.drbd2165.out‚0# memory_percent‚89.9646520614624 disk.sda.in‚0& disk.nvme0c0n1.out‚67055.279296875 disk.dm-0.in‚14029.91015625! disk.dm-0.out‚67055.279296875* proc.memory.percent‚17.252087593078613 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.sda1.in‚0$ disk.nvme0c0n1.in‚14029.91015625" disk.nvme0n1.in‚14029.91015625 disk.sda2.in‚0# proc.memory.rssMB‚ 2826.58203125 disk.rbd0.out‚0 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.rbd3.out‚0 disk.rbd1.out‚ 16.37109375) disk./.usagePercent‚37.02567896884589 disk.dm-1.out‚ 284.99609375 disk.sda3.in‚0 disk.rbd3.in‚0$ disk.nvme0n1.out‚67055.279296875 disk.rbd1.in‚0) proc.memory.availableMB‚ 1644.19140625 cpu‚25.56262714598654$ disk./.usageGB‚551.7093734741211²FÐ2Ë- _wandbruntime‚6894  _runtime‚6894¯ä`ÇÌÁ & lr-AdamW/pg1‚0.0006312499983979048 & lr-AdamW/pg2‚0.0006312499983979048 trainer/global_step‚4149 % _timestamp‚1.7842967097908108e+09 _runtime‚6894.007973116 _step‚338ºH;æÌÍÆ & lr-AdamW/pg2‚0.0006312499983979048 trainer/global_step‚4149 % _timestamp‚1.7842967097908108e+09 _runtime‚6894.007973116 _step‚338 & lr-AdamW/pg1‚0.0006312499983979048ÒŒ+í2Î- _wandbruntime‚6894  _runtime‚6894Þ?PSÿ'ÏùO KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚322241465700352 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 % _timestamp‚1.7842967097918627e+09 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3903288888832 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5027791897088 trainer/global_step‚4149 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚35923556 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚251 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11785252 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1195973635444224 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 _runtime‚6894.009186388 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚35923556 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3901943499264 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1191554982588416 _step‚339 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1195972263881216 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1195972263881216 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1191554957160448 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚71302881 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚327269660225024 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3896611082188 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3895266481866 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1192070320381952 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11785006 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1195973635444224 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3901943499264 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚35379325 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚35379325 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1195450223642314 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚22432086 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚71312860 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1195451593670604 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1195972263881216 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1195972263881216 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1191554957160448 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚71302881 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚327269660225024 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3896611082188 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3895266481866 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1192070320381952 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11785006 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1195973635444224 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3901943499264 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚35379325 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚35379325 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1195450223642314 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚22432086 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚71312860 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1195451593670604 I2Ñ- _wandbruntime‚6895  _runtime‚6895^^qfÄÒ‚N [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1192348321112064 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚71329558 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1196252561217024 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3904213931520 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚22437339 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚35387537 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚35387531 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚35932034 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11787718 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚322319320199168 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚35932034 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1192348347285504 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1192348321112064 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1195730398915138 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1195729028886736 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _runtime‚ 6895.39774642 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3904213931520 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚35942021 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚244 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚71329558 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1191832838938624 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1196251189646848 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚71319565 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚22437090 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10649377 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚327347932633600 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1191832864366592 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚322318952108032 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚249 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚4149 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚35387537 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1196252561217024 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚35387531 _step‚340 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3897534548546 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚22437090 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10649377 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚327347932633600 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1191832864366592 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚322318952108032 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚249 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚4149 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚35387537 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1196252561217024 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚35387531 _step‚340 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3897534548546 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚71329558 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1196252561217024 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3904213931520 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚22437339 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚35387537 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚35387531 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚35932034 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11787718 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚322319320199168 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚35932034 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1192348347285504 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1192348321112064 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1195730398915138 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1195729028886736 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _runtime‚ 6895.39774642 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3904213931520 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚35942021 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚244 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚71329558 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1191832838938624 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1196251189646848 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚71319565 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0ÔXÉá2Ô- _wandbruntime‚6895  _runtime‚6895ȃî Õ… 0loss/FRt1/assignment_loss‚1.7300787228216663 0loss/FRt2/assignment_loss‚0.7382440877997357 2loss/SRqqt1/assignment_loss‚0.5303760796371597 1loss/SRqqt1/detection_loss‚0.9708913468658279 % _timestamp‚1.7842967112666938e+09 _runtime‚6895.483643214 1loss/SRqqt2/detection_loss‚0.6346455532044029 trainer/global_step‚4149 _step‚341 2loss/SRqqt2/assignment_loss‚0.5285579454242801 2loss/FBt1/assignment_loss‚0.009874468426201469 3loss/FBt2/assignment_loss‚0.0014990937820676604 %loss/total_loss‚8.375906618197513 /loss/FRt1/detection_loss‚0.5057335522868017 /loss/FRt2/detection_loss‚1.5529288232774305 /loss/FBt1/detection_loss‚0.7519701934523053 /loss/FBt2/detection_loss‚0.4211067512196346 epoch‚10#©ßÖŠ 1loss/SRqqt2/detection_loss‚0.6346455532044029 trainer/global_step‚4149 _step‚341 2loss/SRqqt2/assignment_loss‚0.5285579454242801 2loss/FBt1/assignment_loss‚0.009874468426201469 3loss/FBt2/assignment_loss‚0.0014990937820676604 %loss/total_loss‚8.375906618197513 /loss/FRt1/detection_loss‚0.5057335522868017 /loss/FRt2/detection_loss‚1.5529288232774305 /loss/FBt1/detection_loss‚0.7519701934523053 /loss/FBt2/detection_loss‚0.4211067512196346 epoch‚10 0loss/FRt1/assignment_loss‚1.7300787228216663 0loss/FRt2/assignment_loss‚0.7382440877997357 2loss/SRqqt1/assignment_loss‚0.5303760796371597 1loss/SRqqt1/detection_loss‚0.9708913468658279 % _timestamp‚1.7842967112666938e+09 _runtime‚6895.483643214Õ ĉB”ׂ(:‰ ‹êèÒúê±ö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚299.456' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚59.8912 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0]¢Ë¥ôØ‚(:é ‹êèÒÕ›îù disk.sda.in‚0 disk.dm-1.in‚0 disk.drbd2165.out‚0 disk.sda2.out‚0$ memory_percent‚89.96608257293701 disk.sda1.in‚0 disk.sda.out‚ 285.55078125 disk.dm-1.out‚ 285.55078125 disk.sda1.out‚0 disk.sda3.out‚ 285.55078125( disk.nvme0c0n1.out‚67068.64208984375 disk.rbd1.out‚ 16.43359375 disk.rbd0.in‚0 disk.sda3.in‚0 disk.rbd1.in‚0 network.sent‚ 7.806466e+06 disk.rbd0.out‚0 disk.sda2.in‚0# proc.memory.rssMB‚ 2826.58203125 cpu‚25.562155885329663 network.recv‚ 1.747722e+06 disk.dm-0.in‚ 14030.125# disk.dm-0.out‚67068.64208984375 disk.rbd3.out‚0 proc.cpu.threads‚17) disk./.usagePercent‚37.02598771393218$ disk./.usageGB‚551.7139739990234 disk.nvme0n1.in‚ 14030.125 disk.drbd2165.in‚0 disk.rbd3.in‚0 disk.nvme0c0n1.in‚ 14030.125& disk.nvme0n1.out‚67068.64208984375) proc.memory.availableMB‚ 1643.95703125* proc.memory.percent‚17.252087593078613ÃìÎ&üÙ‚(:ñ šêèÒñšø network.recv‚ 1.752456e+06 disk.sda.out‚ 287.91796875 disk.rbd0.in‚0# disk.dm-0.out‚67091.48486328125& disk.nvme0n1.out‚67091.48486328125 disk.sda.in‚0 disk.rbd3.in‚0 disk.rbd3.out‚0) disk./.usagePercent‚37.02677800919039( disk.nvme0c0n1.out‚67091.48486328125) proc.memory.availableMB‚ 1643.41015625 disk.rbd0.out‚0 disk.sda1.in‚0 disk.sda3.in‚0 disk.sda3.out‚ 287.91796875 disk.nvme0n1.in‚ 14030.328125 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd1.in‚0 network.sent‚ 7.869527e+06 disk.dm-0.in‚ 14030.328125 disk.sda1.out‚0 disk.rbd1.out‚ 16.43359375* proc.memory.percent‚17.252087593078613 cpu‚25.56181805419535$ disk./.usageGB‚551.7257499694824 disk.dm-1.in‚0$ memory_percent‚89.96942043304443 proc.cpu.threads‚17 disk.dm-1.out‚ 287.91796875 disk.drbd2165.in‚0 disk.drbd2165.out‚0" disk.nvme0c0n1.in‚ 14030.328125# proc.memory.rssMB‚ 2826.58203125fܾŸÚ‚(:” šêèÒŸ¹‘ø gpu.0.gpu‚90.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚258.814' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚51.762800000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0eÃÓ&”Û‚(:‰ ©êèҵ؉÷ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚288.968' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚57.7936 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 Ñ•NïÜ‚(:ä ©êèÒáÚçù disk.nvme0n1.in‚ 14116.6875 disk.sda.out‚ 290.328125 disk.dm-0.in‚ 14116.6875 disk.sda1.in‚0 disk.sda3.in‚0 proc.cpu.threads‚17) disk./.usagePercent‚37.02773547736844 disk.sda2.out‚0 disk.rbd1.out‚ 16.49609375 cpu‚25.56138470416128 disk.sda.in‚0 disk.dm-1.in‚0 disk.sda1.out‚0 disk.drbd2165.out‚0# memory_percent‚89.9695634841919 disk.nvme0c0n1.in‚ 14116.6875 disk.sda2.in‚0 disk.rbd1.in‚0 disk.rbd3.in‚0) proc.memory.availableMB‚ 1643.38671875 disk.dm-1.out‚ 290.328125# proc.memory.rssMB‚ 2826.58203125 network.recv‚ 1.756054e+06( disk.nvme0c0n1.out‚67114.97021484375& disk.nvme0n1.out‚67114.97021484375 disk.drbd2165.in‚0* proc.memory.percent‚17.252087593078613$ disk./.usageGB‚551.7400169372559 disk.rbd0.out‚0 disk.rbd0.in‚0# disk.dm-0.out‚67114.97021484375 disk.sda3.out‚ 290.328125 disk.rbd3.out‚0 network.sent‚ 7.873996e+06’꙾ŸÝ‚(:” ¸êèÒ«ÂÛö gpu.0.gpu‚98.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚40.0 gpu.0.powerWatts‚148.016' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚29.603199999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0±Íì/úÞ‚(:ï ¸êèÒ¯ŒÚø& disk.nvme0c0n1.out‚67139.494140625' proc.memory.availableMB‚ 1643.359375* proc.memory.percent‚17.252087593078613 disk.sda3.in‚0# proc.memory.rssMB‚ 2826.58203125 network.recv‚ 1.759658e+06) disk./.usagePercent‚37.02794745160679! disk.dm-0.out‚67139.494140625 disk.sda3.out‚ 292.74609375 disk.drbd2165.out‚0 disk.dm-1.in‚0 disk.dm-0.in‚14116.90234375 disk.sda1.out‚0 disk.rbd3.in‚0 disk.rbd0.out‚0 disk.drbd2165.in‚0$ disk.nvme0c0n1.in‚14116.90234375 disk.sda2.out‚0$ memory_percent‚89.96973037719727 network.sent‚ 7.879589e+06 disk.dm-1.out‚ 292.74609375" disk.nvme0n1.in‚14116.90234375$ disk.nvme0n1.out‚67139.494140625 cpu‚25.56111753358087 disk.rbd0.in‚0 disk.rbd1.out‚ 16.49609375$ disk./.usageGB‚551.7431755065918 disk.sda.in‚0 disk.sda1.in‚0 disk.sda2.in‚0 disk.rbd1.in‚0 proc.cpu.threads‚17 disk.sda.out‚ 292.74609375 disk.rbd3.out‚0ˆ(ÎUžß‚(:“ ÇêèÒ€»÷ö gpu.0.gpu‚80.0 gpu.0.memory‚24, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚317.192' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚63.43840000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1209918616317440 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚36359281 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚22695281 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1209391919692512 trainer/global_step‚4199 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚36359281 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚35789625 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1205970383060992 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3942848195296 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1205449071497216 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10773984 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚72138933 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3949604819456 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3948259429888 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚269 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚326115786084352 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10773979 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1205449046069248 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 % _timestamp‚1.7842967904208717e+09 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1209918616317440 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3949604819456 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404724736 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _runtime‚6974.641990011 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36633600 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1205449071497216 JBDeviceStatsMonitor.on_train_batch_a¢– þ"start/segment.small_pool.current‚820 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10773984 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚72138933 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3949604819456 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3948259429888 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚269 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚326115786084352 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10773979 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1205449046069248 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 % _timestamp‚1.7842967904208717e+09 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1209918616317440 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3949604819456 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404724736 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _runtime‚6974.641990011 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36633600 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1209918616317440 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚36359281 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚22695281 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1209391919692512 trainer/global_step‚4199×–B)ç‚(:„ ÖêèÒõÅ–÷ gpu.0.gpu‚99.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚337.0' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚67.4 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0êØ1Åýè‚(:ò ÖêèÒû¦ëø disk.sda2.in‚0* proc.memory.percent‚17.252087593078613( disk./.usagePercent‚37.0286450721376 disk.drbd2165.in‚0# disk.nvme0c0n1.in‚ 14117.5234375! disk.nvme0n1.in‚ 14117.5234375& disk.nvme0n1.out‚67169.14013671875 disk.rbd1.in‚0# disk.dm-0.out‚67169.14013671875 disk.sda1.out‚0 disk.rbd3.out‚0( disk.nvme0c0n1.out‚67169.14013671875 disk.rbd1.out‚ 16.55859375 disk.sda3.out‚ 294.91015625 cpu‚25.560390130138895 proc.cpu.threads‚17 disk.rbd0.in‚0' proc.memory.availableMB‚ 1643.140625 disk.dm-1.out‚ 294.91015625 network.sent‚ 7.889749e+06$ disk./.usageGB‚551.7535705566406 disk.sda2.out‚0 disk.sda.out‚ 294.91015625 disk.rbd0.out‚0# proc.memory.rssMB‚ 2826.58203125$ memory_percent‚89.97106552124023 disk.sda.in‚0 disk.dm-1.in‚0 disk.dm-0.in‚ 14117.5234375 disk.sda1.in‚0 disk.sda3.in‚0 network.recv‚ 1.766507e+06 disk.drbd2165.out‚0 disk.rbd3.in‚0¥lï€2é- _wandbruntime‚6976  _runtime‚6976>±ÿ 'êƒN _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚326193272492032 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3949184465408 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚36357786 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1209670724937046 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10776527 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚326193640583168 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1205726927847424 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11924222 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 trainer/global_step‚4199 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3942427061220 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1210198913653248 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1210197542083072 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚72165610 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚36357786 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1210198913653248 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚22700749 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1206248383791104 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚22700484 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚35797831 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚35797837 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚331282014659584 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚35797831 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10776522 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3949184465408 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1209669354908644 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3943771661654 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 _step‚344 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1206248357617664 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚36367773 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11923962 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404717568 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36626432 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5088337449984 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3950529862144 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚260 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚6976.041994056 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1206248357617664 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820Ñ‹sQ'ëˆN [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3950529862144 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1205726953275392 % _timestamp‚1.7842967918245819e+09 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1206248383791104 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚22700484 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚35797831 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚35797837 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚331282014659584 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚35797831 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10776522 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3949184465408 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1209669354908644 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3943771661654 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 _step‚344 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1206248357617664 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚36367773 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11923962 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404717568 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36626432 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5088337449984 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3950529862144 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚260 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚6976.041994056 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1206248357617664 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚326193272492032 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3949184465408 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚36357786 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1209670724937046 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10776527 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚326193640583168 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1205726927847424 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11924222 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 trainer/global_step‚4199 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3942427061220 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1210198913653248 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1210197542083072 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚72165610 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚36357786 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1210198913653248 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚22700749 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 EDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _step‚347 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚36785033 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1223864968753664 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚264 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1219870419566592 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5146698519552 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚72974985 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚36775060 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12058332 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3987740708082 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚22959730 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5146735153152 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1223864968753664 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1219343134978048 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10901134 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1223332245714420 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 trainer/global_step‚4249 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚72984964 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 % _timestamp‚1.7842968703453138e+09ÖaP+(þþO [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12058596 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚36199931 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚36199925 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404724736 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚335136804987904 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1223330875686130 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _step‚347 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚36785033 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1223864968753664 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚264 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1219870419566592 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5146698519552 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚72974985 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚36775060 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12058332 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3987740708082 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚22959730 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5146735153152 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1223864968753664 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1219343134978048 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10901134 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1223332245714420 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 trainer/global_step‚4249 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚72984964 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 % _timestamp‚1.7842968703453138e+09 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1219343160406016 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1223866340316672 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10901129 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 _runtime‚7054.563087896 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚36199931 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3994575360512 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚36199925 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1223866340316672 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3995920750080ÛAÕEï2ÿ- _wandbruntime‚7055  _runtime‚7055®ûÇ€ƒN TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚72991669 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3995500396032 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚265 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1220148394123264 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3995500396032 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1220148420296704 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1220148394123264 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1223611050958954 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 _runtime‚7055.956059526 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404717568 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1224143894519296 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚330067960967168 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5147917745152 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1224145266089472 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3996845792768 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36626432 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚72991669 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1224145266089472 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚22965013 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚36793525 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1220148420296704 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1219621042184192 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚335215473994752 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5147881118720 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3990008774762 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 trainer/global_step‚4249 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚36783538 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚22964748 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚335215878712320 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚330067592876032 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚36208137 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚73001662 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 % _timestamp‚1.7842968717386556e+09 _step‚348 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚36208137 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚36793525 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10903677 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820Ÿ€ˆ|'ˆN REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10903672 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3996845792768 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚36208131 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚36783538 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1219621016756224 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚335215473994752 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5147881118720 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3990008774762 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 trainer/global_step‚4249 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚36783538 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚22964748 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚335215878712320 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚330067592876032 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚36208137 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚73001662 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 % _timestamp‚1.7842968717386556e+09 _step‚348 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚36208137 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚36793525 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10903677 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚72991669 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3995500396032 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚265 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1220148394123264 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3995500396032 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1220148420296704 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1220148394123264 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1223611050958954 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 _runtime‚7055.956059526 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404717568 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1224143894519296 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚330067960967168 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5147917745152 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1224145266089472 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3996845792768 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36626432 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚72991669 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1224145266089472 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚22965013 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚36793525 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1220148420296704 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1219621042184192 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 EDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚339070421321728 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5206031102976 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36633600 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚36610225 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4042236680704 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚36610225 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚23223426 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1233237249314816 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12195416 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1237811321189888 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4042236680704 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1233770456072192 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚274 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1233237223886848 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4035322421512 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1237812692752896 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1233770429898752 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 _step‚351 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚11028284 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404724736 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚73811037 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12195147 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚333864426852352 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4040891291136 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1237271201708038 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚36610231 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1233770429898752 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚37200812 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚333864794943488 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚37210785 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4040891291136 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 trainer/global_step‚4299 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 TGDeviceStatsMonitor.on_train_batch_start/allocation.lõ@Nœarge_pool.allocated‚36610231 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚269 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 % _timestamp‚1.7842969509557514e+09 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚23223700 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚73821016 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0[Øä(”þO A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚36610225 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚23223426 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1233237249314816 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12195416 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1237811321189888 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4042236680704 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1233770456072192 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚274 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1233237223886848 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4035322421512 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1237812692752896 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1233770429898752 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 _step‚351 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚11028284 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404724736 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚73811037 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12195147 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚333864426852352 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4040891291136 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1237271201708038 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚36610231 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1233770429898752 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚37200812 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚333864794943488 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚37210785 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4040891291136 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 trainer/global_step‚4299 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚36610231 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚269 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 % _timestamp‚1.7842969509557514e+09 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚23223700 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚73821016 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚37200812 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1237812692752896 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1237811321189888 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1233770456072192 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚37210785 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚339070826046464 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1237272571736328 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5205994469376 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚339070421321728 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5206031102976 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36633600 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚36610225 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4042236680704ßg9^2•- _wandbruntime‚7136  _runtime‚7136ÍmØ*'–‚N J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚37209290 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚339149492694016 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36626432 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5207174716416 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚23228735 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚37219277 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚73827721 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚36618431 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1238091618525696 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 trainer/global_step‚4299 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚36618437 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5207211342848 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1237550006952460 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1238090246955520 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚73837714 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚36618437 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚37219277 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 _step‚352 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4043161723392 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11030827 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12197913 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4041816326656 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚73837714 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚262 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12198170 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1234048430628864 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚37209290 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚333941913260032 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚257 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚333942281351168 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4036245887870 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987!sçZ '—‡N E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚262 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12198170 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1234048430628864 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚37209290 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚333941913260032 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚257 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚333942281351168 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4036245887870 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚37209290 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚339149492694016 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36626432 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5207174716416 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚23228735 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚37219277 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚73827721 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚36618431 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1238091618525696 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 trainer/global_step‚4299 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚36618437 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5207211342848 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1237550006952460 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1238090246955520 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚73837714 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚36618437 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚37219277 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 _step‚352 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4043161723392 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11030827 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12197913 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4041816326656 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚73837714 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0àåçu 2˜- _wandbruntime‚7136  _runtime‚7136'鶙‚ trainer/global_step‚4299 0loss/FRt1/assignment_loss‚1.6354763739043627 /loss/FRt2/detection_loss‚1.5067355304983627 /loss/FBt1/detection_loss‚0.7535357878077775 epoch‚10 _step‚353 0loss/FRt2/assignment_loss‚0.6044446269091028 1loss/SRqqt1/detection_loss‚0.8514764890119079 _runtime‚7136.656410256 1loss/FBt1/assignment_loss‚0.01919412204228858 2loss/FBt2/assignment_loss‚0.010902283654072104 1loss/SRqqt2/detection_loss‚0.6856864404447119 %loss/total_loss‚8.160066192820352 % _timestamp‚1.7842969524392545e+09 1loss/SRqqt1/assignment_loss‚0.509077070327832 2loss/SRqqt2/assignment_loss‚0.4828851418068609 /loss/FRt1/deâi¹Rtection_loss‚0.5513053631800908 /loss/FBt2/detection_loss‚0.5493469632329825J]Aô š‡ 1loss/SRqqt2/detection_loss‚0.6856864404447119 %loss/total_loss‚8.160066192820352 % _timestamp‚1.7842969524392545e+09 1loss/SRqqt1/assignment_loss‚0.509077070327832 2loss/SRqqt2/assignment_loss‚0.4828851418068609 /loss/FRt1/detection_loss‚0.5513053631800908 /loss/FBt2/detection_loss‚0.5493469632329825 trainer/global_step‚4299 0loss/FRt1/assignment_loss‚1.6354763739043627 /loss/FRt2/detection_loss‚1.5067355304983627 /loss/FBt1/detection_loss‚0.7535357878077775 epoch‚10 _step‚353 0loss/FRt2/assignment_loss‚0.6044446269091028 1loss/SRqqt1/detection_loss‚0.8514764890119079 _runtime‚7136.656410256 1loss/FBt1/assignment_loss‚0.01919412204228858 2loss/FBt2/assignment_loss‚0.010902283654072104áŽ7™Ì”›‚(:‰ ûëèÒð Ðö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚295.976' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚59.1952 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0-iþùœ‚(:î ûëèÒˆ³Ýø# disk.nvme0c0n1.in‚ 14739.3046875& disk.nvme0n1.out‚67627.67236328125$ memory_percent‚89.96925354003906* disk./.usagePercent‚37.040081440707176 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.sda2.out‚0 disk.rbd0.in‚0 disk.dm-0.in‚ 14739.3046875 disk.sda3.out‚ 323.0859375 disk.drbd2165.out‚0 disk.rbd3.in‚0 disk.sda2.in‚0 network.sent‚ 8.09754e+06* proc.memory.percent‚17.252087593078613 proc.cpu.threads‚17$ disk./.usageGB‚551.9239807128906 disk.rbd1.in‚0 disk.sda.out‚ 323.0859375 disk.dm-1.in‚0 disk.rbd0.out‚0 disk.sda1.in‚0# proc.memory.rssMB‚ 2826.58203125# disk.dm-0.out‚67627.67236328125 disk.rbd3.out‚0( disk.nvme0c0n1.out‚67627.67236328125! disk.nvme0n1.in‚ 14739.3046875% proc.memory.availableMB‚ 1643.4375 disk.rbd1.out‚ 16.99609375 disk.dm-1.out‚ 323.0859375 disk.sda3.in‚0 network.recv‚ 1.808033e+06 disk.sda.in‚0 cpu‚25.556626726984426˜jã}Ÿ‚(:” ŠìèÒöæÔö gpu.0.gpu‚100.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚331.737' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚66.34740000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0W|*.ýž‚(:ò ŠìèÒ刺ø disk.drbd2165.out‚0$ disk.nvme0c0n1.in‚14739.72265625 disk.rbd1.out‚ 16.99609375) disk./.usagePercent‚37.04068382643525 disk.sda3.out‚ 325.171875 disk.dm-1.out‚ 325.171875# disk.dm-0.out‚67650.09716796875$ memory_percent‚89.97151851654053# proc.memory.rssMB‚ 2826.58203125* proc.memory.percent‚17.252087593078613 disk.sda1.out‚0& disk.nvme0n1.out‚67650.09716796875 disk.dm-0.in‚14739.72265625) proc.memory.availableMB‚ 1643.06640625 proc.cpu.threads‚17 network.sent‚ 8.159874e+06 disk.sda2.out‚0 disk.sda.in‚0 disk.sda.out‚ 325.171875 disk.sda3.in‚0 disk.rbd3.in‚0 disk.dm-1.in‚0$ disk./.usageGB‚551.9329566955566 disk.sda1.in‚0 disk.rbd3.out‚0" disk.nvme0n1.in‚14739.72265625 disk.rbd0.in‚0 cpu‚25.556371078544874 network.recv‚ 1.811854e+06 disk.drbd2165.in‚0( disk.nvme0c0n1.out‚67650.09716796875 disk.sda2.in‚0 disk.rbd1.in‚0 disk.rbd0.out‚0ÅÊj”Ÿ‚(:‰ ™ìèÒ•¬Ò÷ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚290.031' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚58.0062 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‘™h˜â ‚(:× ™ìèÒš£›ø cpu‚25.555977261812483 disk.sda3.in‚0 disk.rbd1.out‚ 17.05859375% proc.memory.availableMB‚ 1643.0625 disk.dm-0.in‚ 14739.9375 disk.drbd2165.out‚0% disk.nvme0c0n1.out‚67771.35546875 disk.sda2.out‚0$ disk./.usageGB‚551.9336204528809 disk.dm-0.out‚67771.35546875 disk.sda1.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda1.out‚0 disk.sda2.in‚0 network.recv‚ 1.815375e+06* disk./.usagePercent‚37.040728371746205 disk.dm-1.in‚0 disk.rbd3.in‚0 disk.nvme0n1.in‚ 14739.9375# disk.nvme0n1.out‚67771.35546875 disk.rbd1.in‚0 proc.cpu.threads‚17 network.sent‚ 8.164886e+06 disk.sda.out‚ 326.34375 disk.nvme0c0n1.in‚ 14739.9375# proc.memory.rssMB‚ 2826.58203125 disk.sda.in‚0 disk.dm-1.out‚ 326.34375 disk.rbd3.out‚0 disk.sda3.out‚ 326.34375 disk.drbd2165.in‚0$ memory_percent‚89.97154235839844* proc.memory.percent‚17.252087593078613`‚’¤’¡‚(:‡ ¨ìèÒÏ›êö gpu.0.gpu‚99.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚342.93' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚68.586 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÃÃ?Wô¢‚(:é ¨ìèÒšüÛø network.sent‚ 8.169893e+06$ disk./.usageGB‚551.9401321411133 disk.sda.out‚ 328.07421875 disk.rbd0.out‚0 disk.sda3.out‚ 328.07421875 disk.dm-1.out‚ 328.07421875 disk.dm-0.in‚ 14844.625# disk.dm-0.out‚67787.22998046875 disk.drbd2165.out‚0 disk.rbd3.in‚0* proc.memory.percent‚17.252087593078613 proc.cpu.threads‚17 disk.dm-1.in‚0 disk.sda3.in‚0 disk.nvme0c0n1.in‚ 14844.625& disk.nvme0n1.out‚67787.22998046875* disk./.usagePercent‚37.041165376607154 disk.sda.in‚0( disk.nvme0c0n1.out‚67787.22998046875 disk.nvme0n1.in‚ 14844.625# proc.memory.rssMB‚ 2826.58203125 disk.drbd2165.in‚0 disk.sda2.out‚0 disk.rbd1.in‚0 network.recv‚ 1.818592e+06 disk.sda1.out‚0 disk.rbd3.out‚0 disk.rbd1.out‚ 17.05859375 cpu‚25.555648914414675 disk.rbd0.in‚0 disk.sda1.in‚0$ memory_percent‚89.97311592102051( proc.memory.availableMB‚ 1642.8046875 disk.sda2.in‚0pH%Ÿ£‚(:” ·ìèÒãÝÚö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚274.182' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚54.836400000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0o€òÒ꤂(:ß ·ìèÒ¤åÔø disk.sda1.out‚0 disk.sda3.in‚0$ memory_percent‚89.97318744659424 disk.rbd3.out‚0$ disk.nvme0c0n1.out‚ 67800.9921875) disk./.usagePercent‚37.04133536560989 disk.sda2.in‚0 disk.dm-1.in‚0 disk.dm-0.out‚ 67800.9921875 disk.sda1.in‚0 disk.drbd2165.out‚0 network.sent‚ 8.178762e+06 network.recv‚ 1.829701e+06" disk.nvme0n1.out‚ 67800.9921875 disk.sda2.out‚0# proc.memory.rssMB‚ 2826.58203125* proc.memory.percent‚17.252087593078613 cpu‚25.55525694030448 disk.rbd1.out‚ 17.05859375 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda3.out‚329.5625) proc.memory.availableMB‚ 1642.79296875 disk.sda.out‚329.5625 disk.dm-1.out‚329.5625 disk.dm-0.in‚14845.04296875$ disk.nvme0c0n1.in‚14845.04296875" disk.nvme0n1.in‚14845.04296875 disk.rbd1.in‚0 disk.drbd2165.in‚0 disk.rbd3.in‚0$ disk./.usageGB‚551.9426651000977 disk.sda.in‚0ç„å2¥- _wandbruntime‚7214  _runtime‚7214QÃKlŦ¿ _step‚354 & lr-AdamW/pg1‚0.0006248346537164348 & lr-AdamW/pg2‚0.0006248346537164348 trainer/global_step‚4349 # _timestamp‚1.78429702999772e+09 _runtime‚7214.214995353ò3Ýr椀 & lr-AdamW/pg2‚0.0006248346537164348 trainer/global_step‚4349 # _timestamp‚1.78429702999772e+09 _runtime‚7214.214995353 _step‚354 & lr-AdamW/pg1‚0.0006248346537164348âeâÈ›2¨- _wandbruntime‚7214  _runtime‚7214ü!¤ÿ'©ùO L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚37626564 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1247670492577792 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚260 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚337739115327488 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5265330058240 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36633600 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚37020531 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚23488200 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1247670466404352 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5265293424640 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚4349 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11155429 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1247131338223616 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 _runtime‚7214.216297893 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1251211527729946 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚265 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4080214934298 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚74647089 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚343004445385728 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚37636537 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚337738747236352 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 _step‚355 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚37636537 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12333031 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1251757673626112 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4087207221760 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚37626564 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12332771 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚343004040660992 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404724736 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1251759045189120 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4088552611328 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1247131312795648 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1251757673626112 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚11155434 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1251211527729946 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚265 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4080214934298 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚74647089 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚343004445385728 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚37636537 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚337738747236352 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 _step‚355 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚37636537 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12333031 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1251757673626112 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4087207221760 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚37626564 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12332771 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚343004040660992 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404724736 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1251759045189120 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4088552611328 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1247131312795648 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1251757673626112 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚11155434 IDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚74673766 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5266511690752 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4082483000978 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚23493478 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5266475064320 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚37635042 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚258 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1251491703002770 _runtime‚7215.603120005 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚74673766 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404717568 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1247409220001792 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11157977 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36626432 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1247948493307904 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4089477654016 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚337816601735168 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚37028731 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚11157972 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚337816233644032 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚23493736 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1247948467134464 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚37028731 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1252037970961920 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚37645029 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚253 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12335506 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1251490332974368 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚74663773 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚343083113425920 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚37635042 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚37028737 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 % _timestamp‚1.7842970313857622e+09 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1252036599391744 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1247948493307904 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4081138400544 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚23493478 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5266475064320 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚37635042 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚258 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1251491703002770 _runtime‚7215.603120005 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚74673766 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404717568 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1247409220001792 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11157977 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36626432 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1247948493307904 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4089477654016 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚337816601735168 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚37028731 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚11157972 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚337816233644032 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚23493736 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1247948467134464 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚37028731 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1252037970961920 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚37645029 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚253 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12335506 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1251490332974368 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚74663773 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚343083113425920 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚37635042 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚37028737 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 % _timestamp‚1.7842970313857622e+09 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1252036599391744 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1247948493307904 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4081138400544 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚74673766 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5266511690752 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4082483000978 EDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 O/ŸP‘@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1261570529083392 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚37430831 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚346938056830976 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1261570529083392 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4134868541952 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 $ _timestamp‚1.784297108719355e+09 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚7292.939933904 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚38052316 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1265704026062336 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚37430831 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚278 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚38062289 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚273 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1265705397625344 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚23752531 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚38062289 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 trainer/global_step‚4399 _step‚359 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚38052316 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1265151853751854) Ù(ÂýO VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1265151853751854 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1261570529083392 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚37430831 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚346938056830976 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1261570529083392 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4134868541952 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 $ _timestamp‚1.784297108719355e+09 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚7292.939933904 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚38052316 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1265704026062336 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚37430831 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚278 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚38062289 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚273 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1265705397625344 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚23752531 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚38062289 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 trainer/global_step‚4399 _step‚359 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚38052316 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648çý¾ÿx2Ã- _wandbruntime‚7294  _runtime‚7294 5y 'ăN \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4135793584640 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚264 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5325771809792 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1261303308910592 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1265984323398144 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4135793584640 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12472944 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1265432029024678 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404717568 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚341690554028032 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚75509818 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚11285122 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36626432 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1265430658996276 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚341690922119168 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1265984323398144 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚75499825 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚37439031 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1261303283482624 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚38060794 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1265982951827968 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚37439037 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 % _timestamp‚1.7842971101016028e+09 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚37439037 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚23758071 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1261848529813504 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4127375513652 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚23757807 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚38070781 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4128720114086 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1261848529813504 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚75499825 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚38070781 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚259 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1265982951827968 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚37439037 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 % _timestamp‚1.7842971101016028e+09 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚37439037 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚23758071 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1261848529813504 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4127375513652 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚23757807 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚38070781 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4128720114086 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1261848529813504 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚75499825 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚38070781 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚259 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚75509818 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚11285122 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36626432 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1265430658996276 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚341690922119168èa'T§2Æ- _wandbruntime‚7294  _runtime‚72943¶C Ç„ 0loss/FRt1/detection_loss‚0.47521456747042395 _runtime‚7294.405728918 2loss/SRqqt1/assignment_loss‚0.5362255729470647 1loss/SRqqt1/detection_loss‚0.9132263630170981 trainer/global_step‚4399 % _timestamp‚1.7842971101887853e+09 _step‚361 2loss/SRqqt2/assignment_loss‚0.5033316183883983 1loss/SRqqt2/detection_loss‚0.6380157580665425 /loss/FBt1/detection_loss‚0.7438630224487772 /loss/FBt2/detection_loss‚0.7256532819980445 %loss/total_loss‚8.461388225450388 0loss/FRt1/assignment_loss‚1.6454005303242756 2loss/FBt2/assignment_loss‚0.029308047858522694 /loss/FRt2/detection_loss‚1.5708981312738124 epoch‚10 0loss/FRt2/assignment_loss‚0.6786367680584446 1loss/FBt1/assignment_loss‚0.00161456359898416 .ȉ _step‚361 2loss/SRqqt2/assignment_loss‚0.5033316183883983 1loss/SRqqt2/detection_loss‚0.6380157580665425 /loss/FBt1/detection_loss‚0.7438630224487772 /loss/FBt2/detection_loss‚0.7256532819980445 %loss/total_loss‚8.461388225450388 0loss/FRt1/assignment_loss‚1.6454005303242756 2loss/FBt2/assignment_loss‚0.029308047858522694 /loss/FRt2/detection_loss‚1.5708981312738124 epoch‚10 0loss/FRt2/assignment_loss‚0.6786367680584446 1loss/FBt1/assignment_loss‚0.00161456359898416 0loss/FRt1/detection_loss‚0.47521456747042395 _runtime‚7294.405728918 2loss/SRqqt1/assignment_loss‚0.5362255729470647 1loss/SRqqt1/detection_loss‚0.9132263630170981 trainer/global_step‚4399 % _timestamp‚1.7842971101887853e+09鬓r\’É‚(:‡  íèÒ•ì€ù gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚340.338' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚68.0676 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ž|:lçÊ‚(:Ü  íèÒöÞ¡ú disk.rbd1.out‚ 17.37109375 disk.rbd0.in‚0 disk.dm-0.in‚ 15008.0078125 disk.rbd3.in‚0# proc.memory.rssMB‚ 2908.79296875 disk.dm-0.out‚ 67915.65625 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.rbd3.out‚0) proc.memory.percent‚17.75386333465576! disk.nvme0n1.in‚ 15008.0078125 disk.nvme0n1.out‚ 67915.65625 disk.dm-1.out‚ 332.6171875 disk.sda1.out‚0 network.sent‚ 8.312273e+06$ disk./.usageGB‚548.8781318664551 disk.sda1.in‚0 network.recv‚ 1.86239e+06# disk.nvme0c0n1.in‚ 15008.0078125" disk.nvme0c0n1.out‚ 67915.65625 disk.rbd1.in‚0 disk.sda.in‚0 disk.rbd0.out‚0 disk.sda3.in‚0 disk.sda3.out‚ 332.6171875 disk.sda2.out‚0 disk.sda.out‚ 332.6171875$ memory_percent‚90.47346115112305' proc.memory.availableMB‚ 1560.828125 proc.cpu.threads‚17 disk.sda2.in‚0 disk.dm-1.in‚0 cpu‚25.556670972593565) disk./.usagePercent‚36.83567196898549rÞt£”Ë‚(:‰ ¯íèÒöÏö gpu.0.gpu‚95.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚333.229' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚66.6458 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0JÎDÊÌ‚(:õ ¯íèÒëÅ¢ú disk.sda2.in‚0 disk.rbd1.out‚ 17.37109375 disk.rbd0.in‚0" disk.nvme0n1.in‚15008.42578125# proc.memory.rssMB‚ 2994.61328125* proc.memory.percent‚18.277668952941895 network.sent‚ 8.374659e+06 disk.sda.in‚0# disk.dm-0.out‚67924.79833984375 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd3.in‚0 disk.dm-0.in‚15008.42578125 disk.sda3.in‚0 disk.drbd2165.in‚0 network.recv‚ 1.866828e+06$ disk./.usageGB‚548.8862075805664 disk.rbd1.in‚0 disk.dm-1.out‚ 332.9140625 disk.rbd0.out‚0 disk.sda3.out‚ 332.9140625 disk.rbd3.out‚0$ disk.nvme0c0n1.in‚15008.42578125& disk.nvme0n1.out‚67924.79833984375$ memory_percent‚91.00115299224854 disk.sda.out‚ 332.9140625) proc.memory.availableMB‚ 1474.37109375 disk.sda2.out‚0 disk.dm-1.in‚0 disk.drbd2165.out‚0( disk.nvme0c0n1.out‚67924.79833984375 cpu‚25.561934253169984 proc.cpu.threads‚17) disk./.usagePercent‚36.83621393693547âNX®2Í- _wandbruntime‚7320  _runtime‚7320ÈW|µ=ί{ 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9205806255340576 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9780850410461426 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5962386131286621 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6022905707359314 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.15172061324119568 +jet/accuracy_1_of_1‚0.30027157068252563 6 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.995812177658081 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5980883240699768 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6096972227096558 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 particle/accuracy_4_of_5‚1 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6562390923500061 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.2508297562599182 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4403779208660126 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.15141548216342926 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6405767798423767 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8539115190505981 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9629628658294678 particle/accuracy_2_of_4‚1 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.17916008830070496 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7249140739440918 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9948381781578064 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.18025222420692444 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.040269773453474045 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7037400603294373 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.46990740299224854 *jet/accuracy_1_of_2‚0.7669115662574768 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 jet/accuracy_1_of_5‚1 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.35375502705574036 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.5923802256584167 particle/accuracy_2_of_5‚1 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6517621278762817 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.1309826523065567 particle/accuracy_3_of_3‚1 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 jet/accuracy_3_of_5‚1 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9949023127555847 particle/accuracy_1_of_1‚1 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7404387593269348 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4825717508792877 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6861127018928528 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6097842454910278 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.1967867612838745 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7693198323249817 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8924533128738403 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8604661226272583 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.16944539546966553 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9615384340286255 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7056478261947632 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6276041269302368 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7857142686843872 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.1591331660747528 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7153249382972717 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8256641626358032 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2872724235057831 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9880425333976746 9"Purity/*FRt2SRqqt0FBt/event_purity‚0.1618700474500656 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 +particle/sensitivity‚0.6308420896530151 jet/accuracy_4_of_5‚0 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6440121531486511 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5972684621810913 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.18121099472045898 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.06059021130204201 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7824074029922485 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.04234784096479416 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9953415393829346 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5718274712562561 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7445918321609497 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8486394286155701 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.22986221313476562 /particle/accuracy_4_of_4‚0.9499182105064392 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.1398511677980423 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 9"Purity/2FRt*SRqqt1FBt/event_purity‚0.1274999976158142 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5656715035438538 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8423280119895935 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 5 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.98738032579422 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7100245952606201 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.625 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8763536810874939 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9981421232223511 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6266539096832275 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.4797360599040985 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 +jet/accuracy_3_of_3‚0.07273191213607788 particle/accuracy_1_of_2‚1 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.17718245089054108 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6428079009056091 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.05640047416090965 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9890425801277161 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.19761475920677185 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.24807651340961456 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.07407407462596893 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 *jet/accuracy_2_of_2‚0.1373170167207718 ,jet/accuracy_4_of_4‚0.013899499550461769 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7142857313156128 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9948350191116333 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5721851587295532 particle/accuracy_2_of_3‚1 particle/accuracy_1_of_5‚1 _runtime‚7320.640192109 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 /particle/accuracy_2_of_2‚0.9998962879180908 /particle/accuracy_3_of_4‚0.9965986609458923 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9958263635635376 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.6858453154563904 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.2864583134651184 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.14276748895645142 *jet/accuracy_1_of_3‚0.9229806065559387 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9380038380622864 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.21242448687553406 +jet/accuracy_3_of_4‚0.08796278387308121 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.35525283217430115 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.3026247024536133 'particle/f_score‚0.6423131823539734 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.1953973025083542 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 6 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.998015820980072 6 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.478217214345932 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.06095177307724953 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.04234784096479416 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.15893132984638214 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 8"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.709722638130188 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7807256579399109 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8479591608047485 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7730422019958496 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7148119807243347 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 +particle/specificity‚0.9088230133056641 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.020213302224874496 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 particle/accuracy_1_of_4‚1 trainer/global_step‚4399 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3499011993408203 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9987789392471313 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.39814817905426025 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5462499856948853 *jet/accuracy_2_of_3‚0.5431353449821472 ) Purity/2FRt*SRqqt2FBt/FRt_purity‚0.75 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.07407407462596893 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7824074029922485 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6522942781448364 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.05612403526902199 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.18422287702560425 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.019189368933439255 7validation_average_jet_accuracy‚0.34669414162635803 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9938129186630249 8"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.598483681678772 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7297150492668152 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8256641626358032 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.07692307978868484 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9976770281791687 jet/accuracy_5_of_5‚0 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5962386131286621 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.3984375 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9904571771621704 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 % _timestamp‚1.7842971364224315e+09 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.17061035335063934 particle/accuracy_5_of_5‚0 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.16511403024196625 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 )jet/accuracy_2_of_4‚0.783995509147644 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9974045753479004 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6380224823951721 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 *jet/accuracy_1_of_4‚0.9633980989456177 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 9"Purity/2FRt0SRqqt*FBt/event_purity‚0.0362999252974987 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.07692307978868484 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7680939435958862 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5468785166740417 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6361938118934631 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8575207591056824 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9974045753479004 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.18422287702560425 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.14701762795448303 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.028403710573911667 )loss/val_total_loss‚8.265921507657524 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 #particle/accuracy‚ 0.84912109375 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9984047412872314 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.16677318513393402 jet/accuracy_2_of_5‚1 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6035244464874268 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 particle/accuracy_1_of_3‚1 epoch‚10 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.14584903419017792 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_3_of_5‚1 _step‚362 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.3653823137283325 8"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.680173933506012 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.010877691209316254 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1ü)ËQï'Ï´{ 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9958263635635376 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.6858453154563904 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.2864583134651184 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.14276748895645142 *jet/accuracy_1_of_3‚0.9229806065559387 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9380038380622864 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.21242448687553406 +jet/accuracy_3_of_4‚0.08796278387308121 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.35525283217430115 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.3026247024536133 'particle/f_score‚0.6423131823539734 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.1953973025083542 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 6 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.998015820980072 6 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.478217214345932 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.06095177307724953 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.04234784096479416 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.15893132984638214 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 8"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.709722638130188 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7807256579399109 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8479591608047485 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7730422019958496 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7148119807243347 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 +particle/specificity‚0.9088230133056641 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.020213302224874496 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 particle/accuracy_1_of_4‚1 trainer/global_step‚4399 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3499011993408203 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9987789392471313 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.39814817905426025 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5462499856948853 *jet/accuracy_2_of_3‚0.5431353449821472 ) Purity/2FRt*SRqqt2FBt/FRt_purity‚0.75 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.07407407462596893 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7824074029922485 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6522942781448364 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.05612403526902199 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.18422287702560425 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.019189368933439255 7validation_average_jet_accuracy‚0.34669414162635803 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9938129186630249 8"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.598483681678772 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7297150492668152 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8256641626358032 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.07692307978868484 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9976770281791687 jet/accuracy_5_of_5‚0 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5962386131286621 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.3984375 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9904571771621704 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 % _timestamp‚1.7842971364224315e+09 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.17061035335063934 particle/accuracy_5_of_5‚0 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.16511403024196625 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 )jet/accuracy_2_of_4‚0.783995509147644 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9974045753479004 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6380224823951721 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 *jet/accuracy_1_of_4‚0.9633980989456177 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 9"Purity/2FRt0SRqqt*FBt/event_purity‚0.0362999252974987 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.07692307978868484 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7680939435958862 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5468785166740417 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6361938118934631 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8575207591056824 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9974045753479004 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.18422287702560425 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.14701762795448303 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.028403710573911667 )loss/val_total_loss‚8.265921507657524 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 #particle/accuracy‚ 0.84912109375 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9984047412872314 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.16677318513393402 jet/accuracy_2_of_5‚1 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6035244464874268 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 particle/accuracy_1_of_3‚1 epoch‚10 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.14584903419017792 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_3_of_5‚1 _step‚362 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.3653823137283325 8"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.680173933506012 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.010877691209316254 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9205806255340576 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9780850410461426 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5962386131286621 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6022905707359314 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.15172061324119568 +jet/accuracy_1_of_1‚0.30027157068252563 6 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.995812177658081 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5980883240699768 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6096972227096558 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 particle/accuracy_4_of_5‚1 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6562390923500061 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.2508297562599182 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4403779208660126 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.15141548216342926 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6405767798423767 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8539115190505981 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9629628658294678 particle/accuracy_2_of_4‚1 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.17916008830070496 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7249140739440918 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9948381781578064 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.18025222420692444 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.040269773453474045 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7037400603294373 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.46990740299224854 *jet/accuracy_1_of_2‚0.7669115662574768 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 jet/accuracy_1_of_5‚1 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.35375502705574036 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.5923802256584167 particle/accuracy_2_of_5‚1 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6517621278762817 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.1309826523065567 particle/accuracy_3_of_3‚1 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 jet/accuracy_3_of_5‚1 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9949023127555847 particle/accuracy_1_of_1‚1 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7404387593269348 7 Purity/*FRt1SRqqt1oôÀ@ËFBt/FRt_purity‚0.4825717508792877 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6861127018928528 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6097842454910278 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.1967867612838745 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7693198323249817 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8924533128738403 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8604661226272583 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.16944539546966553 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9615384340286255 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7056478261947632 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6276041269302368 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7857142686843872 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.1591331660747528 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7153249382972717 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8256641626358032 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2872724235057831 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9880425333976746 9"Purity/*FRt2SRqqt0FBt/event_purity‚0.1618700474500656 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 +particle/sensitivity‚0.6308420896530151 jet/accuracy_4_of_5‚0 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6440121531486511 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5972684621810913 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.18121099472045898 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.06059021130204201 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7824074029922485 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.04234784096479416 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9953415393829346 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5718274712562561 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7445918321609497 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8486394286155701 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.22986221313476562 /particle/accuracy_4_of_4‚0.9499182105064392 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.1398511677980423 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 9"Purity/2FRt*SRqqt1FBt/event_purity‚0.1274999976158142 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5656715035438538 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8423280119895935 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 5 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.98738032579422 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7100245952606201 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.625 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8763536810874939 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9981421232223511 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6266539096832275 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.4797360599040985 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 +jet/accuracy_3_of_3‚0.07273191213607788 particle/accuracy_1_of_2‚1 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.17718245089054108 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6428079009056091 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.05640047416090965 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9890425801277161 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.19761475920677185 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.24807651340961456 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.07407407462596893 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 *jet/accuracy_2_of_2‚0.1373170167207718 ,jet/accuracy_4_of_4‚0.013899499550461769 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7142857313156128 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9948350191116333 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5721851587295532 particle/accuracy_2_of_3‚1 particle/accuracy_1_of_5‚1 _runtime‚7320.640192109 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 /particle/accuracy_2_of_2‚0.9998962879180908 /particle/accuracy_3_of_4‚0.9965986609458923 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt0FBt/event_purity‚0êSF6VŸÐ‚(:” ¾íèÒ¼ûýö gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚272.035' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚54.407000000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Š$“”çÑ‚(:Ü ¾íèÒžº«ù network.sent‚ 8.380168e+06 disk.drbd2165.in‚0 disk.rbd3.in‚0# memory_percent‚91.0010576248169 disk.sda.in‚0 disk.rbd0.in‚0 disk.dm-0.in‚15008.62890625" disk.nvme0n1.out‚ 67942.1484375) proc.memory.availableMB‚ 1474.38671875 proc.cpu.threads‚17$ disk.nvme0c0n1.out‚ 67942.1484375 disk.rbd1.in‚0) disk./.usagePercent‚36.83649810529848 disk.drbd2165.out‚0 disk.rbd3.out‚0 disk.dm-0.out‚ 67942.1484375 disk.sda1.in‚0 cpu‚25.562120485096326 network.recv‚ 1.869751e+06 disk.sda.out‚333.125 disk.dm-1.in‚0 disk.sda3.out‚333.125 disk.sda2.out‚0$ disk./.usageGB‚548.8904418945312 disk.sda3.in‚0$ disk.nvme0c0n1.in‚15008.62890625 disk.sda2.in‚0 disk.rbd1.out‚ 17.43359375# proc.memory.rssMB‚ 2994.61328125* proc.memory.percent‚18.277668952941895 disk.dm-1.out‚333.125 disk.rbd0.out‚0 disk.sda1.out‚0" disk.nvme0n1.in‚15008.62890625ÉÓB´ŸÒ‚(:” ÍíèÒñïö gpu.0.gpu‚92.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚288.151' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚57.630199999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Cò)&áÓ‚(:Ö ÍíèÒÏ܈ù$ disk.nvme0c0n1.out‚ 67951.1328125 disk.rbd0.in‚0 disk.dm-0.in‚ 15008.84375! disk.nvme0c0n1.in‚ 15008.84375 disk.sda2.in‚0 disk.rbd1.in‚0 cpu‚25.56172484126774 disk.drbd2165.out‚0 disk.rbd3.in‚0 disk.rbd0.out‚0 disk.sda1.out‚0 disk.sda3.in‚0 disk.nvme0n1.in‚ 15008.84375 disk.sda.out‚333.4375) proc.memory.availableMB‚ 1474.12890625 network.sent‚ 8.437768e+06* disk./.usagePercent‚36.829486314943175 disk.sda.in‚0 disk.sda1.in‚0* proc.memory.percent‚18.277668952941895 network.recv‚ 1.873894e+06# disk./.usageGB‚548.785961151123 disk.sda2.out‚0 disk.rbd1.out‚ 17.43359375 disk.dm-1.in‚0 proc.cpu.threads‚17 disk.rbd3.out‚0 disk.dm-0.out‚ 67951.1328125$ memory_percent‚91.00263118743896# proc.memory.rssMB‚ 2994.61328125 disk.drbd2165.in‚0" disk.nvme0n1.out‚ 67951.1328125 disk.dm-1.out‚333.4375 disk.sda3.out‚333.4375™v7ïŸÔ‚(:” ÜíèÒö®ùö gpu.0.gpu‚56.0 gpu.0.memory‚17, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚39.0 gpu.0.powerWatts‚110.286' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.057199999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÂÀoðÕ‚(:å ÜíèÒšù disk.sda1.in‚0* proc.memory.percent‚18.277668952941895$ disk./.usageGB‚548.7919540405273$ memory_percent‚91.00403785705566) disk./.usagePercent‚36.82988850277946 disk.drbd2165.in‚0 proc.cpu.threads‚17 network.sent‚ 8.442761e+06! disk.nvme0n1.in‚ 15095.9609375 disk.sda2.out‚0 disk.dm-1.out‚335.4375# proc.memory.rssMB‚ 2994.61328125 disk.drbd2165.out‚0' disk.nvme0c0n1.out‚67967.9111328125 disk.rbd0.in‚0" disk.dm-0.out‚67967.9111328125 disk.sda1.out‚0 disk.sda3.in‚0 disk.sda.in‚0 disk.dm-1.in‚0 disk.rbd0.out‚0 network.recv‚ 1.877301e+06 disk.rbd3.in‚0 disk.rbd3.out‚0% disk.nvme0n1.out‚67967.9111328125 disk.sda2.in‚0 disk.rbd1.in‚0 disk.sda.out‚335.4375( proc.memory.availableMB‚ 1473.8984375 disk.sda3.out‚335.4375# disk.nvme0c0n1.in‚ 15095.9609375 disk.rbd1.out‚ 17.49609375 disk.dm-0.in‚ 15095.9609375 cpu‚25.561396181780893ëž“ì”Ö‚(:‰ ëíèÒž€›÷ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚287.327' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚57.4654 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ucýׂ(:ò ëíèÒ£‚åø# disk.dm-0.out‚67980.86962890625! disk.nvme0n1.in‚ 15096.1640625$ memory_percent‚91.00406169891357 proc.cpu.threads‚17) disk./.usagePercent‚36.83003929121713 disk.sda.in‚0 disk.sda1.out‚0& disk.nvme0n1.out‚67980.86962890625 disk.drbd2165.in‚0# proc.memory.rssMB‚ 2994.61328125* proc.memory.percent‚18.277668952941895$ disk./.usageGB‚548.7942008972168 disk.rbd1.in‚0 disk.sda3.out‚ 336.4921875) proc.memory.availableMB‚ 1473.89453125 cpu‚25.561004037803222 disk.sda2.out‚0 disk.dm-0.in‚ 15096.1640625 disk.sda1.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 336.4921875 disk.rbd0.in‚0 disk.rbd0.out‚0# disk.nvme0c0n1.in‚ 15096.1640625( disk.nvme0c0n1.out‚67980.86962890625 network.sent‚ 8.447578e+06 network.recv‚ 1.880044e+06 disk.rbd1.out‚ 17.49609375 disk.sda.out‚ 336.4921875 disk.sda3.in‚0 disk.drbd2165.out‚0 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.sda2.in‚0ÕÖäãžØ‚(:“ úíèÒìÂáö gpu.0.gpu‚94.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚46.0 gpu.0.powerWatts‚339.413' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚67.88260000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0rO­þÙ‚(:ó úíèÒÒ¼”ù disk.drbd2165.out‚0) proc.memory.availableMB‚ 1473.87109375 network.recv‚ 1.88286e+06% disk.nvme0n1.out‚67992.1396484375" disk.nvme0n1.in‚15096.37890625 network.sent‚ 8.451824e+06# disk./.usageGB‚548.801097869873 disk.sda.out‚ 337.05078125 disk.rbd3.out‚0' disk.nvme0c0n1.out‚67992.1396484375* proc.memory.percent‚18.277668952941895) disk./.usagePercent‚36.83050215283904 disk.sda.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda3.in‚0 disk.sda3.out‚ 337.05078125 disk.rbd3.in‚0$ memory_percent‚91.00420475006104 disk.rbd0.in‚0 disk.rbd1.out‚ 17.55859375# proc.memory.rssMB‚ 2994.61328125 cpu‚25.560675757823006$ disk.nvme0c0n1.in‚15096.37890625 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd1.in‚0 disk.dm-1.out‚ 337.05078125" disk.dm-0.out‚67992.1396484375 disk.drbd2165.in‚0 proc.cpu.threads‚17 disk.dm-1.in‚0 disk.rbd0.out‚0 disk.dm-0.in‚15096.37890625 2µî2Ú- _wandbruntime‚7397  _runtime‚7397äY0ýÆÛÀ $ _timestamp‚1.784297213708382e+09 _runtime‚7397.925539283 _step‚363 & lr-AdamW/pg1‚0.0006215190830333612 & lr-AdamW/pg2‚0.0006215190830333612 trainer/global_step‚4449/µˆ¥ËÜÅ _runtime‚7397.925539283 _step‚363 & lr-AdamW/pg1‚0.0006215190830333612 & lr-AdamW/pg2‚0.0006215190830333612 trainer/global_step‚4449 $ _timestamp‚1.784297213708382e+09ëñóßô2Ý- _wandbruntime‚7397  _runtime‚7397Šžÿ'ÞùO _step‚364 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚245 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚240 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1279298547023872 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11418647 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚345535868342272 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4179862892273 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚37973137 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5392174312448 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚76510181 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4187027084288 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1282925287866097 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1279298520850432 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚38547023 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚37973131 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4187027084288 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚24062672 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1283484202545152 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚350928042654720 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚37973137 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1283485574108160 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚32439296 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1278745424973824 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚11418652 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5392141873152 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12644020 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚38537050 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1283485574108160 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1279298547023872 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11418647 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚345535868342272 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4179862892273 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚37973137 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5392174312448 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚76510181 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4187027084288 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1282925287866097 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1279298520850432 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚38547023 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚37973131 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4187027084288 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚24062672 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1283484202545152 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚350928042654720 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚37973137 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1283485574108160 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚32439296 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1278745424973824 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚11418652 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5392141873152 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12644020 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚38537050 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1283485574108160 IDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚76526865 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚24067727 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚229 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚32432128 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚76536858 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚37981337 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚11421190 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚351006340114432 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1283202723082229 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1279576521580544 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1283763128310784 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚76536858 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚234 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12646766 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4186606730240 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚37981337 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 _step‚365 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 _runtime‚7399.312449918 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚76526865 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11421195 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0ÃMþ( 'â‡N EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚76536858 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚234 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12646766 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4186606730240 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚37981337 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 _step‚365 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 _runtime‚7399.312449918 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚76526865 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11421195 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4180786358631 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚76526865 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚24067727 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚229 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚32432128 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚76536858 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚37981337 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚11421190 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚351006340114432 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1283202723082229 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1279576521580544 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1283763128310784 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640í¡tûL2ã- _wandbruntime‚7399  _runtime‚7399#hšäþ /loss/FRt2/detection_loss‚1.4500363340143227 %loss/total_loss‚8.361651593682595 _runtime‚7399.398994845 /loss/FRt1/assignment_loss‚1.707396032586037 1loss/SRqqt1/assignment_loss‚0.559502313726189 .loss/FBt1/detection_loss‚0.763032958603331 /loss/FBt2/detection_loss‚0.5245265831067788 epoch‚11 _step‚366 2loss/FBt2/assignment_loss‚0.004809426318917459 /loss/FRt1/detection_loss‚0.5806502763994651 1loss/SRqqt1/detection_loss‚0.9983219617764668 0loss/SRqqt2/detection_loss‚0.610412182859874 trainer/global_step‚4449 $ _timestamp‚1.784297215182054e+09 1loss/FBt1/assignment_loss‚0.01168791187888587 0loss/FRt2/assignment_loss‚0.6995696962018024 2loss/SRqqt2/assignment_loss‚0.4517059162105243 7d  åƒ 0loss/FRt2/assignment_loss‚0.6995696962018024 2loss/SRqqt2/assignment_loss‚0.4517059162105243 /loss/FRt2/detection_loss‚1.4500363340143227 %loss/total_loss‚8.361651593682595 _runtime‚7399.398994845 /loss/FRt1/assignment_loss‚1.707396032586037 1loss/SRqqt1/assignment_loss‚0.559502313726189 .loss/FBt1/detection_loss‚0.763032958603331 /loss/FBt2/detection_loss‚0.5245265831067788 epoch‚11 _step‚366 2loss/FBt2/assignment_loss‚0.004809426318917459 /loss/FRt1/detection_loss‚0.5806502763994651 1loss/SRqqt1/detection_loss‚0.9983219617764668 0loss/SRqqt2/detection_loss‚0.610412182859874 trainer/global_step‚4449 $ _timestamp‚1.784297215182054e+09 1loss/FBt1/assignment_loss‚0.01168791187888587î6Kžžæ‚(:“ ‰îèÒ«¥Œø gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚269.376' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚53.87519999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0úu/‰ôç‚(:é ‰îèÒô¾ ù disk.rbd0.out‚0 disk.sda3.out‚ 337.7734375 disk.sda2.in‚0) disk./.usagePercent‚36.83100418361368 disk.dm-1.out‚ 337.7734375 disk.dm-0.in‚ 15096.796875$ proc.memory.availableMB‚1474.375 cpu‚25.56031727502489 network.sent‚ 8.456881e+06 disk.sda.in‚0 disk.sda.out‚ 337.7734375 disk.rbd0.in‚0# disk.dm-0.out‚67999.50439453125 disk.sda1.in‚0 disk.sda3.in‚0 disk.rbd3.in‚0& disk.nvme0n1.out‚67999.50439453125 network.recv‚ 1.886329e+06 disk.dm-1.in‚0 disk.sda1.out‚0 disk.sda2.out‚0 disk.rbd1.out‚ 17.55859375 proc.cpu.threads‚17$ disk./.usageGB‚548.8085784912109 disk.rbd3.out‚0* proc.memory.percent‚18.277668952941895 disk.drbd2165.in‚0 disk.drbd2165.out‚0" disk.nvme0c0n1.in‚ 15096.796875 disk.nvme0n1.in‚ 15096.796875# proc.memory.rssMB‚ 2994.61328125( disk.nvme0c0n1.out‚67999.50439453125 disk.rbd1.in‚0$ memory_percent‚91.00112915039062 %‚”è‚(:‰ ˜îèÒŒÐÿö gpu.0.gpu‚93.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚47.0 gpu.0.powerWatts‚339.166' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚67.8332 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ùkžøé‚(:í ˜îèÒ¼è™ù% proc.memory.availableMB‚ 1474.0625 disk.sda1.out‚0* proc.memory.percent‚18.277668952941895 proc.cpu.threads‚17 network.recv‚ 1.890604e+06) disk./.usagePercent‚36.83127939171299 disk.rbd3.out‚0 disk.dm-1.out‚ 338.234375# disk.dm-0.out‚68013.14599609375# proc.memory.rssMB‚ 2994.61328125 disk.sda3.in‚0 disk.sda3.out‚ 338.234375" disk.nvme0n1.in‚15176.72265625 disk.sda.in‚0 network.sent‚ 8.519403e+06 disk.drbd2165.in‚0 disk.rbd3.in‚0$ disk.nvme0c0n1.in‚15176.72265625 disk.sda2.in‚0$ memory_percent‚91.00303649902344$ disk./.usageGB‚548.8126792907715 disk.drbd2165.out‚0 disk.sda2.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 17.62109375 disk.dm-1.in‚0 cpu‚25.55999469891399 disk.sda1.in‚0( disk.nvme0c0n1.out‚68013.14599609375& disk.nvme0n1.out‚68013.14599609375 disk.sda.out‚ 338.234375 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.dm-0.in‚15176.72265625ÆIi)”ê‚(:‰ §îèÒÿúãö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚272.007' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.4014 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0q.üÝôë‚(:é §îèÒÒ¾‘ù disk.rbd3.in‚0$ memory_percent‚91.00303649902344 disk.sda2.out‚0 disk.rbd1.in‚0 disk.sda3.out‚ 338.58984375 disk.drbd2165.out‚0* proc.memory.percent‚18.277668952941895 disk.sda.in‚0 disk.sda1.out‚0& disk.nvme0n1.out‚68021.72900390625# proc.memory.rssMB‚ 2994.61328125 cpu‚25.559606593993877 proc.cpu.threads‚17 disk.dm-1.in‚0 disk.dm-1.out‚ 338.58984375# disk.dm-0.out‚68021.72900390625 disk.nvme0n1.in‚ 15176.9375 network.recv‚ 1.893986e+06* disk./.usagePercent‚36.831766062035584 disk.sda.out‚ 338.58984375( disk.nvme0c0n1.out‚68021.72900390625% proc.memory.availableMB‚ 1474.0625$ disk./.usageGB‚548.8199310302734 disk.rbd1.out‚ 17.62109375 disk.dm-0.in‚ 15176.9375 disk.rbd3.out‚0 disk.nvme0c0n1.in‚ 15176.9375 disk.sda2.in‚0 disk.rbd0.out‚0 disk.sda3.in‚0 disk.drbd2165.in‚0 network.sent‚ 8.525041e+06 disk.rbd0.in‚0 disk.sda1.in‚0ZÜ7žì‚(:“ ¶îèÒ¾ÎÖö gpu.0.gpu‚99.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚288.679' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚57.73579999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0@nhRí‚(:õ ¶îèÒØúù disk.sda1.out‚0# proc.memory.rssMB‚ 2994.61328125* disk./.usagePercent‚36.831552551752026$ disk./.usageGB‚548.8167495727539 disk.drbd2165.out‚0$ disk.nvme0c0n1.in‚15177.35546875 disk.dm-1.in‚0" disk.dm-0.out‚68032.2412109375 network.sent‚ 8.530083e+06 network.recv‚ 1.897322e+06" disk.nvme0n1.in‚15177.35546875 disk.rbd1.in‚0 disk.dm-1.out‚ 338.88671875 disk.rbd0.out‚0 disk.dm-0.in‚15177.35546875% disk.nvme0n1.out‚68032.2412109375 proc.cpu.threads‚17 disk.rbd3.out‚0 disk.rbd0.in‚0 disk.sda1.in‚0# memory_percent‚91.0038709640503* proc.memory.percent‚18.277668952941895 disk.sda.in‚0 disk.sda3.in‚0 disk.sda2.in‚0) proc.memory.availableMB‚ 1473.92578125 cpu‚25.559280663928828 disk.sda3.out‚ 338.88671875 disk.rbd1.out‚ 17.68359375 disk.sda.out‚ 338.88671875 disk.drbd2165.in‚0 disk.rbd3.in‚0' disk.nvme0c0n1.out‚68032.2412109375 disk.sda2.out‚0Ï6f/žî‚(:“ ÅîèÒÈÆŠ÷ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚294.095' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚58.81900000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0L.TÛøï‚(:í ÅîèÒðýŠø( disk.nvme0c0n1.out‚68045.12353515625 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.sda.out‚ 339.09375 disk.rbd0.in‚0# disk.dm-0.out‚68045.12353515625 disk.sda3.out‚ 339.09375& disk.nvme0n1.out‚68045.12353515625( proc.memory.availableMB‚ 1473.9296875* proc.memory.percent‚18.277668952941895$ disk./.usageGB‚548.8210868835449 disk.sda.in‚0 disk.sda2.out‚0$ memory_percent‚91.00384712219238 network.recv‚ 1.900837e+06 disk.dm-0.in‚15177.55859375$ disk.nvme0c0n1.in‚15177.55859375 disk.sda2.in‚0 disk.rbd1.out‚ 17.68359375) disk./.usagePercent‚36.83184363231845 disk.dm-1.in‚0 disk.sda3.in‚0 disk.drbd2165.out‚0 disk.rbd1.in‚0# proc.memory.rssMB‚ 2994.61328125 cpu‚25.55890838814253 network.sent‚ 8.535087e+06" disk.nvme0n1.in‚15177.55859375 proc.cpu.threads‚17 disk.dm-1.out‚ 339.09375 disk.rbd0.out‚0 disk.rbd3.in‚0 disk.rbd3.out‚0ã»'_2ð- _wandbruntime‚7476  _runtime‚7476Bè³OÆñÀ & lr-AdamW/pg2‚0.0006198346927536336 trainer/global_step‚4499 $ _timestamp‚1.784297292230804e+09 _runtime‚7476.448001293 _step‚367 & lr-AdamW/pg1‚0.0006198346927536336kõQËòÅ $ _timestamp‚1.784297292230804e+09 _runtime‚7476.448001293 _step‚367 & lr-AdamW/pg1‚0.0006198346927536336 & lr-AdamW/pg2‚0.0006198346927536336 trainer/global_step‚4499ï—̾2ó- _wandbruntime‚7476  _runtime‚7476)ô ôùO UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1293198583529472 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1297431926544384 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12782213 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404724736 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚349409820635136 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4231997625344 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚38383431 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1292639513882624 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5452475017216 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚38972775 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4233343014912 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_start/inactive_spli‡l .êt.all.current‚271 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1297430554981376 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚77356212 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12782479 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1296864243859715 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1297430554981376 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1293198557356032 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚354862259018752 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚38972775 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚38962802 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4224755405059 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1296865613888005 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 _step‚368 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4231997625344 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11545797 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚77346233 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1292639488454656 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4226100005381 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚11545802 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚38383437 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36633600 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 trainer/global_step‚4499 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 % _timestamp‚1.7842972922319064e+09 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1293198557356032 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4233343014912 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚266 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚354862663743488 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚24328281 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚38383437 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚38383431 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚77356212 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12782479 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1296864243859715 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1297430554981376 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1293198557356032 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚354862259018752 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚38972775 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚38962802 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4224755405059 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1296865613888005 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 _step‚368 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4231997625344 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11545797 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚77346233 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1292639488454656 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4226100005381 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚11545802 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚38383437 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36633600 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 trainer/global_step‚4499 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 % _timestamp‚1.7842972922319064e+09 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1293198557356032 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4233343014912 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚266 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚354862663743488 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚24328281 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚38383437 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚38383431 IDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚24333305 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚265 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12785225 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚77372910 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12784965 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚38391637 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1292917370232832 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚38971280 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4234268057600 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1297143049104137 _step‚369 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚38981267 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚77362917 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚349487675133952 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚38391643 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4225678871305 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚77372910 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1297144419132539 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4232922660864 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚38971280 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 _runtime‚7477.835248515 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚11548340 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5453662163968 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36626432 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚354941337297920 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚349487307042816 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1293476584259584 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚260 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 % _timestamp‚1.7842972936178656e+09 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4232922660864 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚77362917 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4234268057600 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚354940932580352 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚24333305 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚265 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12785225 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚77372910 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12784965 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚38391637 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1292917370232832 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 J=DeviceStatsMonitor.on_train_batc7Zch_end/active.small_pool.freed‚38971280 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4234268057600 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1297143049104137 _step‚369 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚38981267 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚77362917 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚349487675133952 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚38391643 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4225678871305 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚77372910 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1297144419132539 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4232922660864 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚38971280 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 _runtime‚7477.835248515 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚11548340 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5453662163968 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36626432 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚354941337297920 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚349487307042816 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1293476584259584 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚260 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 % _timestamp‚1.7842972936178656e+09 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4232922660864 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚77362917 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4234268057600 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚354940932580352ñ>òøc2ù- _wandbruntime‚7477  _runtime‚7477fúVwú‚ 0loss/FRt1/assignment_loss‚1.6012389332170232 2loss/SRqqt2/assignment_loss‚0.4606030766751894 2loss/FBt1/assignment_loss‚0.037125501362590345 /loss/FRt2/detection_loss‚1.5082342381063867 epoch‚11 _runtime‚7477.921427232 0loss/FRt2/assignment_loss‚0.6271841963885519 -loss/FRt1/detection_loss‚0.52531494208935 1loss/SRqqt1/detection_loss‚0.8533083654695591 0loss/FBt2/detection_loss‚0.48495471885713237 trainer/global_step‚4499 % _timestamp‚1.7842972937045083e+09 _step‚370 2loss/SRqqt1/assignment_loss‚0.5414544223154867 2loss/FBt2/assignment_loss‚0.021882220687741306 1loss/SRqqt2/detection_loss‚0.6240756298713058 .loss/FBt1/detection_loss‚2.069057126698846 %loss/total_loss‚9.354433371739162cì;A û‡ 0loss/FBt2/detection_loss‚0.48495471885713237 trainer/global_step‚4499 % _timestamp‚1.7842972937045083e+09 _step‚370 2loss/SRqqt1/assignment_loss‚0.5414544223154867 2loss/FBt2/assignment_loss‚0.021882220687741306 1loss/SRqqt2/detection_loss‚0.6240756298713058 .loss/FBt1/detection_loss‚2.069057126698846 %loss/total_loss‚9.354433371739162 0loss/FRt1/assignment_loss‚1.6012389332170232 2loss/SRqqt2/assignment_loss‚0.4606030766751894 2loss/FBt1/assignment_loss‚0.037125501362590345 /loss/FRt2/detection_loss‚1.5082342381063867 epoch‚11 _runtime‚7477.921427232 0loss/FRt2/assignment_loss‚0.6271841963885519 -loss/FRt1/detection_loss‚0.52531494208935 1loss/SRqqt1/detection_loss‚0.8533083654695591ò«†=Ÿü‚(:” ÔîèÒ§›òø gpu.0.gpu‚99.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚255.879' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚51.175799999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0/ÖÆìý‚(:á ÔîèÒíÝðù$ disk.nvme0c0n1.in‚15281.02734375 disk.sda2.out‚0 disk.sda.out‚339.25 disk.dm-1.in‚0 disk.sda1.in‚0' disk.nvme0c0n1.out‚68052.9423828125 disk.sda2.in‚0 cpu‚25.55861145449172 disk.sda1.out‚0 disk.sda3.out‚339.25 disk.sda.in‚0 disk.dm-0.in‚15281.02734375 disk.rbd3.out‚0' proc.memory.availableMB‚ 1473.890625 disk.drbd2165.in‚0* proc.memory.percent‚18.277668952941895 proc.cpu.threads‚17 network.sent‚ 8.540104e+06 network.recv‚ 1.903665e+06* disk./.usagePercent‚36.832365887688304 disk.rbd1.out‚ 17.74609375 disk.dm-1.out‚339.25 disk.rbd0.out‚0 disk.drbd2165.out‚0" disk.nvme0n1.in‚15281.02734375$ disk./.usageGB‚548.8288688659668 disk.rbd0.in‚0" disk.dm-0.out‚68052.9423828125 disk.sda3.in‚0% disk.nvme0n1.out‚68052.9423828125$ memory_percent‚91.00408554077148# proc.memory.rssMB‚ 2994.61328125 disk.rbd1.in‚0 disk.rbd3.in‚0â[Æמþ‚(:“ ãîèÒ³«÷ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚275.487' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚55.09740000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0˪'Eñÿ‚(:æ ãîèÒ«™ßø# disk.nvme0c0n1.in‚ 15281.4453125 disk.rbd1.in‚0 disk.drbd2165.in‚0 cpu‚25.558203580109627 proc.cpu.threads‚17& disk.nvme0c0n1.out‚68068.177734375! disk.nvme0n1.in‚ 15281.4453125! disk.dm-0.out‚68068.177734375 disk.sda3.in‚0 disk.rbd3.out‚0 disk.sda3.out‚ 339.5703125) disk./.usagePercent‚36.83340758234513$ disk./.usageGB‚548.8443908691406 disk.rbd3.in‚0$ disk.nvme0n1.out‚68068.177734375 disk.sda.in‚0 disk.rbd0.out‚0 disk.sda1.out‚0 disk.drbd2165.out‚0 network.recv‚ 1.90812e+06 disk.rbd0.in‚0$ memory_percent‚91.00418090820312* proc.memory.percent‚18.277668952941895 disk.rbd1.out‚ 17.74609375 disk.dm-1.out‚ 339.5703125 disk.dm-0.in‚ 15281.4453125 network.sent‚ 8.602636e+06 disk.sda2.in‚0 disk.dm-1.in‚0$ proc.memory.availableMB‚1473.875 disk.sda2.out‚0 disk.sda.out‚ 339.5703125 disk.sda1.in‚0# proc.memory.rssMB‚ 2994.61328125Ê褔€‚(:‰ òîèÒô¥éö gpu.0.gpu‚95.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚293.854' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚58.7708 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÍŽïšý‚(:ò òîèÒ¤š†ù network.sent‚ 8.608175e+06 network.recv‚ 1.911335e+06 disk.drbd2165.in‚0 disk.rbd0.in‚0 disk.drbd2165.out‚0 disk.rbd3.in‚0 disk.sda2.in‚0# proc.memory.rssMB‚ 2994.61328125 cpu‚25.557886602171028 disk.dm-1.out‚ 339.7890625 disk.dm-0.in‚ 15281.6484375 disk.sda3.in‚0 disk.rbd1.out‚ 17.80859375) disk./.usagePercent‚36.83369738287389 disk.sda1.in‚0 disk.sda1.out‚0& disk.nvme0n1.out‚68076.35205078125 disk.sda2.out‚0) proc.memory.availableMB‚ 1473.85546875 proc.cpu.threads‚17 disk.sda.out‚ 339.7890625# disk.dm-0.out‚68076.35205078125 disk.sda3.out‚ 339.7890625 disk.rbd3.out‚0( disk.nvme0c0n1.out‚68076.35205078125$ memory_percent‚91.00430011749268$ disk./.usageGB‚548.8487091064453! disk.nvme0n1.in‚ 15281.6484375 disk.rbd1.in‚0 disk.sda.in‚0 disk.dm-1.in‚0 disk.rbd0.out‚0# disk.nvme0c0n1.in‚ 15281.6484375* proc.memory.percent‚18.277668952941895 Ї„”‚‚(:‰ ïèÒÇ“ñö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚265.822' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.1644 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0v‚dûƒ‚(:ð ïèÒÞ¦†ù disk.sda1.in‚0 disk.sda1.out‚0 disk.sda3.out‚ 340.109375# proc.memory.rssMB‚ 2994.61328125 disk.sda3.in‚0 cpu‚25.55760615249266 network.recv‚ 1.914822e+06( disk.nvme0c0n1.out‚68088.33349609375 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.dm-0.in‚15281.86328125# disk.dm-0.out‚68088.33349609375 disk.rbd1.in‚0 disk.rbd3.out‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda.in‚0 disk.drbd2165.out‚0$ disk./.usageGB‚548.8556022644043 disk.drbd2165.in‚0$ memory_percent‚91.00613594055176( proc.memory.availableMB‚ 1473.5546875 network.sent‚ 8.613164e+06$ disk.nvme0c0n1.in‚15281.86328125" disk.nvme0n1.in‚15281.86328125& disk.nvme0n1.out‚68088.33349609375 disk.sda.out‚ 340.109375 proc.cpu.threads‚17 disk.dm-1.out‚ 340.109375) disk./.usagePercent‚36.83415998848826* proc.memory.percent‚18.277668952941895 disk.rbd3.in‚0 disk.rbd1.out‚ 17.80859375 Íàìž„‚(:“ ïèÒëò“÷ gpu.0.gpu‚90.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚294.335' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚58.86699999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÛØ¿…‚(:õ ïèÒã·Õø disk.dm-1.out‚ 340.23828125 disk.sda1.in‚0" disk.nvme0n1.in‚15355.33984375$ disk.nvme0c0n1.in‚15355.33984375( disk.nvme0c0n1.out‚68102.19873046875 proc.cpu.threads‚17 network.recv‚ 1.918089e+06 disk.rbd1.out‚ 17.87109375 disk.dm-1.in‚0 cpu‚25.55719741287123 disk.sda.out‚ 340.23828125 disk.sda3.out‚ 340.23828125 disk.drbd2165.out‚0 disk.rbd3.in‚0 disk.rbd3.out‚0$ memory_percent‚91.00465774536133# proc.memory.rssMB‚ 2994.61328125$ disk./.usageGB‚548.8622817993164 disk.rbd0.in‚0# disk.dm-0.out‚68102.19873046875 disk.sda1.out‚0* proc.memory.percent‚18.277668952941895) disk./.usagePercent‚36.83460825768072 disk.rbd0.out‚0 disk.dm-0.in‚15355.33984375 disk.drbd2165.in‚0 network.sent‚ 8.618313e+06 disk.sda.in‚0 disk.sda3.in‚0' proc.memory.availableMB‚ 1473.796875& disk.nvme0n1.out‚68102.19873046875 disk.sda2.in‚0 disk.rbd1.in‚0 disk.sda2.out‚0^ÑÁ|2†- _wandbruntime‚7555  _runtime‚7555+é¹Ç‡Á _runtime‚7555.125919833 _step‚371 & lr-AdamW/pg1‚0.0006181327097164662 & lr-AdamW/pg2‚0.0006181327097164662 trainer/global_step‚4549 % _timestamp‚1.7842973709087493e+09~m‘DÌˆÆ & lr-AdamW/pg1‚0.0006181327097164662 & lr-AdamW/pg2‚0.0006181327097164662 trainer/global_step‚4549 % _timestamp‚1.7842973709087493e+09 _runtime‚7555.125919833 _step‚371ó»}LÇ2‰- _wandbruntime‚7555  _runtime‚7555’)^ÿ'ŠùO =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1311376907417600 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚39398527 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1311376907417600 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1307098593861632 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1306533602791424 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 _runtime‚7555.127155065 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5512071347712 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1306533577363456 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚265 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1310805939909913 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1310804569881623 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚353284141019136 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4278313555968 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚24594024 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4272337118489 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚358796580457984 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4279658945536 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1307098593861632 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5512036811264 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1311378278980608 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚39388554 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1311378278980608 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4279658945536 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚39388554 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1307098620035072 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚78182285 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚260 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 % _timestamp‚1.7842973709098153e+09 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚358796580457984 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4279658945536 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1307098593861632 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5512036811264 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1311378278980608 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚39388554 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1311378278980608 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4279658945536 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚39388554 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1307098620035072 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚78182285 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚260 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 % _timestamp‚1.7842973709098153e+09 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1311376907417600 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1307098593861632 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1306533602791424 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0ôWøG.2Œ- _wandbruntime‚7556  _runtime‚7556P„} 'ƒN WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1307376594591744 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1311084745154447 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚358875261103616 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚78208962 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚39407019 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1311655833183232 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚38801937 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4280583988224 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1307376594591744 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5513231056384 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1311657204753408 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚358874858483200 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚78198969 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1307376620765184 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1311657204753408 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚38801943 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚39397032 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12923828 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12923574 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚24599064 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚11675490 _step‚373 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4280583988224 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1306811459141632 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 % _timestamp‚1.7842973722947767e+09 _runtime‚7556.512101624 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚78208962 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1307376594591744 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5513231056384 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1311657204753408 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚358874858483200 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚78198969 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1307376620765184 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1311657204753408 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚38801943 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚39397032 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12923828 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12923574 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚24599064 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚11675490 _step‚373 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4280583988224 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1306811459141632 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 % _timestamp‚1.7842973722947767e+09 _runtime‚7556.512101624 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚78208962 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚78208962 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚39407019 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1311655833183232 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚38801937 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4280583988224 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648õ#Þd2- _wandbruntime‚7556  _runtime‚7556r;^‚ 0loss/FRt2/assignment_loss‚0.6596468664257904 1loss/FBt1/assignment_loss‚0.24948328626007674 /loss/FBt1/detection_loss‚0.7049773057576182 /loss/FBt2/detection_loss‚0.5957539397273675 $loss/total_loss‚8.43721221480665 _step‚374 0loss/FRt1/assignment_loss‚1.6519693578315886 2loss/SRqqt2/assignment_loss‚0.5147028862401728 1loss/SRqqt1/detection_loss‚0.7609763744543644 trainer/global_step‚4549 0loss/FRt1/detection_loss‚0.49825276175972916 /loss/FRt2/detection_loss‚1.6180646568358636 % _timestamp‚1.7842973723819392e+09 1loss/SRqqt1/assignment_loss‚0.592393093887148 2loss/FBt2/assignment_loss‚0.006443656606685009 1loss/SRqqt2/detection_loss‚0.5845480290202454 epoch‚11 _runtime‚7556.598911191Ç¥¼4 ‘‡ 1loss/SRqqt1/assignment_loss‚0.592393093887148 2loss/FBt2/assignment_loss‚0.006443656606685009 1loss/SRqqt2/detection_loss‚0.5845480290202454 epoch‚11 _runtime‚7556.598911191 0loss/FRt2/assignment_loss‚0.6596468664257904 1loss/FBt1/assignment_loss‚0.24948328626007674 /loss/FBt1/detection_loss‚0.7049773057576182 /loss/FBt2/detection_loss‚0.5957539397273675 $loss/total_loss‚8.43721221480665 _step‚374 0loss/FRt1/assignment_loss‚1.6519693578315886 2loss/SRqqt2/assignment_loss‚0.5147028862401728 1loss/SRqqt1/detection_loss‚0.7609763744543644 trainer/global_step‚4549 0loss/FRt1/detection_loss‚0.49825276175972916 /loss/FRt2/detection_loss‚1.6180646568358636 % _timestamp‚1.7842973723819392e+09ö;E$”’‚(:‰ ŸïèÒ£îéø gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚347.757' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚69.5514 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0–íi“‚(:÷ ŸïèÒøêêù* proc.memory.percent‚18.277668952941895" disk.nvme0n1.in‚15355.54296875 disk.rbd1.out‚ 17.87109375& disk.nvme0n1.out‚68112.47998046875 network.sent‚ 8.623326e+06 disk.sda3.in‚0 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.sda.in‚0 disk.sda3.out‚ 340.48046875 disk.sda2.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.dm-0.in‚15355.54296875 disk.sda1.in‚0$ disk./.usageGB‚548.8672256469727 disk.dm-1.in‚0$ memory_percent‚91.00308418273926( disk.nvme0c0n1.out‚68112.47998046875 disk.sda1.out‚0 proc.cpu.threads‚17$ disk.nvme0c0n1.in‚15355.54296875 disk.sda.out‚ 340.48046875( proc.memory.availableMB‚ 1474.0546875 cpu‚25.55691121548583 network.recv‚ 1.921693e+06 disk.drbd2165.in‚0 disk.sda2.out‚0 disk.rbd1.in‚0 disk.dm-1.out‚ 340.48046875# disk.dm-0.out‚68112.47998046875* disk./.usagePercent‚36.834940043445094 disk.drbd2165.out‚0# proc.memory.rssMB‚ 2994.61328125× R—””‚(:‰ ®ïèÒÑ÷Þö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚293.512' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚58.7024 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ³Qtã•‚(:Ø ®ïèÒøÏíø disk.sda.in‚0 disk.sda.out‚ 340.9296875 disk.sda3.in‚0 disk.rbd3.in‚0! disk.nvme0n1.in‚ 15355.7578125 disk.nvme0n1.out‚68120.75 disk.rbd1.out‚ 17.93359375* proc.memory.percent‚18.277668952941895$ disk./.usageGB‚548.8742256164551 disk.sda2.in‚0 disk.rbd1.in‚0) proc.memory.availableMB‚ 1473.82421875 disk.rbd0.in‚0 disk.sda2.out‚0$ memory_percent‚91.00449085235596 cpu‚25.556509595892773* disk./.usagePercent‚36.835409817270424 disk.dm-1.in‚0 disk.rbd0.out‚0 disk.sda1.in‚0 disk.sda3.out‚ 340.9296875 disk.nvme0c0n1.out‚68120.75# proc.memory.rssMB‚ 2994.61328125 proc.cpu.threads‚17 disk.dm-1.out‚ 340.9296875 disk.dm-0.in‚ 15355.7578125 disk.sda1.out‚0 disk.drbd2165.out‚0# disk.nvme0c0n1.in‚ 15355.7578125 network.sent‚ 8.685825e+06 network.recv‚ 1.925794e+06 disk.dm-0.out‚68120.75 disk.drbd2165.in‚0 disk.rbd3.out‚0æÑ“ž˜‚(:“ ÌïèÒšŸø gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚262.359' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚52.47179999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0é¼·™‚(:÷ ÌïèÒÈŽìø disk.rbd1.in‚ 0.01171875 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda1.out‚0# proc.memory.rssMB‚ 2994.61328125* proc.memory.percent‚18.277668952941895 disk.sda.out‚ 341.421875 disk.dm-1.out‚ 341.421875 network.recv‚ 1.932669e+06$ disk./.usageGB‚548.8892974853516 disk.rbd3.in‚0" disk.nvme0n1.in‚15433.41796875 disk.sda3.in‚0 disk.sda3.out‚ 341.421875 disk.drbd2165.out‚0 proc.cpu.threads‚17) disk./.usagePercent‚36.83642130303821% disk.nvme0n1.out‚68143.6611328125 disk.drbd2165.in‚0 network.sent‚ 8.696495e+06$ disk.nvme0c0n1.in‚15433.41796875 disk.rbd1.out‚18.21630859375 disk.sda.in‚0 disk.dm-0.in‚15433.41796875 disk.rbd3.out‚0' disk.nvme0c0n1.out‚68143.6611328125" disk.dm-0.out‚68143.6611328125 cpu‚25.555828658475203 disk.sda2.out‚0$ memory_percent‚91.00151062011719% proc.memory.availableMB‚ 1474.3125 disk.sda2.in‚0 disk.dm-1.in‚0 disk.sda1.in‚0ÃC´š‚(:’ ÛïèÒù…Ùö gpu.0.gpu‚11.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚328.497' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚65.69940000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ™Ë ›‚(:þ ÛïèÒê˜ôø& disk.nvme0n1.out‚68155.34716796875( proc.memory.availableMB‚ 1473.5703125 proc.cpu.threads‚17( disk.nvme0c0n1.out‚68155.34716796875 cpu‚25.55555506786304 disk.sda2.out‚0# disk./.usageGB‚548.896183013916 disk.sda.in‚0 disk.sda1.in‚0 disk.sda3.out‚ 341.66796875 disk.sda.out‚ 341.66796875 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.sda1.out‚0! disk.nvme0n1.in‚ 15433.6328125 network.sent‚ 8.701563e+06# disk.dm-0.out‚68155.34716796875 disk.drbd2165.out‚0 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.sda2.in‚0* proc.memory.percent‚18.277668952941895 disk.rbd1.in‚ 0.01171875 disk.dm-1.out‚ 341.66796875 disk.dm-0.in‚ 15433.6328125# proc.memory.rssMB‚ 2994.61328125 disk.drbd2165.in‚0# disk.nvme0c0n1.in‚ 15433.6328125 network.recv‚ 1.936164e+06 disk.rbd1.out‚18.21630859375 disk.rbd0.out‚0 disk.sda3.in‚0$ memory_percent‚91.00604057312012) disk./.usagePercent‚36.83688339663751ã±Av2œ- _wandbruntime‚7633  _runtime‚7633¢,Tľ % lr-AdamW/pg1‚0.000616413243237049 % lr-AdamW/pg2‚0.000616413243237049 trainer/global_step‚4599 $ _timestamp‚1.784297449597351e+09 _runtime‚7633.814507124 _step‚375"¹ qɞà % lr-AdamW/pg1‚0.000616413243237049 % lr-AdamW/pg2‚0.000616413243237049 trainer/global_step‚4599 $ _timestamp‚1.784297449597351e+09 _runtime‚7633.814507124 _step‚375÷—ÆØ<2Ÿ- _wandbruntime‚7633  _runtime‚7633._( î øO G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚39204031 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5572031244800 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1320998656540672 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1324744895903531 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚254ÂlTf%27968 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1325324631416832 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13059169 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1325324631416832 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4324629486592 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4325974876160 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1325323259853824 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4324629486592 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1320998630367232 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚39204037 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚362730895275520 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1324746265931821 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚362730492647936 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚79028316 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚39814306 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚39824279 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚11800102 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚39204031 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1320998630367232 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚357158461403136 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1320427691700224 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1320427666272256 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚39814306 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4317229631275 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚79018337 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4318574231597 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚376 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13059420 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 _runtime‚ 7633.81573409 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1325323259853824 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11800097 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0„õ¢(¡ýO UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚39204037 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚362730895275520 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1324746265931821 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚362730492647936 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚79028316 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚39814306 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚39824279 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚11800102 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚39204031 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1320998630367232 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚357158461403136 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1320427691700224 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1320427666272256 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚39814306 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4317229631275 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚79018337 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4318574231597 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚376 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13059420 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 _runtime‚ 7633.81573409 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1325323259853824 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11800097 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚39204031 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5572031244800 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1320998656540672 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1324744895903531 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1325324631416832 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13059169 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1325324631416832 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4324629486592 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4325974876160 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1325323259853824 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4324629486592 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1320998630367232 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648øs¦g³Ÿ¢‚(:” êïèÒÆ«üö gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚271.506' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚54.301199999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¢óÖ £‚(:€ êïèÒ©±¬ù disk.rbd3.out‚0" disk.nvme0n1.in‚15434.05078125 disk.rbd1.out‚ 18.4482421875 cpu‚25.555149379667938 disk.rbd0.in‚0$ memory_percent‚90.99962711334229# proc.memory.rssMB‚ 2994.61328125 network.sent‚ 8.706684e+06 network.recv‚ 1.939861e+06 disk.sda1.out‚0' disk.nvme0c0n1.out‚68168.9130859375% disk.nvme0n1.out‚68168.9130859375 disk.rbd1.in‚ 0.01171875 disk.sda.out‚ 341.80078125 disk.dm-1.out‚ 341.80078125 disk.drbd2165.out‚0 disk.sda2.out‚0 proc.cpu.threads‚17 disk.dm-0.in‚15434.05078125 disk.sda2.in‚0* proc.memory.percent‚18.277668952941895 disk.sda3.in‚0$ disk.nvme0c0n1.in‚15434.05078125 disk.sda.in‚0 disk.dm-1.in‚0) proc.memory.availableMB‚ 1474.62109375* disk./.usagePercent‚36.837354706508044 disk.sda1.in‚0# disk./.usageGB‚548.903205871582" disk.dm-0.out‚68168.9130859375 disk.sda3.out‚ 341.80078125 disk.rbd0.out‚0 disk.drbd2165.in‚0 disk.rbd3.in‚0Û†WÈ2¤- _wandbruntime‚7635  _runtime‚7635”ˆ '¥ƒN WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1325603557189632 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1321276631097344 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚39212243 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4325554522112 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚249 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚357235947810816 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚79035021 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚39212243 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1325603557189632 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1321276657270784 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚357236315901952 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4326899918848 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚39822784 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13062154 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4319497697955 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 _runtime‚7635.202181951 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚24864550 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1320705573478400 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1325602185619456 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5573264896512 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1325025071176355 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 trainer/global_step‚4599 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1325023701147953 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 _step‚377 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4326899918848 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1321276631097344 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13061910 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5573230367232 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚79045014 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11802645 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚39822784 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1320705548050432 % _timestamp‚1.7842974509845877e+09 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1321276657270784 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚79045014 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚11802640 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4325554522112 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚39832771 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0/XLñצˆN QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1321276631097344 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13061910 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_äW6n7!pool.freed‚ 5573230367232 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚79045014 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11802645 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚39822784 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1320705548050432 % _timestamp‚1.7842974509845877e+09 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1321276657270784 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚79045014 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚11802640 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4325554522112 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚39832771 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1325603557189632 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1321276631097344 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚39212243 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4325554522112 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚249 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚357235947810816 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚79035021 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚39212243 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1325603557189632 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1321276657270784 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚357236315901952 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4326899918848 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚39822784 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13062154 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4319497697955 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 _runtime‚7635.202181951 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚24864550 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1320705573478400 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1325602185619456 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5573264896512 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1325025071176355 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 trainer/global_step‚4599 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1325023701147953 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 _step‚377 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4326899918848 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175ù¯ñ΂2§- _wandbruntime‚7635  _runtime‚7635íP?@ ¨… epoch‚11 trainer/global_step‚4599 2loss/FBt1/assignment_loss‚0.040485973497846474 /loss/FRt2/detection_loss‚1.4321461551926202 1loss/SRqqt1/detection_loss‚0.8014540767954703 /loss/FBt1/detection_loss‚0.9634600697151784 _runtime‚7635.287980401 %loss/total_loss‚8.363919697290724 0loss/FRt1/assignment_loss‚1.6854690699697814 2loss/SRqqt1/assignment_loss‚0.5581261781655265 2loss/FBt2/assignment_loss‚0.002334168237557602 1loss/SRqqt2/detection_loss‚0.6065967542422103 % _timestamp‚1.7842974510710666e+09 _step‚378 0loss/FRt2/assignment_loss‚0.7123759809835457 2loss/SRqqt2/assignment_loss‚0.5205454735485575 /loss/FRt1/detection_loss‚0.5532502612280865 0loss/FBt2/detection_loss‚0.48767553571434247¹zõ©Š 2loss/FBt2/assignment_loss‚0.002334168237557602 1loss/SRqqt2/detection_loss‚0.6065967542422103 % _timestamp‚1.7842974510710666e+09 _step‚378 0loss/FRt2/assignment_loss‚0.7123759809835457 2loss/SRqqt2/assignment_loss‚0.5205454735485575 /loss/FRt1/detection_loss‚0.5532502612280865 0loss/FBt2/detection_loss‚0.48767553571434247 epoch‚11 trainer/global_step‚4599 2loss/FBt1/assignment_loss‚0.040485973497846474 /loss/FRt2/detection_loss‚1.4321461551926202 1loss/SRqqt1/detection_loss‚0.8014540767954703 /loss/FBt1/detection_loss‚0.9634600697151784 _runtime‚7635.287980401 %loss/total_loss‚8.363919697290724 0loss/FRt1/assignment_loss‚1.6854690699697814 2loss/SRqqt1/assignment_loss‚0.5581261781655265úlØY(žª‚(:“ ùïèÒ络÷ gpu.0.gpu‚100.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚331.35' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚66.27000000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0{wµ «‚(:÷ ùïèÒ¾µ¯ú disk.drbd2165.in‚0 disk.rbd3.out‚0( disk.nvme0c0n1.out‚68178.85009765625" disk.nvme0n1.in‚15434.25390625 disk.sda.in‚0 disk.rbd1.in‚ 0.01171875 disk.rbd1.out‚ 18.4482421875) disk./.usagePercent‚36.83762863456968# disk.dm-0.out‚68178.85009765625 disk.sda3.in‚0& disk.nvme0n1.out‚68178.85009765625 disk.sda2.out‚0 disk.dm-1.in‚0 disk.dm-1.out‚342.1875 disk.rbd0.out‚0 disk.dm-0.in‚15434.25390625 disk.drbd2165.out‚0 network.sent‚ 8.759185e+06 network.recv‚ 1.944515e+06$ disk./.usageGB‚548.9072875976562 disk.sda1.in‚0# proc.memory.rssMB‚ 2994.61328125* proc.memory.percent‚18.277668952941895 proc.cpu.threads‚17 disk.sda.out‚342.1875 disk.sda1.out‚0 disk.sda3.out‚342.1875 disk.sda2.in‚0$ memory_percent‚90.99962711334229 cpu‚25.554871602865305 disk.rbd0.in‚0 disk.rbd3.in‚0$ disk.nvme0c0n1.in‚15434.25390625) proc.memory.availableMB‚ 1474.62109375×?a”¬‚(:‰ ˆðèÒ‰‰Ùö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚273.366' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.6732 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ?­‚(:õ ˆðèÒ§©åø) disk./.usagePercent‚36.83808765607858$ disk./.usageGB‚548.9141273498535# proc.memory.rssMB‚ 2994.61328125 disk.dm-1.out‚ 342.32421875 disk.rbd0.out‚0 disk.sda1.out‚0 cpu‚25.554515347463017 network.recv‚ 1.948397e+06 disk.sda3.in‚0' disk.nvme0c0n1.out‚68186.5927734375 disk.nvme0n1.in‚ 15498.09375( proc.memory.availableMB‚ 1474.6171875* proc.memory.percent‚18.277668952941895 disk.rbd0.in‚0 disk.drbd2165.out‚0% disk.nvme0n1.out‚68186.5927734375# memory_percent‚90.9996509552002 disk.sda.in‚0 disk.dm-1.in‚0 disk.dm-0.in‚ 15498.09375 disk.sda3.out‚ 342.32421875 disk.rbd3.in‚0 disk.drbd2165.in‚0! disk.nvme0c0n1.in‚ 15498.09375 disk.sda2.out‚0 disk.rbd1.out‚ 18.5107421875 proc.cpu.threads‚17 disk.sda.out‚ 342.32421875 disk.sda1.in‚0 disk.rbd3.out‚0 disk.rbd1.in‚ 0.01171875 network.sent‚ 8.811599e+06" disk.dm-0.out‚68186.5927734375 disk.sda2.in‚0ÜÎÿ”®‚(:‰ —ðèÒ̉ùö gpu.0.gpu‚99.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚286.263' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚57.2526 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ʯ‹º ¯‚(:þ —ðèÒïÒ”ù disk.rbd0.in‚0" disk.nvme0n1.in‚15498.51171875 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.sda.out‚ 342.859375* proc.memory.percent‚18.277668952941895$ disk.nvme0c0n1.in‚15498.51171875 disk.sda3.out‚ 342.859375* disk./.usagePercent‚36.838561014009386) proc.memory.availableMB‚ 1474.86328125 proc.cpu.threads‚17 disk.sda3.in‚0 disk.rbd3.in‚0 disk.dm-1.out‚ 342.859375 cpu‚25.554211866816715 disk.rbd0.out‚0 disk.sda2.in‚0 disk.sda1.in‚0 disk.sda.in‚0$ memory_percent‚90.99814891815186# proc.memory.rssMB‚ 2994.61328125( disk.nvme0c0n1.out‚68196.97412109375& disk.nvme0n1.out‚68196.97412109375 disk.sda2.out‚0 disk.rbd1.out‚ 18.5107421875 disk.dm-1.in‚0 network.sent‚ 8.816438e+06 network.recv‚ 1.950987e+06$ disk./.usageGB‚548.9211807250977 disk.drbd2165.out‚0 disk.rbd1.in‚ 0.01171875 disk.dm-0.in‚15498.51171875# disk.dm-0.out‚68196.97412109375 disk.rbd3.out‚03´é’°‚(:‡ ¦ðèÒ˜Üïö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚287.91' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚57.582 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÔsÎ᱂(:ý ¦ðèÒ¨æ†ù disk.rbd0.in‚0 disk.rbd3.out‚0 disk.sda2.in‚0 disk.sda2.out‚0$ memory_percent‚90.99812507629395 proc.cpu.threads‚17 disk.rbd1.in‚ 0.01171875 disk.sda1.in‚0 cpu‚25.55391292172684 disk.rbd1.out‚ 18.5888671875 disk.sda.out‚ 343.0234375* disk./.usagePercent‚36.838840574236784# disk.dm-0.out‚68209.43115234375 disk.drbd2165.out‚0* proc.memory.percent‚18.277668952941895$ disk./.usageGB‚548.9253463745117 disk.dm-1.out‚ 343.0234375( disk.nvme0c0n1.out‚68209.43115234375 network.sent‚ 8.8207e+06 disk.rbd0.out‚0 disk.sda1.out‚0$ disk.nvme0c0n1.in‚15498.71484375 disk.sda.in‚0 disk.drbd2165.in‚0 disk.rbd3.in‚0# proc.memory.rssMB‚ 2994.61328125 disk.sda3.out‚ 343.0234375 network.recv‚ 1.953763e+06 disk.dm-0.in‚15498.71484375 disk.sda3.in‚0" disk.nvme0n1.in‚15498.71484375& disk.nvme0n1.out‚68209.43115234375( proc.memory.availableMB‚ 1474.8671875 disk.dm-1.in‚0óîf”²‚(:‰ µðèÒÓ¡öø gpu.0.gpu‚90.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚270.943' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.1886 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Qéþ‚³‚(:ù µðèÒš“¬ù( proc.memory.availableMB‚ 1474.8671875& disk.nvme0n1.out‚68219.16259765625 disk.sda.out‚ 343.546875# disk.nvme0c0n1.in‚ 15498.9296875 disk.dm-1.out‚ 343.546875 disk.rbd3.in‚0 disk.rbd3.out‚0 proc.cpu.threads‚17 disk.sda1.in‚0 disk.sda3.out‚ 343.546875 disk.sda.in‚0 disk.dm-0.in‚ 15498.9296875* proc.memory.percent‚18.277668952941895 cpu‚25.55354777061012! disk.nvme0n1.in‚ 15498.9296875 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-1.in‚0 disk.sda3.in‚0 disk.drbd2165.in‚0( disk.nvme0c0n1.out‚68219.16259765625$ memory_percent‚90.99812507629395$ disk./.usageGB‚548.9321403503418 disk.rbd0.out‚0 disk.sda1.out‚0 disk.drbd2165.out‚0# proc.memory.rssMB‚ 2994.61328125# disk.dm-0.out‚68219.16259765625 network.sent‚ 8.825758e+06 network.recv‚ 1.957404e+06* disk./.usagePercent‚36.839296523655264 disk.rbd1.in‚ 0.01171875 disk.rbd1.out‚ 18.5888671875 disk.rbd0.in‚0žÐï2´- _wandbruntime‚7712  _runtime‚7712Üì†úĵ¾ % lr-AdamW/pg1‚0.000614676403753501 % lr-AdamW/pg2‚0.000614676403753501 trainer/global_step‚4649 $ _timestamp‚1.784297528089369e+09 _runtime‚7712.306532402 _step‚379V¦É=ɶà % lr-AdamW/pg1‚0.000614676403753501 % lr-AdamW/pg2‚0.000614676403753501 trainer/global_step‚4649 $ _timestamp‚1.784297528089369e+09 _runtime‚7712.306532402 _step‚379ûêsI¥2·- _wandbruntime‚7712  _runtime‚7712€ÆsRÿ'¸ùO VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1339269612290048 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚39614331 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚40240058 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4372290806784 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚366665315061248 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5632165182976 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚39614331 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚366664912433664 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1338685221925439 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5632130646528 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 _step‚380 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1334321755181056 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1339270983853056 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1334321780609024 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚79854389 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1339269612290048 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚11927252 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 trainer/global_step‚4649 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚40240058 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13197307 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1334898693046272 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚25124812 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4364811344705 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 _runtime‚7712.307737607 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1338686591953729 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 % _timestamp‚1.7842975280904372e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11927247 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1334898666872832 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚253 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1339270983853056 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚40250031 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4363466744383 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1334898666872832 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4370945417216 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13197560 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚39614337 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 “ õý¹þO ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1334321755181056 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1339270983853056 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1334321780609024 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚79854389 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1339269612290048 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚11927252 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 trainer/global_step‚4649 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚40240058 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13197307 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1334898693046272 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚25124812 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4364811344705 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 _runtime‚7712.307737607 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1338686591953729 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 % _timestamp‚1.7842975280904372e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11927247 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1334898666872832 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚253 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚133;õá9270983853056 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚40250031 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4363466744383 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1334898666872832 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4370945417216 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13197560 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚39614337 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1339269612290048 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚39614331 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚40240058 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4372290806784 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚366665315061248 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5632165182976 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚39614331 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚366664912433664 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1338685221925439 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5632130646528 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 _step‚380 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0ü¸ŒeÉ2º- _wandbruntime‚7713  _runtime‚7713†QVà '»ƒN XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚366743610030592 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚79881066 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚79881066 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4364390210629 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚25129849 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚11929790 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚39622537 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚366744010553856 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚32432128 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚40258523 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4371870452736 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1339549909625856 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1339548538055680 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5633341835776 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1335176667602944 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 % _timestamp‚1.7842975294749002e+09 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚25130089 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820~z(4'¼ˆN _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚361110268194816 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4371870452736 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚40248536 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1335176693776384 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚79881066 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚79881066 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4364390210629 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚25129849 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚11929790 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚39622537 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚366744010553856 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚32432128 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚40258523 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4371870452736 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1339549909625856 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1339548538055680 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5633341835776 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1335176667602944 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 % _timestamp‚1.7842975294749002e+09 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚25130089 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚366743610030592 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚370599375066624 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1353215964726272 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1348798703378432 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 _step‚384 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1353215964726272 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚25390288 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚4699 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚40675783 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚40665810 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚40024631 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1348215869517824 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚40024631 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1352626917975637 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚80690441 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚40665810 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 400530432 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚32439296 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5692272895488 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚40675783 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1353217336289280 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4409703857491 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0«¬ºÌ(ÏýO PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚40024637 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4417261347840 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚370599375066624 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1353215964726272 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1348798703378432 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 _step‚384 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1353215964726272 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚25390288 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚4699 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚40675783 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚40665810 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚40024631 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1348215869517824 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚40024631 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1352626917975637 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚80690441 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚40665810 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 400530432 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚32439296 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5692272895488 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚40675783 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1353217336289280 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4409703857491 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 % _timestamp‚1.7842976068064156e+09 _runtime‚ 7791.02373377 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚40024637 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51€ãôj·2Ð- _wandbruntime‚7792  _runtime‚7792sÊCó¥&уN I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚32432128 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚25395326 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1348493751296000 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13338386 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚364984588578816 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚40032837 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5693478219264 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚80707125 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4411971924171 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚80717118 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1353496262062080 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1349076704108544 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚40674288 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚364984956669952 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4410627323737 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚40684275 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚25395554 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚223 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚40684275 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 _step‚385 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚40674288 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚80707125 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1349076730281984 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 400523264 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1352905723220171 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚80717118 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1353494890491904 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚40032843 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4419531780096 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_"¶~d_pool.peak‚ 1719664640 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175NO²'ÒˆN H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚40684275 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚25395554 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚223 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚40684275 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 _step‚385 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚40674288 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚80707125 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1349076730281984 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 400523264 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1352905723220171 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚80717118 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1353494890491904 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚40032843 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4419531780096 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚32432128 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚25395326 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1348493751296000 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13338386 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚364984588578816 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚40032837 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5693478219264 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚80707125 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4411971924171 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚80717118 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1353496262062080 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1349076704108544 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚40674288 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚364984956669952 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4410627323737—ƒóý2Ó- _wandbruntime‚7792  _runtime‚7792…fäÔ 0loss/FBt2/detection_loss‚0.42388422204802434 %loss/total_loss‚8.277696405571984 _step‚386 .loss/FRt1/assignment_loss‚1.63675352161037 .loss/FRt1/detection_loss‚0.523249533584143 1loss/SRqqt1/detection_loss‚0.8434518419912663 % _timestamp‚1.7842976082767627e+09 0loss/FRt2/assignment_loss‚0.8226407975560487 2loss/SRqqt2/assignment_loss‚0.5318360903515024 1loss/SRqqt2/detection_loss‚0.6539510993679977 epoch‚11 2loss/SRqqt1/assignment_loss‚0.5417696224140516 2loss/FBt1/assignment_loss‚0.011387189277479264 2loss/FBt2/assignment_loss‚0.009051959145809196 trainer/global_step‚4699 _runtime‚7792.493679292 .loss/FRt2/detection_loss‚1.555628235936353 /loss/FBt1/detection_loss‚0.7240922922889391uQ&Ø Õ† %loss/total_loss‚8.277696405571984 _step‚386 .loss/FRt1/assignment_loss‚1.63675352161037 .loss/FRt1/detection_loss‚0.523249533584143 1loss/SRqqt1/detection_loss‚0.8434518419912663 % _timestamp‚1.7842976082767627e+09 0loss/FRt2/assignment_loss‚0.8226407975560487 2loss/SRqqt2/assignment_loss‚0.5318360903515024 1loss/SRqqt2/detection_loss‚0.6539510993679977 epoch‚11 2loss/SRqqt1/assignment_loss‚0.5417696224140516 2loss/FBt1/assignment_loss‚0.011387189277479264 2loss/FBt2/assignment_loss‚0.009051959145809196 trainer/global_step‚4699 _runtime‚7792.493679292 .loss/FRt2/detection_loss‚1.555628235936353 /loss/FBt1/detection_loss‚0.7240922922889391 0loss/FBt2/detection_loss‚0.42388422204802434‚¡àˆŸÖ‚(:” ñèÒáòÛö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚266.668' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚53.333600000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 >ׂ̣(:ø ñèÒ€ñ…ù network.sent‚ 8.914126e+06 network.recv‚ 1.978423e+06 disk.drbd2165.out‚0& disk.nvme0n1.out‚68281.67431640625$ disk./.usageGB‚548.9748878479004 disk.sda.out‚ 345.16796875 disk.rbd0.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-1.out‚ 345.16796875 disk.sda1.in‚0 disk.drbd2165.in‚0 disk.dm-1.in‚0 disk.sda3.in‚0 disk.rbd1.in‚ 0.01171875# proc.memory.rssMB‚ 2994.61328125 disk.rbd1.out‚ 18.7919921875 disk.rbd0.out‚0 disk.dm-0.in‚ 15638.859375 disk.sda3.out‚ 345.16796875 disk.rbd3.out‚0* proc.memory.percent‚18.277668952941895 cpu‚25.551623528935146) disk./.usagePercent‚36.84216534408395 disk.sda1.out‚0" disk.nvme0c0n1.in‚ 15638.859375( disk.nvme0c0n1.out‚68281.67431640625$ proc.memory.availableMB‚1473.125 proc.cpu.threads‚17 disk.sda.in‚0# disk.dm-0.out‚68281.67431640625 disk.rbd3.in‚0 disk.nvme0n1.in‚ 15638.859375$ memory_percent‚91.00875854492188jÌ&ŸØ‚(:” žñèÒ•Ýç÷ gpu.0.gpu‚100.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚331.737' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚66.34740000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0zúD4Ù‚(:û žñèÒÁµÛø disk.dm-1.in‚0 disk.dm-1.out‚ 348.2109375" disk.nvme0n1.in‚15639.07421875 disk.rbd1.in‚ 0.01171875 cpu‚25.551335642450994 network.recv‚ 1.982917e+06 disk.dm-0.in‚15639.07421875 disk.sda1.out‚0 disk.sda3.out‚ 348.2109375 disk.rbd3.out‚0$ disk.nvme0c0n1.in‚15639.07421875 disk.rbd1.out‚ 18.8701171875# memory_percent‚91.0088062286377 disk.sda3.in‚0 disk.sda.out‚ 348.2109375( proc.memory.availableMB‚ 1473.1171875 disk.rbd0.in‚0 disk.rbd0.out‚0" disk.dm-0.out‚68311.5751953125# proc.memory.rssMB‚ 2994.61328125 disk.sda1.in‚0 network.sent‚ 8.977199e+06 disk.drbd2165.out‚0 disk.drbd2165.in‚0 disk.rbd3.in‚0% disk.nvme0n1.out‚68311.5751953125* proc.memory.percent‚18.277668952941895) disk./.usagePercent‚36.84298072808051' disk.nvme0c0n1.out‚68311.5751953125 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda.in‚0 proc.cpu.threads‚17$ disk./.usageGB‚548.9870376586914‰MGTŸÚ‚(:” ­ñèÒÊ…÷ö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚269.302' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚53.860400000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0vŠÞnùÛ‚(:î ­ñèÒó‹ù network.sent‚ 8.982264e+06 disk.rbd3.in‚0$ disk.nvme0c0n1.in‚15639.27734375# disk.nvme0n1.out‚68330.44921875 disk.sda1.in‚0) disk./.usagePercent‚36.84297330386202% disk.nvme0c0n1.out‚68330.44921875 disk.sda2.out‚0 disk.sda.in‚0 disk.rbd0.in‚0 disk.dm-0.out‚68330.44921875# memory_percent‚91.0088062286377* proc.memory.percent‚18.277668952941895 disk.drbd2165.in‚0 disk.rbd1.in‚ 0.01171875 disk.dm-0.in‚15639.27734375 disk.sda1.out‚0 disk.sda3.in‚0$ disk./.usageGB‚548.9869270324707 disk.drbd2165.out‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 351.03125 disk.rbd0.out‚0 disk.sda3.out‚ 351.03125( proc.memory.availableMB‚ 1473.1171875" disk.nvme0n1.in‚15639.27734375 disk.sda2.in‚0 disk.sda.out‚ 351.03125# proc.memory.rssMB‚ 2994.61328125 disk.rbd3.out‚0 cpu‚25.55094829660276 proc.cpu.threads‚17 network.recv‚ 1.986382e+06 disk.rbd1.out‚ 18.8701171875Võ| “Ü‚(:ˆ ¼ñèÒ¡çüö gpu.0.gpu‚99.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚288.775' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚57.755 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0#›HþÝ‚(:ó ¼ñèÒšº©ù disk.sda3.in‚0 disk.drbd2165.in‚0 disk.rbd3.out‚0# disk.nvme0n1.out‚68343.51171875# memory_percent‚91.0088300704956 network.recv‚ 1.989721e+06 disk.sda1.out‚0 disk.drbd2165.out‚0% disk.nvme0c0n1.out‚68343.51171875 disk.rbd1.in‚ 0.01171875# proc.memory.rssMB‚ 2994.61328125 disk.sda.out‚ 353.4296875 disk.dm-1.out‚ 353.4296875 disk.dm-0.out‚68343.51171875 disk.sda2.out‚0 proc.cpu.threads‚17 disk.sda3.out‚ 353.4296875) proc.memory.availableMB‚ 1473.11328125* proc.memory.percent‚18.277668952941895 disk.dm-1.in‚0 disk.sda1.in‚0 disk.rbd3.in‚0 disk.sda2.in‚0 disk.sda.in‚0 disk.rbd0.in‚0$ disk./.usageGB‚548.9930229187012 disk.rbd0.out‚0! disk.nvme0n1.in‚ 15639.6953125 cpu‚25.55069092869547 network.sent‚ 8.987289e+06* disk./.usagePercent‚36.843382403901735 disk.dm-0.in‚ 15639.6953125# disk.nvme0c0n1.in‚ 15639.6953125 disk.rbd1.out‚ 18.9326171875äñNžÞ‚(:“ ËñèÒúÍêö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚300.808' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚60.16160000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0D™o†ß‚(:û ËñèÒÿûø disk.rbd0.out‚0 disk.sda1.out‚0& disk.nvme0n1.out‚68361.70751953125 disk.rbd1.out‚ 18.9326171875) proc.memory.availableMB‚ 1473.11328125$ disk./.usageGB‚548.9944190979004 disk.sda.in‚0# disk.dm-0.out‚68361.70751953125 disk.sda3.in‚0 disk.sda3.out‚ 354.8046875 disk.drbd2165.in‚0# disk.nvme0c0n1.in‚ 15830.6953125 cpu‚25.550311501402703 disk.dm-1.out‚ 354.8046875 disk.rbd0.in‚0 disk.dm-0.in‚ 15830.6953125 disk.sda1.in‚0 disk.drbd2165.out‚0 disk.rbd3.out‚0 disk.sda2.out‚0 disk.rbd1.in‚ 0.01171875 network.recv‚ 1.993331e+06 disk.sda.out‚ 354.8046875 disk.dm-1.in‚0# proc.memory.rssMB‚ 2994.61328125* proc.memory.percent‚18.277668952941895 network.sent‚ 8.99233e+06 disk.rbd3.in‚0! disk.nvme0n1.in‚ 15830.6953125 disk.sda2.in‚0# memory_percent‚91.0088300704956 proc.cpu.threads‚17( disk.nvme0c0n1.out‚68361.70751953125) disk./.usagePercent‚36.84347610265927I0|2à- _wandbruntime‚7869  _runtime‚7869)3µÇáÁ & lr-AdamW/pg1‚0.0006111510530985008 & lr-AdamW/pg2‚0.0006111510530985008 trainer/global_step‚4749 % _timestamp‚1.7842976852571454e+09 _runtime‚7869.474304372 _step‚387³OÌâÆ & lr-AdamW/pg1‚0.0006111510530985008 & lr-AdamW/pg2‚0.0006111510530985008 trainer/global_step‚4749 % _timestamp‚1.7842976852571454e+09 _runtime‚7869.474304372 _step‚387ƒ=o©62ã- _wandbruntime‚7869  _runtime‚7869ºíÚƒþ'äøO WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4463577278464 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5752718289408 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4457285570921 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1367163688725504 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4463577278464 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1362109932998656 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1362698766057472 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 $ _timestamp‚1.784297685258258e+09 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚241 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚41091562 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1367162317162496 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚25656000 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13474207 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 trainer/global_step‚4749 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚246 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1362698766057472 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚368781422555136 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚40434931 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1367163688725504 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1366567243997545 _step‚388 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 _runtime‚7869.475561916 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚374534541374976 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1366565873969255 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1367162317162496 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4464922668032 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5752750728704 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚40434937 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚12181552 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1362698739884032 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4455940970599 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚40434937 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚368781790646272 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚374534140844544 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1362698739884032 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚81526493 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚81536472 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚25655754 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12181547 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚41101535 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚40434931 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚41091562 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13474448pÊ1D­ åýO aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚368781422555136 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚40434931 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1367163688725504 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1366567243997545 _step‚388 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 _runtime‚7869.475561916 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚374534541374976 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1366565873969255 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1367162317162496 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4464922668032 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5752750728704 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚40434937 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚12181552 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1362698739884032 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4455940970599 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚40434937 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚368781790646272 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚374534140844544 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1362698739884032 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚81526493 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚81536472 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚25655754 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12181547 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚41101535 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚40434931 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚41091562 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13474448 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4463577278464 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5752718289408 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4457285570921 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1367163688725504 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4463577278464 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1362109932998656 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1362698766057472 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 $ _timestamp‚1.784297685258258e+09 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚241 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚41091562 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1367162317162496 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚25656000 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13474207 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 trainer/global_step‚4749 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚246 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1362698766057472 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627„.£î2æ- _wandbruntime‚7870  _runtime‚7870;êë 'çƒN NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚41100040 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚374613237400064 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 trainer/global_step‚4749 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚41100040 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4464502313984 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚389 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5753927913984 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚81553170 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚40443143 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1362387814776832 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚25661039 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚368858908962816 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1366846049242079 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚40443137 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚40443143 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640õ/Qß'èˆN 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1366846049242079 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚40443137 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚40443143 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚41100040 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚374613237400064 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 trainer/global_step‚4749 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚41100040 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4464502313984 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚389 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5753927913984 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚81553170 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚40443143 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1362387814776832 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚25661039 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚368858908962816 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0…˽.U2é- _wandbruntime‚7870  _runtime‚7870¡5¶Üê€ 0loss/FRt2/assignment_loss‚0.7643660436304964 2loss/SRqqt1/assignment_loss‚0.5689259613923213 .loss/FRt2/detection_loss‚1.618825641996167 _runtime‚7870.947123341 1loss/SRqqt1/detection_loss‚0.8681465523571453 1loss/SRqqt2/detection_loss‚0.6064175014367585 .loss/FBt1/detection_loss‚0.871095043527553 /loss/FBt2/detection_loss‚0.4469447673901039 epoch‚11 trainer/global_step‚4749 0loss/FRt1/assignment_loss‚1.6457719270267415 2loss/SRqqt2/assignment_loss‚0.5688340084174998 2loss/FBt2/assignment_loss‚0.017726321569379356 /loss/FRt1/detection_loss‚0.5235349049933976 %loss/total_loss‚8.560571010052785 0loss/FBt1/assignment_loss‚0.0599823363152215 % _timestamp‚1.7842976867302146e+09 _step‚390Á#MW ë… 0loss/FRt2/assignment_loss‚0.7643660436304964 2loss/SRqqt1/assignment_loss‚0.5689259613923213 .loss/FRt2/detection_loss‚1.618825641996167 _runtime‚7870.947123341 1loss/SRqqt1/detection_loss‚0.8681465523571453 1loss/SRqqt2/detection_loss‚0.6064175014367585 .loss/FBt1/detection_loss‚0.871095043527553 /loss/FBt2/detection_loss‚0.4469447673901039 epoch‚11 trainer/global_step‚4749 0loss/FRt1/assignment_loss‚1.6457719270267415 2loss/SRqqt2/assignment_loss‚0.5688340084174998 2loss/FBt2/assignment_loss‚0.017726321569379356 /loss/FRt1/detection_loss‚0.5235349049933976 %loss/total_loss‚8.560571010052785 0loss/FBt1/assignment_loss‚0.0599823363152215 % _timestamp‚1.7842976867302146e+09 _step‚390†$ŽÖÌ”ì‚(:‰ ÚñèÒèÿÎø gpu.0.gpu‚90.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚247.797' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚49.5594 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0JiÿNí‚(:ù ÚñèÒöµ¤û$ memory_percent‚90.99893569946289 network.recv‚ 1.9969e+06 disk.drbd2165.out‚0 disk.rbd1.out‚ 19.0107421875 disk.dm-1.out‚ 355.48828125 disk.sda1.in‚0 disk.sda2.in‚0 disk.sda.out‚ 355.48828125 disk.rbd0.out‚0 cpu‚25.550041998581346 disk.rbd1.in‚ 0.01171875 disk.sda.in‚0 disk.rbd0.in‚0 disk.sda3.in‚0 disk.rbd3.out‚0# disk.nvme0c0n1.in‚ 15830.8984375' disk.nvme0c0n1.out‚68369.9130859375 disk.sda2.out‚0" disk.dm-0.out‚68369.9130859375% disk.nvme0n1.out‚68369.9130859375 disk.dm-1.in‚0' proc.memory.availableMB‚ 1474.734375* proc.memory.percent‚18.277668952941895 disk.dm-0.in‚ 15830.8984375 network.sent‚ 8.997485e+06 disk.drbd2165.in‚0) disk./.usagePercent‚36.84376743923323 disk.sda3.out‚ 355.48828125! disk.nvme0n1.in‚ 15830.8984375# proc.memory.rssMB‚ 2994.61328125 proc.cpu.threads‚17$ disk./.usageGB‚548.9987602233887 disk.sda1.out‚0 disk.rbd3.in‚0¡zŸî‚(:” éñèÒó€Ìø gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚38.0 gpu.0.powerWatts‚129.949' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚25.989800000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Èû¾ ï‚(:ÿ éñèÒ¤¦‰û$ disk.nvme0c0n1.in‚15831.31640625# proc.memory.rssMB‚ 2994.61328125 disk.rbd1.out‚ 19.0107421875$ disk./.usageGB‚549.0079956054688 disk.rbd3.in‚0) proc.memory.availableMB‚ 1474.78515625* proc.memory.percent‚18.277668952941895" disk.dm-0.out‚68382.7216796875 disk.sda1.in‚0" disk.nvme0n1.in‚15831.31640625% disk.nvme0n1.out‚68382.7216796875 cpu‚25.54969587771481 disk.dm-1.out‚ 355.88671875 disk.rbd0.out‚0' disk.nvme0c0n1.out‚68382.7216796875 proc.cpu.threads‚17 disk.sda3.in‚0$ memory_percent‚90.99862575531006 network.sent‚ 9.059986e+06 network.recv‚ 2.001694e+06 disk.sda.in‚0 disk.rbd0.in‚0 disk.dm-0.in‚15831.31640625 disk.sda1.out‚0 disk.rbd3.out‚0 disk.sda2.in‚0 disk.sda.out‚ 355.88671875 disk.rbd1.in‚ 0.01171875) disk./.usagePercent‚36.84438723347363 disk.sda2.out‚0 disk.dm-1.in‚0 disk.sda3.out‚ 355.88671875 disk.drbd2165.in‚0 disk.drbd2165.out‚0Çßy”ð‚(:‰ øñèÒ•»Éø gpu.0.gpu‚92.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚257.762' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚51.5524 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ð*Úîñ‚(:ñ øñèÒË­ìú disk.rbd0.out‚0# disk.dm-0.out‚68397.29150390625 cpu‚25.549414144196486 disk.rbd0.in‚0$ memory_percent‚90.99712371826172 disk.drbd2165.in‚0! disk.nvme0c0n1.in‚ 15831.53125 disk.nvme0n1.in‚ 15831.53125 disk.rbd1.in‚ 0.01171875 network.sent‚ 9.065037e+06 disk.dm-0.in‚ 15831.53125 disk.sda3.out‚ 357.140625 disk.drbd2165.out‚0 disk.rbd3.in‚0 disk.dm-1.out‚ 357.140625 disk.rbd1.out‚ 19.0732421875# pÔ«?roc.memory.rssMB‚ 2994.61328125) disk./.usagePercent‚36.84481809415375 disk.sda1.out‚0 disk.sda3.in‚0( disk.nvme0c0n1.out‚68397.29150390625& proc.memory.availableMB‚ 1475.03125 network.recv‚ 2.005276e+06$ disk./.usageGB‚549.0144157409668 disk.sda.out‚ 357.140625 disk.rbd3.out‚0& disk.nvme0n1.out‚68397.29150390625 disk.sda2.out‚0 disk.sda.in‚0 disk.sda1.in‚0 disk.sda2.in‚0* proc.memory.percent‚18.277668952941895 proc.cpu.threads‚17 disk.dm-1.in‚0þqEÞžò‚(:“ ‡òèÒ¤…àø gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚333.524' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚66.70479999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0CVæ@ ó‚(:þ ‡òèÒÀ™û disk.drbd2165.in‚0! disk.nvme0n1.in‚ 15903.2734375 disk.sda2.in‚0$ disk./.usageGB‚549.0204925537109 disk.sda.out‚ 360.25390625 disk.sda1.out‚0 network.sent‚ 9.070209e+06 disk.dm-1.in‚0 disk.sda3.out‚ 360.25390625( proc.memory.availableMB‚ 1475.0234375 proc.cpu.threads‚17 network.recv‚ 2.009113e+06 disk.sda.in‚0# disk.dm-0.out‚68426.21337890625 disk.sda3.in‚0 disk.rbd0.in‚0& disk.nvme0n1.out‚68426.21337890625 disk.sda2.out‚0 disk.rbd1.in‚ 0.01171875 disk.rbd1.out‚ 19.0732421875$ memory_percent‚90.99717140197754( disk./.usagePercent‚36.8452259141558 disk.drbd2165.out‚0 disk.rbd3.out‚0# disk.nvme0c0n1.in‚ 15903.2734375( disk.nvme0c0n1.out‚68426.21337890625 cpu‚25.549130894588686 disk.dm-1.out‚ 360.25390625 disk.rbd0.out‚0 disk.dm-0.in‚ 15903.2734375 disk.sda1.in‚0* proc.memory.percent‚18.277668952941895 disk.rbd3.in‚0# proc.memory.rssMB‚ 2994.61328125ˆy  ”ô‚(:‰ –òèÒÈë«÷ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚293.271' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚58.6542 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0äÈU†üõ‚(:ñ –òèÒáÊÁø proc.cpu.threads‚17$ disk.nvme0c0n1.in‚15903.69140625 disk.drbd2165.in‚0 disk.rbd3.in‚0$ memory_percent‚91.00017547607422 cpu‚25.548803745885504" disk.nvme0n1.in‚15903.69140625 disk.dm-1.in‚0 disk.rbd3.out‚0( disk.nvme0c0n1.out‚68465.65380859375 disk.sda.out‚362.875 disk.dm-0.in‚15903.69140625 disk.sda1.out‚0& proc.memory.availableMB‚ 1474.53125 disk.rbd0.out‚0 disk.sda2.out‚0 disk.rbd1.in‚ 0.01171875 disk.dm-1.out‚362.875 disk.sda3.in‚0# proc.memory.rssMB‚ 2994.61328125* proc.memory.percent‚18.277668952941895 network.recv‚ 2.012436e+06) disk./.usagePercent‚36.84665110809892 disk.sda2.in‚0# disk.dm-0.out‚68465.65380859375 disk.sda1.in‚0$ disk./.usageGB‚549.0417289733887& disk.nvme0n1.out‚68465.65380859375 disk.rbd1.out‚ 19.1513671875 disk.sda.in‚0 disk.rbd0.in‚0 disk.sda3.out‚362.875 disk.drbd2165.out‚0 network.sent‚ 9.075214e+06LÅ£2ö- _wandbruntime‚7948  _runtime‚7948ãÎæÇ÷Á & lr-AdamW/pg2‚0.0006093627683537337 trainer/global_step‚4799 % _timestamp‚1.7842977641594272e+09 _runtime‚7948.376539946 _step‚391 & lr-AdamW/pg1‚0.0006093627683537337Gª²ßÌøÆ trainer/global_step‚4799 % _timestamp‚1.7842977641594272e+09 _runtime‚7948.376539946 _step‚391 & lr-AdamW/pg1‚0.0006093627683537337 & lr-AdamW/pg2‚0.0006093627683537337‡áàH2ù- _wandbruntime‚7948  _runtime‚7948òÍÞ%þ'úøO TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1376598776389632 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4509893209088 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4503522684029 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5813166296576 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚41527287 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4509893209088 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚41517314 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1381110041161728 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚25921244 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚372656111030272 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12308697 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚41527287 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚32439296 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13612547 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1380507570019453 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 _runtime‚7948.377728319 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚25921480 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚231 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1381108669598720 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1381108669598720 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13612778 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚372655742939136 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 $ _timestamp‚1.784297764160431e+09 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4511238598656 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚378468909235712 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1376004021907456 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚40845237 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545òr·k(ûýO IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5813166296576 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚41527287 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4509893209088 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚41517314 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1381110041161728 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚25921244 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚372656111030272 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12308697 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚41527287 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚32439296 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13612547 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1380507570019453 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 _runtime‚7948.377728319 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚25921480 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚231 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1381108669598720 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1381108669598720 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13612778 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚372655742939136 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 $ _timestamp‚1.784297764160431e+09 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4511238598656 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚378468909235712 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1376004021907456 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚40845237 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1376598776389632 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4509893209088 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4503522684029 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640ˆ­rƒÕ2ü- _wandbruntime‚7949  _runtime‚7949ó¸I 6 ýƒN QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚25926548 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1376876803293184 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚82379229 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚378547608241664 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚40853443 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4510818244608 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚41525792 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚41525792 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1376876803293184 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚41535779 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1376876777119744 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚40853443 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚12311245 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1380785005235585 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12311240 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚372733597437952 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4504446150387 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚25926770 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13615308 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1376281929113600 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 398426112 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚40853437 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚82389222 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1376281903685632 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 % _timestamp‚1.7842977655459113e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4512163641344 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5814409229824 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peaTÖÓk‚ 295881216 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4512163641344 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚217 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚82379229 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4510818244608 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚30334976 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚40853437 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚222 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12311240 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚372733597437952 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4504446150387 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚25926770 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13615308 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1376281929113600 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 398426112 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚40853437 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚82389222 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1376281903685632 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 % _timestamp‚1.7842977655459113e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4512163641344 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5814409229824 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4512163641344 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚217 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚82379229 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4510818244608 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚30334976 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚40853437 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚222 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚25926548 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1376876803293184 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚82379229 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚378547608241664 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚40853443 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4510818244608 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚41525792‰ÙŸ2ÿ- _wandbruntime‚7949  _runtime‚7949ïmÙ €„ _runtime‚7949.849634033 1loss/FBt1/assignment_loss‚0.01385523829361793 1loss/SRqqt1/detection_loss‚0.8650700774234839 0loss/FRt2/assignment_loss‚0.7230928133013491 2loss/SRqqt2/assignment_loss‚0.4934078744747117 .loss/FRt2/detection_loss‚1.497631124898114 trainer/global_step‚4799 _step‚394 2loss/SRqqt1/assignment_loss‚0.5076702172769116 /loss/FRt1/detection_loss‚0.5206903732604922 1loss/SRqqt2/detection_loss‚0.6836269687030276 /loss/FBt1/detection_loss‚0.7740369561006841 0loss/FBt2/detection_loss‚0.47243936844304096 epoch‚11 % _timestamp‚1.7842977656325548e+09 0loss/FRt1/assignment_loss‚1.6400312241236072 4loss/FBt2/assignment_loss‚0.00021109667639865106 $loss/total_loss‚8.19176333297544)Ç£‰ 2loss/SRqqt1/assignment_loss‚0.5076702172769116 /loss/FRt1/detection_loss‚0.5206903732604922 1loss/SRqqt2/detection_loss‚0.6836269687030276 /loss/FBt1/detection_loss‚0.7740369561006841 0loss/FBt2/detection_loss‚0.47243936844304096 epoch‚11 % _timestamp‚1.7842977656325548e+09 0loss/FRt1/assignment_loss‚1.6400312241236072 4loss/FBt2/assignment_loss‚0.00021109667639865106 $loss/total_loss‚8.19176333297544 _runtime‚7949.849634033 1loss/FBt1/assignment_loss‚0.01385523829361793 1loss/SRqqt1/detection_loss‚0.8650700774234839 0loss/FRt2/assignment_loss‚0.7230928133013491 2loss/SRqqt2/assignment_loss‚0.4934078744747117 .loss/FRt2/detection_loss‚1.497631124898114 trainer/global_step‚4799 _step‚394Š=fŸ‚‚(:” ¥òèÒá±ðö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚41.0 gpu.0.powerWatts‚271.794' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚54.358799999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¨ºéûƒ‚(:ð ¥òèÒÂÃåû disk.sda1.out‚0 disk.drbd2165.in‚0 disk.nvme0n1.in‚ 15903.90625 disk.dm-0.in‚ 15903.90625 disk.sda1.in‚0$ memory_percent‚90.99948406219482$ disk.nvme0n1.out‚68493.373046875 disk.sda.in‚0 disk.sda3.out‚ 364.9453125 disk.rbd3.out‚0) proc.memory.availableMB‚ 1474.64453125* proc.memory.percent‚18.277668952941895 network.sent‚ 9.080814e+06 disk.dm-1.in‚0 disk.rbd0.in‚0! disk.dm-0.out‚68493.373046875 disk.sda3.in‚0 disk.drbd2165.out‚0# disk./.usageGB‚549.075984954834& disk.nvme0c0n1.out‚68493.373046875 network.recv‚ 2.015901e+06 disk.rbd1.in‚ 0.01171875 disk.dm-1.out‚ 364.9453125 disk.rbd0.out‚0# proc.memory.rssMB‚ 2994.61328125! disk.nvme0c0n1.in‚ 15903.90625 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd1.out‚ 19.1513671875 disk.rbd3.in‚0) disk./.usagePercent‚36.84895005575639 disk.sda.out‚ 364.9453125 cpu‚25.548533151092478 proc.cpu.threads‚174èüÚ”„‚(:‰ ´òèÒ’åö gpu.0.gpu‚87.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚37.0 gpu.0.powerWatts‚109.347' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.8694 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 yV©ü…‚(:ñ ´òèÒéè”ú) disk./.usagePercent‚36.84694449273315 disk.dm-0.in‚ 15904.109375 disk.sda2.in‚0 cpu‚25.5531473629632 disk.sda.in‚0 disk.sda.out‚ 367.2734375 disk.sda1.in‚0 disk.sda3.out‚ 367.2734375 proc.cpu.threads‚17 disk.sda1.out‚0" disk.nvme0c0n1.in‚ 15904.109375( disk.nvme0c0n1.out‚68541.10986328125 disk.nvme0n1.in‚ 15904.109375) proc.memory.percent‚17.96414852142334 network.sent‚ 9.143238e+06 disk.dm-1.out‚ 367.2734375 disk.rbd0.in‚0 disk.sda3.in‚0 disk.rbd3.out‚0& disk.nvme0n1.out‚68541.10986328125 network.recv‚ 2.019558e+06$ disk./.usageGB‚549.0461006164551 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.sda2.out‚0 disk.rbd1.out‚ 19.2138671875# proc.memory.availableMB‚1526.25 disk.dm-1.in‚0 disk.rbd0.out‚0 disk.rbd3.in‚0$ memory_percent‚90.68450927734375# proc.memory.rssMB‚ 2943.24609375# disk.dm-0.out‚68541.10986328125 disk.rbd1.in‚ 0.01171875Ñ×¹62†- _wandbruntime‚7976  _runtime‚7976aë{Î=‡È{ & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4939424991607666 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9840365052223206 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7537762522697449 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.15416666865348816 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 trainer/global_step‚4799 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.3226770758628845 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.22927626967430115 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9966524243354797 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9913028478622437 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8769274353981018 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 ( Purity/2FRt1SRqqt1FBt/FRt_purity‚0.5 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8711261749267578 8"Purity/*FRt0SRqqt2FBt/event_purity‚0.940463662147522 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6795551776885986 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9814814329147339 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.22262728214263916 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8708849549293518 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7333904504776001 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 *jet/accuracy_2_of_4‚0.7673223614692688 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 6 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.998015820980072 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9351851344108582 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9920955896377563 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3385416865348816 jet/accuracy_5_of_5‚0 particle/accuracy_3_of_5‚1 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.04539789259433746 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 *jet/accuracy_1_of_2‚0.7847416400909424 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.625 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5826589465141296 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.20658427476882935 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 particle/accuracy_1_of_4‚1 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.19188006222248077 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.17598465085029602 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 +jet/accuracy_3_of_4‚0.10396572202444077 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.1538461595773697 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9823988080024719 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.21637512743473053 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7136723399162292 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6643086671829224 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8769274353981018 particle/accuracy_1_of_3‚1 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7192391753196716 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.48679864406585693 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.02443106099963188 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.05717236548662186 *jet/accuracy_1_of_3‚0.9240972399711609 6validation_average_jet_accuracy‚0.3631298542022705 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.1385987550020218 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.4305555522441864 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.1923319548368454 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.03329009935259819 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5857857465744019 particle/accuracy_2_of_3‚1 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9914528727531433 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.8001700639724731 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2289508879184723 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9976071119308472 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7182549238204956 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.6671303510665894 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 *jet/accuracy_1_of_1‚0.3199763298034668 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5938868522644043 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7561883926391602 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9944234490394592 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.21524325013160706 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6592324376106262 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6034421324729919 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6219885349273682 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.7988637089729309 particle/accuracy_5_of_5‚0 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.15026766061782837 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.20540541410446167 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7138828039169312 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.21637512743473053 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8333333134651184 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6195424199104309 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 (loss/val_total_loss‚8.11074777372906 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.6626371145248413 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4453125 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 /particle/accuracy_3_of_3‚0.9995090961456299 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6227791905403137 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7184542417526245 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7111990451812744 6 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.987064003944397 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.04690416529774666 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 6 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.205563023686409 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9976890087127686 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8782381415367126 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5758333206176758 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 , Purity/0FRt2SRqqt*FBt/FBt_purity‚0.96875 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.1879427134990692 +jet/accuracy_2_of_2‚0.14240285754203796 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7857142686843872 _step‚395 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 particle/accuracy_1_of_1‚1 &particle/f_score‚0.645457923412323 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.1538461595773697 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9939138293266296 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6644032597541809 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.15465877950191498 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 +particle/specificity‚0.9113637208938599 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.12962964177131653 , Purity/0FRt2SRqqt1FBt/FBt_purity‚0.96875 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.05717236548662186 _runtime‚7976.096446307 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7142857313156128 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2293408066034317 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.20911286771297455 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3666580319404602 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 particle/accuracy_2_of_5‚1 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.12962964177131653 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6636092066764832 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 epoch‚11 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9928897619247437 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.24926428496837616 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 (particle/accuracy‚0.8511440753936768 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.30982232093811035 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.27128541469573975 8"Purity/*FRt*SRqqt1FBt/event_purity‚0.758416473865509 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.16857023537158966 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9987789392471313 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.06837237626314163 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6457955241203308 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9110284447669983 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6284690499305725 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 8"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.695704460144043 particle/accuracy_2_of_4‚1 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9230769276618958 % _timestamp‚1.7842977918786507e+09 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6632993221282959 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6174706220626831 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9966365098953247 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 particle/accuracy_3_of_4‚1 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.06405845284461975 /particle/accuracy_4_of_4‚0.9287341237068176 ) Purity/2FRt*SRqqt2FBt/FRt_purity‚0.75 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6536458134651184 particle/accuracy_1_of_5‚1 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9930979609489441 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9795289039611816 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.2007020115852356 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.7988637089729309 )jet/accuracy_1_of_4‚0.965107798576355 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.37112441658973694 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.33392712473869324 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6620410680770874 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.6999548673629761 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6227791905403137 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03294536843895912 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 +jet/accuracy_3_of_3‚0.07728885859251022 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6061819791793823 8"Purity/1FRt*SRqqt1FBt/event_purity‚0.518464982509613 particle/accuracy_1_of_2‚1 jet/accuracy_2_of_5‚1 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4715427756309509 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 jet/accuracy_1_of_5‚1 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.16412082314491272 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7597108483314514 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.07381193339824677 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 ,jet/accuracy_4_of_4‚0.013899499550461769 particle/accuracy_2_of_2‚1 jet/accuracy_3_of_5‚1 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6635184288024902 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.5987978577613831 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.34329670667648315 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8333333134651184 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8808767199516296 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6189305186271667 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9680728912353516 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.22928987443447113 particle/accuracy_4_of_5‚0 9"Purity/*FRt2SRqqt0FBt/event_purity‚0.1530379056930542 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6536256670951843 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5890583992004395 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 *jet/accuracy_2_of_3‚0.5251742601394653 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9053214192390442 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9966365098953247 jet/accuracy_4_of_5‚0 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.07387133687734604 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8752266764640808 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.28050026297569275 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.15195053815841675 7 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.0684213638305664 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9814814329147339 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 +particle/sensitivity‚0.6309585571289062Ö¦¬ÆVˆÍ{ 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9914528727531433 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.8001700639724731 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2289508879184723 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9976071119308472 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7182549238204956 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.6671303510665894 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 *jet/accuracy_1_of_1‚0.3199763298034668 ("Puri<¹};ty/2FRt1SRqqt*FBt/event_purity‚0 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5938868522644043 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7561883926391602 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9944234490394592 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.21524325013160706 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6592324376106262 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6034421324729919 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6219885349273682 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.7988637089729309 particle/accuracy_5_of_5‚0 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.15026766061782837 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.20540541410446167 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7138828039169312 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.21637512743473053 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8333333134651184 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6195424199104309 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 (loss/val_total_loss‚8.11074777372906 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.6626371145248413 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4453125 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 /particle/accuracy_3_of_3‚0.9995090961456299 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6227791905403137 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7184542417526245 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7111990451812744 6 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.987064003944397 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.04690416529774666 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 6 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.205563023686409 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9976890087127686 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8782381415367126 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5758333206176758 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 , Purity/0FRt2SRqqt*FBt/FBt_purity‚0.96875 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.1879427134990692 +jet/accuracy_2_of_2‚0.14240285754203796 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7857142686843872 _step‚395 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 particle/accuracy_1_of_1‚1 &particle/f_score‚0.645457923412323 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.1538461595773697 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9939138293266296 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6644032597541809 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.15465877950191498 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 +particle/specificity‚0.9113637208938599 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.12962964177131653 , Purity/0FRt2SRqqt1FBt/FBt_purity‚0.96875 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.05717236548662186 _runtime‚7976.096446307 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7142857313156128 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2293408066034317 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.20911286771297455 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3666580319404602 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 particle/accuracy_2_of_5‚1 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.12962964177131653 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6636092066764832 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 epoch‚11 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9928897619247437 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.24926428496837616 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 (particle/accuracy‚0.8511440753936768 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.30982232093811035 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.27128541469573975 8"Purity/*FRt*SRqqt1FBt/event_purity‚0.758416473865509 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.16857023537158966 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9987789392471313 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.06837237626314163 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6457955241203308 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9110284447669983 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6284690499305725 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 8"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.695704460144043 particle/accuracy_2_of_4‚1 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9230769276618958 % _timestamp‚1.7842977918786507e+09 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6632993221282959 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6174706220626831 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9966365098953247 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 particle/accuracy_3_of_4‚1 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.06405845284461975 /particle/accuracy_4_of_4‚0.9287341237068176 ) Purity/2FRt*SRqqt2FBt/FRt_purity‚0.75 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6536458134651184 particle/accuracy_1_of_5‚1 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9930979609489441 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9795289039611816 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.2007020115852356 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.7988637089729309 )jet/accuracy_1_of_4‚0.965107798576355 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.37112441658973694 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.33392712473869324 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6620410680770874 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.6999548673629761 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6227791905403137 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03294536843895912 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 +jet/accuracy_3_of_3‚0.07728885859251022 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6061819791793823 8"Purity/1FRt*SRqqt1FBt/event_purity‚0.518464982509613 particle/accuracy_1_of_2‚1 jet/accuracy_2_of_5‚1 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4715427756309509 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 jet/accuracy_1_of_5‚1 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.16412082314491272 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7597108483314514 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.07381193339824677 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 ,jet/accuracy_4_of_4‚0.013899499550461769 particle/accuracy_2_of_2‚1 jet/accuracy_3_of_5‚1 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6635184288024902 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.5987978577613831 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.34329670667648315 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8333333134651184 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8808767199516296 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6189305186271667 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9680728912353516 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.22928987443447113 particle/accuracy_4_of_5‚0 9"Purity/*FRt2SRqqt0FBt/event_purity‚0.1530379056930542 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6536256670951843 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5890583992004395 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 *jet/accuracy_2_of_3‚0.5251742601394653 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9053214192390442 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9966365098953247 jet/accuracy_4_of_5‚0 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.07387133687734604 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8752266764640808 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.28050026297569275 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.15195053815841675 7 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.0684213638305664 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9814814329147339 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 +particle/sensitivity‚0.6309585571289062 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4939424991607666 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9840365052223206 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7537762522697449 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.15416666865348816 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 trainer/global_step‚4799 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.3226770758628845 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.22927626967430115 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9966524243354797 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9913028478622437 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8769274353981018 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 ( Purity/2FRt1SRqqt1FBt/FRt_purity‚0.5 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8711261749267578 8"Purity/*FRt0SRqqt2FBt/event_purity‚0.940463662147522 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6795551776885986 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9814814329147339 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.22262728214263916 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8708849549293518 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7333904504776001 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 *jet/accuracy_2_of_4‚0.7673223614692688 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 6 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.998015820980072 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9351851344108582 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9920955896377563 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3385416865348816 jet/accuracy_5_of_5‚0 particle/accuracy_3_of_5‚1 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.04539789259433746 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 *jet/accuracy_1_of_2‚0.7847416400909424 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.625 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5826589465141296 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.20658427476882935 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 particle/accuracy_1_of_4‚1 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.19188006222248077 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.17598465085029602 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 +jet/accuracy_3_of_4‚0.10396572202444077 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.1538461595773697 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9823988080024719 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.21637512743473053 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7136723399162292 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6643086671829224 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8769274353981018 particle/accuracy_1_of_3‚1 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7192391753196716 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.48679864406585693 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.02443106099963188 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.05717236548662186 *jet/accuracy_1_of_3‚0.9240972399711609 6validation_average_jet_accuracy‚0.3631298542022705 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.1385987550020218 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.4305555522441864 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.1923319548368454 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.03329009935259819 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5857857465744019 particle/accuracy_2_of_3‚1‹æ–Ò ‰‚(:’ ÃòèÒûÖÖø gpu.0.gpu‚18.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚31.0 gpu.0.powerWatts‚90.439' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚18.087799999999998 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0vDÃRŠ‚(:ø ÃòèÒÔÕñù disk.sda1.in‚0 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.rbd1.in‚ 0.01171875 network.sent‚ 9.148079e+06" disk.dm-0.out‚68559.4287109375$ disk.nvme0c0n1.in‚16029.49609375% disk.nvme0n1.out‚68559.4287109375* proc.memory.percent‚20.158839225769043$ disk./.usageGB‚549.0483474731445 disk.sda.in‚0 disk.dm-1.out‚ 370.93359375 disk.rbd0.out‚0 disk.sda2.out‚0 disk.rbd1.out‚ 19.2138671875 disk.sda.out‚ 370.93359375 disk.sda1.out‚0 disk.rbd3.out‚0 disk.sda2.in‚0 disk.dm-0.in‚16029.49609375 disk.sda3.out‚ 370.93359375' disk.nvme0c0n1.out‚68559.4287109375" disk.nvme0n1.in‚16029.49609375# memory_percent‚84.3902587890625) disk./.usagePercent‚36.84709528117082 disk.rbd0.in‚0 disk.sda3.in‚0" proc.memory.availableMB‚2557.5# proc.memory.rssMB‚ 3302.82421875 disk.dm-1.in‚0 disk.rbd3.in‚0 cpu‚25.556423236869218 proc.cpu.threads‚17 network.recv‚ 2.023195e+06§¯m“‹‚(:ˆ ÒòèÒßÑãö gpu.0.gpu‚26.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚334.052' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚66.8104 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0nŠ“†ýŒ‚(:ò ÒòèÒòĦù$ disk./.usageGB‚549.1125144958496 disk.rbd3.in‚0 disk.rbd3.out‚0! disk.nvme0n1.in‚ 16029.7109375* proc.memory.percent‚17.965221405029297 proc.cpu.threads‚17 disk.drbd2165.in‚0% disk.nvme0c0n1.out‚70851.48828125 disk.rbd1.out‚ 19.2919921875 network.recv‚ 2.027735e+06 disk.dm-1.out‚ 373.12109375 disk.rbd0.in‚0 disk.sda2.out‚0 disk.rbd0.out‚0 disk.dm-0.in‚ 16029.7109375 disk.sda3.in‚0# disk.nvme0c0n1.in‚ 16029.7109375) disk./.usagePercent‚36.85140158390428 disk.sda.out‚ 373.12109375 disk.dm-1.in‚0# disk.nvme0n1.out‚70851.48828125$ memory_percent‚82.20586776733398 cpu‚25.55607210197908 disk.sda.in‚0 disk.dm-0.out‚70851.48828125 disk.rbd1.in‚ 0.01171875' proc.memory.availableMB‚ 2915.390625! proc.memory.rssMB‚ 2943.421875 disk.sda1.in‚0 network.sent‚ 9.205382e+06 disk.sda3.out‚ 373.12109375 disk.drbd2165.out‚0 disk.sda1.out‚0 disk.sda2.in‚0çBJr”‚(:‰ áòèÒ•³áö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚275.808' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.1616 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÛIÐŽ‚(:ý áòèÒ»Åíø disk.sda3.in‚0 disk.sda3.out‚ 375.17578125 disk.drbd2165.in‚0 disk.sda1.out‚0 disk.drbd2165.out‚0 disk.rbd3.out‚0 disk.dm-1.out‚ 375.17578125 disk.rbd0.in‚0 disk.dm-0.in‚ 16029.9140625) proc.memory.availableMB‚ 2915.47265625 cpu‚25.55572141265748 network.recv‚ 2.030992e+06) disk./.usagePercent‚36.85154597215359 disk.rbd0.out‚0 disk.rbd3.in‚0 disk.rbd1.in‚ 0.01171875# disk.nvme0c0n1.in‚ 16029.9140625! disk.nvme0n1.in‚ 16029.9140625 disk.sda2.in‚0* proc.memory.percent‚17.965221405029297 proc.cpu.threads‚17 network.sent‚ 9.210928e+06 disk.dm-1.in‚0 disk.sda1.in‚0( disk.nvme0c0n1.out‚70871.22314453125 disk.sda2.out‚0 disk.sda.in‚0$ memory_percent‚82.20536708831787$ disk./.usageGB‚549.1146659851074# disk.dm-0.out‚70871.22314453125& disk.nvme0n1.out‚70871.22314453125 disk.rbd1.out‚ 19.2919921875 disk.sda.out‚ 375.17578125! proc.memory.rssMB‚ 2943.421875¢Œ“‚(:ˆ ðòèÒËðþö gpu.0.gpu‚79.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚330.145' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚66.029 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0f½ÙS‚(:ý ðòèÒ¿‰ù disk.dm-1.in‚0 disk.sda3.out‚ 376.68359375$ disk.nvme0c0n1.in‚16030.33203125" disk.nvme0n1.in‚16030.33203125! proc.memory.rssMB‚ 2943.421875" disk.dm-0.out‚70887.2490234375% disk.nvme0n1.out‚70887.2490234375 disk.sda2.out‚0* proc.memory.percent‚17.965221405029297$ disk./.usageGB‚549.1240653991699 disk.dm-0.in‚16030.33203125 disk.rbd3.in‚0 disk.sda2.in‚0 disk.rbd1.in‚ 0.01171875 cpu‚25.555428705957414 proc.cpu.threads‚17 network.sent‚ 9.215972e+06 disk.rbd3.out‚0' disk.nvme0c0n1.out‚70887.2490234375 network.recv‚ 2.03422e+06) disk./.usagePercent‚36.85217677471796 disk.dm-1.out‚ 376.68359375 disk.sda3.in‚0) proc.memory.availableMB‚ 2915.46484375 disk.sda.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.drbd2165.out‚0$ memory_percent‚82.20541477203369 disk.sda.out‚ 376.68359375 disk.rbd1.out‚ 19.3544921875 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd2165.in‚0Ê«DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1398888846084096 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚12444770 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚41387543 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚42002053 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1393724045176832 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚83399569 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1394326820503552 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1398888846084096 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚41387537 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚376578543661056 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚376578175569920 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1398890217647104 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 _step‚397 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚26228825 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5880738921472 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚42012026 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1398279634105411 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚83389590 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _runtime‚8058.965142574 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1394326820503552 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1398278264077121 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1398890217647104 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚42012026 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4563397143552 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4562051753984 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4563397143552 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚42002053 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚232 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13784060 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚382459282582528 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1398888846084096 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚12444770 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚41387543 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚42002053 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1393724045176832 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚83399569 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1394326820503552 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1398888846084096 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚41387537 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚376578543661056 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚376578175569920 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1398890217647104 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 _step‚397 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚26228825 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5880738921472 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚42012026 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1398279634105411 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚83389590 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _runtime‚8058.965142574 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1394326820503552 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1398278264077121 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1398890217647104 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚42012026 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4563397143552 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4562051753984 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4563397143552 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚42002053 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚232 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13784060 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚382459282582528 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚83416267 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 trainer/global_step‚4849 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚83416267 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚225 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13786797 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 400523264 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1394604821233664 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚376655661977600 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4555167794503 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4564322186240 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚41395749 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1394001926955008 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1394604821233664 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4562976789504 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚26234105 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13787022 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚83406274 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1394604795060224 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12447308 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚41395743 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1394604795060224 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4562976789504 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7842978761322584e+09 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1399167771849728 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚26234335 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 _runtime‚8060.349592785 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚382537587709952 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5881925732352 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1398557069321543 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 _step‚398 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚42010531 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4564322186240 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚230 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119bÄ'ˆN VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚12447313 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚376656030068736 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚83416267 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 trainer/global_step‚4849 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚83416267 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚225 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13786797 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 400523264 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1394604821233664 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚376655661977600 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4555167794503 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4564322186240 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚41395749 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1394001926955008 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1394604821233664 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4562976789504 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚26234105 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13787022 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚83406274 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1394604795060224 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12447308 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚41395743 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1394604795060224 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4562976789504 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7842978761322584e+09 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1399167771849728 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚26234335 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 _runtime‚8060.349592785 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚382537587709952 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5881925732352 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1398557069321543 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 _step‚398 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚42010531 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4564322186240 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚230 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚382537988233216 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0Ž^¹k2ž- _wandbruntime‚8060  _runtime‚8060#ˆi¾Ÿƒ /loss/FRt2/detection_loss‚1.6002468971481312 % _timestamp‚1.7842978762194695e+09 0loss/FRt2/assignment_loss‚0.7688244071978967 1loss/FBt1/assignment_loss‚0.03593196123119553 /loss/FRt1/detection_loss‚0.5624501270681519 1loss/SRqqt1/detection_loss‚0.8369406271058677 /loss/FBt2/detection_loss‚0.4906071200966835 %loss/total_loss‚8.083019049617963 _runtime‚8060.436390998 _step‚399 1loss/SRqqt2/det†4ãKection_loss‚0.6450131131574175 trainer/global_step‚4849 epoch‚12 2loss/FBt2/assignment_loss‚0.023044181468279856 /loss/FBt1/detection_loss‚0.5404802465269788 0loss/FRt1/assignment_loss‚1.6075124865996826 2loss/SRqqt1/assignment_loss‚0.5414213516047909 2loss/SRqqt2/assignment_loss‚0.4305465304128869„ßKÒ ˆ 0loss/FRt2/assignment_loss‚0.7688244071978967 1loss/FBt1/assignment_loss‚0.03593196123119553 /loss/FRt1/detection_loss‚0.5624501270681519 1loss/SRqqt1/detection_loss‚0.8369406271058677 /loss/FBt2/detection_loss‚0.4906071200966835 %loss/total_loss‚8.083019049617963 _runtime‚8060.436390998 _step‚399 1loss/SRqqt2/detection_loss‚0.6450131131574175 trainer/global_step‚4849 epoch‚12 2loss/FBt2/assignment_loss‚0.023044181468279856 /loss/FBt1/detection_loss‚0.5404802465269788 0loss/FRt1/assignment_loss‚1.6075124865996826 2loss/SRqqt1/assignment_loss‚0.5414213516047909 2loss/SRqqt2/assignment_loss‚0.4305465304128869 /loss/FRt2/detection_loss‚1.6002468971481312 % _timestamp‚1.7842978762194695e+09^`µÖ¡‚(:ú óèÒ±ñËù disk.sda.in‚0" disk.dm-0.out‚70928.4365234375 disk.sda3.out‚ 380.7421875" disk.nvme0n1.in‚16146.70703125 disk.sda2.in‚0 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.drbd2165.out‚0% disk.nvme0n1.out‚70928.4365234375) proc.memory.availableMB‚ 2914.96484375 disk.sda1.in‚0$ disk./.usageGB‚549.1391792297363 disk.sda.out‚ 380.7421875' disk.nvme0c0n1.out‚70928.4365234375 disk.rbd1.in‚ 0.01171875! proc.memory.rssMB‚ 2943.421875 disk.rbd0.out‚0 disk.sda3.in‚0$ disk.nvme0c0n1.in‚16146.70703125 cpu‚25.55440780207883) disk./.usagePercent‚36.85319107656862 disk.drbd2165.in‚0 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.rbd1.out‚ 19.4326171875 network.recv‚ 2.044719e+06 disk.dm-1.in‚0 disk.dm-0.in‚16146.70703125$ memory_percent‚82.20846652984619* proc.memory.percent‚17.965221405029297 network.sent‚ 9.231202e+06 disk.dm-1.out‚ 380.7421875 disk.sda1.out‚0 disk.sda2.out‚0ñm‰i¢‚(:’ óèÒ¶øª€ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚283.03' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚56.60599999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0º7ÜJ‘£‚(:† ¬óèÒºÌßö gpu.0.gpu‚98.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚195.55' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚39.11 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0I ŠO¤‚(:ö ¬óèÒóÓ¡ù disk.rbd3.in‚0 disk.rbd3.out‚0 disk.rbd0.out‚0$ memory_percent‚82.20510482788086 network.sent‚ 9.293836e+06 disk.sda3.in‚0( disk.nvme0c0n1.out‚73191.78857421875 disk.dm-1.in‚0 disk.dm-1.out‚ 382.81640625 disk.rbd0.in‚0( disk./.usagePercent‚37.0006347758013& disk.nvme0n1.out‚73191.78857421875 disk.sda1.in‚0 disk.sda3.out‚ 382.81640625) proc.memory.percent‚17.96560287475586$ disk./.usageGB‚551.3361968994141" disk.nvme0c0n1.in‚ 16146.921875 disk.sda2.out‚0 disk.sda.out‚ 382.81640625! proc.memory.rssMB‚ 2943.484375 cpu‚25.55418595930968 network.recv‚ 2.049269e+06 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.nvme0n1.in‚ 16146.921875 disk.dm-0.in‚ 16146.921875 proc.cpu.threads‚17 disk.drbd2165.out‚0 disk.sda2.in‚0 disk.rbd1.in‚ 0.01171875 disk.rbd1.out‚ 19.4951171875 disk.sda.in‚0' proc.memory.availableMB‚ 2915.515625# disk.dm-0.out‚73191.78857421875°ìŸ¥‚(:” »óèÒŒ°ñö gpu.0.gpu‚87.0 gpu.0.memory‚25, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚290.369' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚58.073800000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ãçÉÎñ¦‚(:æ »óèÒÃÊ¡ù) proc.memory.percent‚17.96560287475586 disk.dm-0.out‚ 73206.390625! disk.nvme0n1.out‚ 73206.390625! disk.nvme0c0n1.in‚ 16239.21875 disk.rbd1.out‚ 19.4951171875 proc.cpu.threads‚17) disk./.usagePercent‚37.00064168800473 disk.rbd3.in‚0$ memory_percent‚82.20510482788086 disk.sda2.out‚0# disk.nvme0c0n1.out‚ 73206.390625 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.sda3.out‚ 384.84765625 cpu‚25.553901227882363 disk.sda1.in‚0 disk.sda2.in‚0 disk.drbd2165.out‚0 disk.nvme0n1.in‚ 16239.21875 disk.sda.out‚ 384.84765625 disk.rbd0.out‚0 network.sent‚ 9.299383e+06 disk.dm-1.out‚ 384.84765625 network.recv‚ 2.052735e+06$ disk./.usageGB‚551.3362998962402 disk.drbd2165.in‚0 disk.rbd3.out‚0 disk.sda.in‚0 disk.dm-0.in‚ 16239.21875 disk.sda1.out‚0 disk.sda3.in‚0 disk.rbd1.in‚ 0.01171875' proc.memory.availableMB‚ 2915.515625! proc.memory.rssMB‚ 2943.484375 x‰”§‚(:‰ ÊóèÒÞ…èö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚278.522' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.7044 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0(Ö© ¨‚(:ÿ ÊóèÒ“›‹ù network.sent‚ 9.304442e+06 disk.sda2.out‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 385.89453125 disk.rbd0.in‚0 disk.dm-0.in‚16239.63671875 disk.rbd3.in‚0$ disk.nvme0c0n1.in‚16239.63671875& disk.nvme0n1.out‚73219.84423828125$ memory_percent‚82.20198154449463) disk./.usagePercent‚37.00101929911773 disk.sda1.in‚0 disk.sda3.out‚ 385.89453125( disk.nvme0c0n1.out‚73219.84423828125 disk.rbd1.out‚ 19.6357421875) proc.memory.availableMB‚ 2916.02734375! proc.memory.rssMB‚ 2943.484375# disk./.usageGB‚551.341926574707 disk.sda.out‚ 385.89453125 network.recv‚ 2.056104e+06 disk.sda.in‚0 disk.rbd0.out‚0# disk.dm-0.out‚73219.84423828125 disk.sda3.in‚0 disk.drbd2165.out‚0 disk.rbd3.out‚0) proc.memory.percent‚17.96560287475586 disk.sda1.out‚0" disk.nvme0n1.in‚16239.63671875 proc.cpu.threads‚17 disk.drbd2165.in‚0 disk.sda2.in‚0 disk.rbd1.in‚ 0.01171875 cpu‚25.553587942362302¥=N]ž©‚(:“ ÙóèÒõ¼äö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚329.225' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚65.84500000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Äu ÿøª‚(:í ÙóèҊЃù disk.sda.out‚ 386.3359375 cpu‚25.553367035625513) disk./.usagePercent‚37.00133214032457 disk.dm-0.in‚ 16239.8515625 disk.drbd2165.in‚0 disk.dm-1.in‚0 proc.cpu.threads‚17 network.sent‚ 9.309436e+06# disk.nvme0c0n1.in‚ 16239.8515625 disk.sda2.in‚0 disk.rbd1.in‚ 0.01171875 disk.rbd1.out‚ 19.6396484375 disk.dm-1.out‚ 386.3359375 disk.rbd0.out‚0 disk.dm-0.out‚ 75478.4609375 disk.drbd2165.out‚0$ disk.nvme0c0n1.out‚ 75478.4609375 disk.sda3.out‚ 386.3359375 disk.rbd3.out‚0" disk.nvme0n1.out‚ 75478.4609375 disk.sda2.out‚0$ memory_percent‚82.20372200012207! proc.memory.rssMB‚ 2943.484375 network.recv‚ 2.059482e+06 disk.sda3.in‚0( proc.memory.availableMB‚ 2915.7421875 disk.sda1.in‚0 disk.rbd0.in‚0 disk.rbd3.in‚0$ disk./.usageGB‚551.3465881347656! disk.nvme0n1.in‚ 16239.8515625 disk.sda.in‚0) proc.memory.percent‚17.96560287475586 disk.sda1.out‚0}¡2«- _wandbruntime‚8139  _runtime‚81391P šÇ¬Á & lr-AdamW/pg1‚0.0006057355543132256 & lr-AdamW/pg2‚0.0006057355543132256 trainer/global_step‚4899 % _timestamp‚1.7842979555855463e+09 _runtime‚8139.802881085 _step‚400'g߯Ì­Æ _runtime‚8139.802881085 _step‚400 & lr-AdamW/pg1‚0.0006057355543132256 & lr-AdamW/pg2‚0.0006057355543132256 trainer/global_step‚4899 % _timestamp‚1.7842979555855463e+09Ö~2®- _wandbruntime‚8139  _runtime‚8139E¥öîÿ'¯ùO ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1412219960127319 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚84225642 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1408226830835712 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚42427805 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚42437778 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1407618108657664 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚41797843 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚26492820 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4608367684608 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4600481441365 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚42427805 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4609713074176 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚84235621 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13920900 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1412835198520320 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1408226857009152 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚386393738005504 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1412836570083328 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13920654 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1412835198520320 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5941276588032 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1408226830835712 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12571915 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5941242051584 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1412218590099029 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚26492569 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚42437778 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097m£Z#(°þO MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚41797843 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚26492820 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4608367684608 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4600481441365 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚42427805 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4609713074176 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚84235621 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13920900 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1412835198520320 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1408226857009152 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚386393738005504 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1412836570083328 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13920654 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1412835198520320 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5941276588032 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1408226830835712 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12571915 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5941242051584 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1412218590099029 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚26492569 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚42437778 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1412219960127319 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚84225642 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1408226830835712 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚42427805 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚42437778 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1407618108657664‘š*Bè2±- _wandbruntime‚8141  _runtime‚8141Ãb}÷ ²ƒN EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 T?DeviceStatX½=bsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1413114124285952 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1407895990435840 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚4899 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1408504857739264 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4610638116864 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚41806043 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4601404907611 _runtime‚8141.213235768 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚84242326 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1408504831565824 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚26497547 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 % _timestamp‚1.7842979569956853e+09 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚42446270 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚386472429761536 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚42446270 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚42436283 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12574458 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1412497395343451 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚84252319 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1408504831565824 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚41806049 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚26497788 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1407896015863808 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚12574463 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚236 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4602749508045 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1413115495856128 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1413115495856128 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚42436283 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚380530350452736 _step‚402 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚41806049 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚84252319 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚386472832381952 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1412498765371853°ÙA'³ˆN M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚84242326 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1408504831565824 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚26497547 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 % _timestamp‚1.7842979569956853e+09 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚42446270 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚386472429761536 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚42446270 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚42436283 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12574458 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1412497395343451 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚84252319 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1408504831565824 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚41806049 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚26497788 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1407896015863808 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚12574463 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚236 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4602749508045 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1413115495856128 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1413115495856128 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚42436283 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚380530350452736 _step‚402 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚41806049 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚84252319 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚386472832381952 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1412498765371853 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1413114124285952 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1407895990435840 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚4899 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1408504857739264 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4610638116864 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚41806043 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4601404907611 _runtime‚8141.213235768 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434’³é72´- _wandbruntime‚8141  _runtime‚81412<n µ† epoch‚12 0loss/FRt2/assignment_loss‚0.7003684095015713 3loss/FBt1/assignment_loss‚0.0016002317845464611 /loss/FRt2/detection_loss‚1.4883377206096353 3loss/SRqqt2/assignment_loss‚0.48575839111125885 % _timestamp‚1.7842979570740993e+09 %loss/total_loss‚8.224866724596396 0loss/FRt1/assignment_loss‚1.6571384704694516 2loss/FBt2/assignment_loss‚0.003364113514671018 1loss/SRqqt1/detection_loss‚0.8690206558572559 1loss/SRqqt2/detection_loss‚0.6003005250238939 /loss/FBt2/detection_loss‚0.3998181063167334 trainer/global_step‚4899 _step‚403 2loss/SRqqt1/assignment_loss‚0.4924985435572811 /loss/FRt1/detection_loss‚0.4809947879049646 /loss/FBt1/detection_loss‚1.0456667689451327 _runtime‚8141.291328528¦@*³¶‹ _step‚403 2loss/SRqqt1/assignment_loss‚0.4924985435572811 /loss/FRt1/detection_loss‚0.4809947879049646 /loss/FBt1/detection_loss‚1.0456667689451327 _runtime‚8141.291328528 epoch‚12 0loss/FRt2/assignment_loss‚0.7003684095015713 3loss/FBt1/assignment_loss‚0.0016002317845464611 /loss/FRt2/detection_loss‚1.4883377206096353 3loss/SRqqt2/assignment_loss‚0.48575839111125885 % _timestamp‚1.7842979570740993e+09 %loss/total_loss‚8.224866724596396 0loss/FRt1/assignment_loss‚1.6571384704694516 2loss/FBt2/assignment_loss‚0.003364113514671018 1loss/SRqqt1/detection_loss‚0.8690206558572559 1loss/SRqqt2/detection_loss‚0.6003005250238939 /loss/FBt2/detection_loss‚0.3998181063167334 trainer/global_step‚4899“ê±ÏQž·‚(:“ èóèÒ×ä÷ gpu.0.gpu‚83.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚46.0 gpu.0.powerWatts‚340.097' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚68.01939999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0%…L¬¸‚(:÷ èóèÒ¨ñø disk.sda2.out‚0 disk.rbd1.out‚ 19.7021484375 cpu‚25.553086421344798 network.sent‚ 9.314458e+06 network.recv‚ 2.06254e+06 disk.drbd2165.out‚0( disk.nvme0c0n1.out‚75487.18896484375' proc.memory.availableMB‚ 2915.734375$ disk./.usageGB‚551.3536605834961 disk.rbd3.in‚0# disk.nvme0c0n1.in‚ 16240.0546875 disk.sda.out‚ 386.8984375) proc.memory.percent‚17.96560287475586 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda3.out‚ 386.8984375! proc.memory.rssMB‚ 2943.484375 disk.dm-1.out‚ 386.8984375 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda3.in‚0! disk.nvme0n1.in‚ 16240.0546875 disk.sda.in‚0 proc.cpu.threads‚17 disk.rbd3.out‚0 disk.rbd1.in‚ 0.01171875$ memory_percent‚82.20376968383789) disk./.usagePercent‚37.00180677829305 disk.dm-0.in‚ 16240.0546875& disk.nvme0n1.out‚75487.18896484375 disk.dm-1.in‚0# disk.dm-0.out‚75487.18896484375 disk.drbd2165.in‚0 disk.sda2.in‚0щDÌ“¹‚(:ˆ ÷óèÒ€»óö gpu.0.gpu‚99.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚289.215' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚57.843 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¸q˜…º‚(:ý ÷óèÒ熎ù$ disk./.usageGB‚551.3699951171875 disk.drbd2165.in‚0 disk.rbd0.out‚0$ memory_percent‚82.20393657684326$ disk.nvme0c0n1.in‚16379.67578125 disk.dm-0.in‚16379.67578125# disk.dm-0.out‚77298.27392578125 disk.sda1.out‚0! proc.memory.rssMB‚ 2943.484375 proc.cpu.threads‚17 network.sent‚ 9.377667e+06 network.recv‚ 2.067136e+06 disk.sda3.in‚0 disk.sda.out‚ 387.3046875 disk.drbd2165.out‚0" disk.nvme0n1.in‚16379.67578125 disk.sda3.out‚ 387.3046875 disk.rbd1.out‚ 19.7021484375 disk.dm-1.out‚ 387.3046875) proc.memory.percent‚17.96560287475586 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.sda1.in‚0 cpu‚25.552804245309964) disk./.usagePercent‚37.00290300255467& disk.nvme0n1.out‚77298.27392578125 disk.sda2.in‚0 disk.rbd1.in‚ 0.01171875 disk.sda.in‚0 disk.dm-1.in‚0 disk.rbd0.in‚0( disk.nvme0c0n1.out‚77298.27392578125 disk.sda2.out‚0) proc.memory.availableMB‚ 2915.70703125Œ­=~Ÿ»‚(:” †ôèÒî°…÷ gpu.0.gpu‚98.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚263.941' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚52.788199999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0"¯bż‚(:ù †ôèÒ…­£ù network.recv‚ 2.070561e+06 disk.sda3.in‚0# disk.dm-0.out‚77751.59033203125! proc.memory.rssMB‚ 2943.484375 cpu‚25.552523699637458 proc.cpu.threads‚17 disk.dm-1.in‚0 disk.dm-1.out‚ 387.43359375 disk.sda1.in‚0 disk.drbd2165.in‚0 disk.nvme0n1.in‚ 16379.890625 disk.sda2.out‚0 disk.rbd1.out‚ 19.7958984375 disk.sda.in‚0 network.sent‚ 9.382676e+06) disk./.usagePercent‚37.00313750545604 disk.rbd0.in‚0( disk.nvme0c0n1.out‚77751.59033203125& disk.nvme0n1.out‚77751.59033203125$ disk./.usageGB‚551.3734893798828 disk.sda.out‚ 387.43359375 disk.rbd0.out‚0 disk.dm-0.in‚ 16379.890625 disk.sda1.out‚0 disk.drbd2165.out‚0 disk.sda2.in‚0( proc.memory.availableMB‚ 2915.9296875 disk.sda3.out‚ 387.43359375 disk.rbd3.in‚0" disk.nvme0c0n1.in‚ 16379.890625) proc.memory.percent‚17.96560287475586 disk.rbd3.out‚0 disk.rbd1.in‚ 0.01171875$ memory_percent‚82.20257759094238*œ˜ž½‚(:“ •ôèÒÍêöö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚294.623' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚58.92459999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0E”¹›ÿ¾‚(:ô •ôèÒíµ‘ù# disk.dm-0.out‚77762.26416015625 disk.sda1.in‚0 disk.nvme0n1.in‚ 16380.09375! proc.memory.rssMB‚ 2943.484375) proc.memory.percent‚17.96560287475586 proc.cpu.threads‚17# disk./.usageGB‚551.380184173584 disk.sda.in‚0) proc.memory.availableMB‚ 2915.66796875 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda3.in‚0 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.rbd1.in‚ 0.01171875 disk.rbd1.out‚ 19.7958984375 disk.sda2.out‚0 disk.dm-1.out‚ 387.6171875 disk.sda1.out‚0 disk.rbd3.in‚0& disk.nvme0n1.out‚77762.26416015625 disk.sda2.in‚0$ memory_percent‚82.20417499542236* disk./.usagePercent‚37.003586798678626 disk.dm-0.in‚ 16380.09375 disk.sda3.out‚ 387.6171875 disk.rbd3.out‚0 cpu‚25.552215508946876 network.sent‚ 9.387704e+06 network.recv‚ 2.073986e+06 disk.sda.out‚ 387.6171875 disk.dm-1.in‚0! disk.nvme0c0n1.in‚ 16380.09375( disk.nvme0c0n1.out‚77762.26416015625šíž”¿‚(:‰ ¤ôèÒ¹ùÝö gpu.0.gpu‚99.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚273.366' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.6732 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÅSpíÀ‚(:ú ¤ôèÒŸ¬ýø! proc.memory.rssMB‚ 2943.484375 disk.sda.in‚0 disk.sda2.in‚0 disk.rbd1.in‚ 0.01171875 disk.sda.out‚ 387.98828125 disk.rbd3.out‚0) proc.memory.availableMB‚ 2915.89453125 cpu‚25.551999840700095 network.recv‚ 2.077278e+06 disk.rbd0.out‚0 disk.sda3.in‚0 disk.drbd2165.out‚0" disk.nvme0c0n1.in‚ 16380.515625& disk.nvme0n1.out‚77774.87548828125) proc.memory.percent‚17.96560287475586$ disk./.usageGB‚549.1979484558105 disk.sda1.in‚0 disk.drbd2165.in‚0( disk.nvme0c0n1.out‚77774.87548828125 disk.sda3.out‚ 387.98828125 network.sent‚ 9.392765e+06 disk.dm-0.in‚ 16380.515625 disk.sda1.out‚0 disk.nvme0n1.in‚ 16380.515625 disk.sda2.out‚0 disk.rbd1.out‚ 19.8583984375 disk.rbd0.in‚0 proc.cpu.threads‚17) disk./.usagePercent‚36.85713512864112 disk.dm-1.in‚0 disk.dm-1.out‚ 387.98828125# disk.dm-0.out‚77774.87548828125 disk.rbd3.in‚0$ memory_percent‚82.20279216766357%?Þ„žÁ‚(:“ ³ôèÒ„ä‚÷ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚272.83' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚54.565999999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0õ2‚(:÷ ³ôèÒãè¹ù* disk./.usagePercent‚36.857134616626055 disk.sda1.in‚0" disk.nvme0c0n1.in‚ 16443.390625& disk.nvme0n1.out‚77791.51416015625 disk.dm-0.in‚ 16443.390625 disk.sda3.in‚0 disk.sda3.out‚ 390.41796875 disk.rbd3.out‚0 disk.dm-1.in‚0# disk.dm-0.out‚77791.51416015625 disk.sda1.out‚0 disk.dm-1.out‚ 390.41796875# memory_percent‚82.2014331817627! proc.memory.rssMB‚ 2943.484375) proc.memory.percent‚17.96560287475586# disk./.usageGB‚549.197940826416 disk.sda2.out‚0 disk.rbd1.in‚ 0.01171875 disk.sda.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 proc.cpu.threads‚17 disk.drbd2165.out‚0 disk.nvme0n1.in‚ 16443.390625 disk.rbd1.out‚ 19.8583984375 network.sent‚ 9.397769e+06 disk.drbd2165.in‚0( disk.nvme0c0n1.out‚77791.51416015625 disk.sda2.in‚0 disk.rbd3.in‚0 disk.sda.out‚ 390.41796875( proc.memory.availableMB‚ 2916.1171875 cpu‚25.55165955975201 network.recv‚ 2.080871e+06…ÑgÙ2Ã- _wandbruntime‚8220  _runtime‚8220‚Y[¿ÆÄÀ & lr-AdamW/pg1‚0.0006038968579866644 & lr-AdamW/pg2‚0.0006038968579866644 trainer/global_step‚4949 $ _timestamp‚1.784298036314987e+09 _runtime‚8220.532177296 _step‚404š4@‚ËÅÅ & lr-AdamW/pg1‚0.0006038968579866644 & lr-AdamW/pg2‚0.0006038968579866644 trainer/global_step‚4949 $ _timestamp‚1.784298036314987e+09 _runtime‚8220.532177296 _step‚404”HÌ2Æ- _wandbruntime‚8220  _runtime‚8220=z·NÇùO PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚42208143 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚42853557 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1426782922519552 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4654683615232 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚42853557 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14056686 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1421512222994432 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1422126893514752 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚26755503 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 _runtime‚8220.537176995 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚42208137 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6001291959296 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4648063154795 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 % _timestamp‚1.7842980363160474e+09 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚42208143 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚42863530 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1426782922519552 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚26755756 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚248 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚253 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1426158916120937 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚42208137 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚1jëã±2097 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚85071673 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14056438 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1426781550956544 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4654683615232 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1422126867341312 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚384327184429056 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 trainer/global_step‚4949 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _step‚405 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚390328073760768 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4646718554473 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚42863530 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0zîÓl(ÈþO ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1421512197566464 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1422126867341312 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚85061694 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1422126893514752 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚12699070 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚390328073760768 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4646718554473 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚42863530 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚42208143 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚42853557 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1426782922519552 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4654683615232 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚42853557 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14056686 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1421512222994432 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1422126893514752 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚26755503 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 _runtime‚8220.537176995 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚42208137 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6001291959296 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4648063154795 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 % _timestamp‚1.7842980363160474e+09 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚42208143 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚42863530 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1426782922519552 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚26755756 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚248 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚253 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1426158916120937 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚42208137 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚85071673 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14056438 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1426781550956544 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4654683615232 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1422126867341312 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚384327184429056 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 trainer/global_step‚4949 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _step‚405 IDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚12701613 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚42216343 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1422404868071424 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚390407161198592 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1426437721365359 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚42872022 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1427060476722176 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚42216343 % _timestamp‚1.7842980377100585e+09 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚85088371 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6002490361856 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14059134 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1421790104772608 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4648986621153 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚85088371 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚85078378 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4656954047488 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚42872022 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚26760742 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12701608 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14059374 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1422404868071424 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14059134 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1421790104772608 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4648986621153 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚85088371 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚85078378 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4656954047488 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚42872022 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚26760742 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12701608 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14059374 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1422404868071424 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚12701613 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚42216343 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1422404868071424 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚390407161198592 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1426437721365359 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119–Àj*F2Ì- _wandbruntime‚8222  _runtime‚8222vSyä Í… 1loss/SRqqt1/detection_loss‚0.7748421887869941 /loss/FBt1/detection_loss‚0.6857581594792593 % _timestamp‚1.7842980377933881e+09 _step‚407 3loss/SRqqt2/assignment_loss‚0.48308218608012843 trainer/global_step‚4949 3loss/FBt1/assignment_loss‚0.0024833435032698098 /loss/FRt2/detection_loss‚1.4611925113349198 %loss/total_loss‚7.820315177894322 0loss/FRt2/assignment_loss‚0.7292128041768685 2loss/FBt2/assignment_loss‚0.011226229311163965 1loss/SRqqt2/detection_loss‚0.6340450089272128 /loss/FBt2/detection_loss‚0.3943843425053065 epoch‚12 _runtime‚8222.010352615 /loss/FRt1/assignment_loss‚1.540183639503639 2loss/SRqqt1/assignment_loss‚0.5375007201228073 /loss/FRt1/detection_loss‚0.5664040441627526>Ý+Ί 0loss/FRt2/assignment_loss‚0.7292128041768685 2loss/FBt2/assignment_loss‚0.011226229311163965 1loss/SRqqt2/detection_loss‚0.6340450089272128 /loss/FBt2/detection_loss‚0.3943843425053065 epoch‚12 _runtime‚8222.010352615 /loss/FRt1/assignment_loss‚1.540183639503639 2loss/SRqqt1/assignment_loss‚0.5375007201228073 /loss/FRt1/detection_loss‚0.5664040441627526 1loss/SRqqt1/detection_loss‚0.7748421887869941 /loss/FBt1/detection_loss‚0.6857581594792593 % _timestamp‚1.7842980377933881e+09 _step‚407 3loss/SRqqt2/assignment_loss‚0.48308218608012843 trainer/global_step‚4949 3loss/FBt1/assignment_loss‚0.0024833435032698098 /loss/FRt2/detection_loss‚1.4611925113349198 %loss/total_loss‚7.820315177894322—Ù‡žÏ‚(:“ ÂôèÒí¸íö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚340.626' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚68.12519999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÂI5ýЂ(:ò ÂôèÒ±¦¶ù network.sent‚ 9.402788e+06 disk.rbd0.in‚0 disk.rbd3.out‚0! disk.nvme0n1.in‚ 16473.3046875! proc.memory.rssMB‚ 2943.484375 network.recv‚ 2.083964e+06$ disk./.usageGB‚549.2012596130371 disk.sda.in‚0 disk.sda.out‚ 390.87109375 disk.dm-0.out‚77800.35546875 cpu‚25.55198965261061 disk.rbd0.out‚0 disk.dm-0.in‚ 16473.3046875 disk.drbd2165.out‚0 disk.rbd1.out‚ 19.9365234375 proc.cpu.threads‚17 disk.dm-1.out‚ 390.87109375 disk.rbd3.in‚0$ memory_percent‚82.20529556274414* disk./.usagePercent‚36.857357343180844 disk.sda1.out‚0# disk.nvme0n1.out‚77800.35546875 disk.sda1.in‚0 disk.drbd2165.in‚0# disk.nvme0c0n1.in‚ 16473.3046875 disk.dm-1.in‚0 disk.sda3.in‚0 disk.sda3.out‚ 390.87109375% disk.nvme0c0n1.out‚77800.35546875 disk.sda2.in‚0 disk.rbd1.in‚ 0.01171875' proc.memory.availableMB‚ 2915.484375) proc.memory.percent‚17.96560287475586 disk.sda2.out‚0ÓûçžÑ‚(:“ ÑôèÒ­ûåö gpu.0.gpu‚90.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚275.782' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚55.15639999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0[¾þ*Ò‚(:û ÑôèÒº½¾ù network.sent‚ 9.46586e+06 disk.dm-1.out‚ 391.28515625 disk.rbd3.out‚0 disk.rbd1.in‚ 0.01171875) disk./.usagePercent‚36.85807058017124$ disk./.usageGB‚549.2118873596191 disk.sda.out‚ 391.28515625 network.recv‚ 2.088182e+06 disk.rbd0.out‚0 disk.drbd2165.out‚0 disk.rbd3.in‚0 proc.cpu.threads‚17 disk.sda1.out‚0 disk.sda3.in‚0 disk.rbd1.out‚ 19.9365234375 cpu‚25.55216709068785# disk.dm-0.out‚77819.28955078125 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.dm-0.in‚ 16473.7265625 disk.sda2.out‚0) proc.memory.availableMB‚ 1943.85546875! proc.memory.rssMB‚ 2943.484375( disk.nvme0c0n1.out‚77819.28955078125! disk.nvme0n1.in‚ 16473.7265625& disk.nvme0n1.out‚77819.28955078125 disk.sda1.in‚0 disk.sda.in‚0 disk.sda3.out‚ 391.28515625 disk.drbd2165.in‚0$ memory_percent‚88.13564777374268) proc.memory.percent‚17.96560287475586# disk.nvme0c0n1.in‚ 16473.7265625 disk.sda2.in‚0éc+ƒ”Ó‚(:‰ àôèÒ´é“÷ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚268.238' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.6476 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ûfÔ‚(:÷ àôèÒ©ˆÿø network.recv‚ 2.090953e+06! disk.nvme0n1.in‚ 16473.9453125# memory_percent‚88.1340742111206! proc.memory.rssMB‚ 2943.484375) proc.memory.percent‚17.96560287475586 proc.cpu.threads‚17 disk.drbd2165.in‚0# disk.nvme0c0n1.in‚ 16473.9453125& disk.nvme0n1.out‚77826.92138671875# disk./.usageGB‚549.216495513916 disk.sda3.out‚ 391.4921875 disk.rbd3.in‚0 disk.rbd3.out‚0( disk.nvme0c0n1.out‚77826.92138671875 disk.sda2.out‚0 disk.dm-1.out‚ 391.4921875 disk.sda1.in‚0 disk.rbd1.out‚ 20.0615234375 disk.rbd0.in‚0) proc.memory.availableMB‚ 1944.11328125 disk.dm-1.in‚0 network.sent‚ 9.470944e+06( disk./.usagePercent‚36.8583798372726 disk.sda1.out‚0 disk.drbd2165.out‚0 disk.sda2.in‚0 disk.rbd1.in‚ 0.01171875 disk.sda.in‚0 disk.dm-0.in‚ 16473.9453125# disk.dm-0.out‚77826.92138671875 disk.sda3.in‚0 disk.sda.out‚ 391.4921875 disk.rbd0.out‚0 cpu‚25.552407585117113x&єՂ(:‰ ïôèÒ³Öåö gpu.0.gpu‚95.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚216.796' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚43.3592 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0iÖ‚(:ú ïôèÒØé‚ù) disk./.usagePercent‚36.85882708243492$ disk./.usageGB‚549.2231597900391 disk.sda.out‚ 391.88671875 disk.sda1.out‚0! proc.memory.rssMB‚ 2943.484375 disk.sda3.out‚ 391.88671875& disk.nvme0n1.out‚77835.82666015625 disk.rbd3.in‚0 disk.sda.in‚0) proc.memory.percent‚17.96560287475586 disk.drbd2165.out‚0# disk.nvme0c0n1.in‚ 16474.1484375 disk.sda1.in‚0 disk.sda3.in‚0 disk.sda2.out‚0 disk.rbd1.in‚ 0.01171875 disk.rbd0.out‚0# disk.dm-0.out‚77835.82666015625' proc.memory.availableMB‚ 1944.109375 disk.drbd2165.in‚0 disk.rbd3.out‚0( disk.nvme0c0n1.out‚77835.82666015625 disk.rbd0.in‚0 disk.dm-0.in‚ 16474.1484375$ memory_percent‚88.13409805297852 proc.cpu.threads‚17 disk.dm-1.in‚0 disk.dm-1.out‚ 391.88671875 cpu‚25.55276428839598 disk.sda2.in‚0 network.sent‚ 9.475207e+06 network.recv‚ 2.094517e+06! disk.nvme0n1.in‚ 16474.1484375 disk.rbd1.out‚ 20.0615234375_¢1ž×‚(:“ þôèÒ´‘æö gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚280.367' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚56.07340000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0[*±(úØ‚(:ï þôèÒÈ×ù disk.rbd1.in‚ 0.01171875 disk.sda.in‚0 disk.dm-1.in‚0 disk.sda1.in‚0 disk.drbd2165.in‚0) proc.memory.percent‚17.96560287475586* disk./.usagePercent‚36.859139667634224$ disk./.usageGB‚549.2278175354004 disk.rbd0.in‚0 disk.sda3.out‚392.0625# disk.nvme0c0n1.in‚ 16562.7421875 network.sent‚ 9.480213e+06 network.recv‚ 2.097771e+06 disk.rbd0.out‚0 disk.sda1.out‚0 disk.drbd2165.out‚0 disk.rbd3.out‚0 disk.dm-1.out‚392.0625 disk.dm-0.in‚ 16562.7421875 disk.rbd3.in‚0 cpu‚25.55299979690328 disk.rbd1.out‚ 20.1396484375 proc.cpu.threads‚17 disk.sda2.in‚0& disk.nvme0n1.out‚77844.09814453125$ memory_percent‚88.13409805297852' proc.memory.availableMB‚ 1944.109375 disk.sda2.out‚0 disk.sda.out‚392.0625# disk.dm-0.out‚77844.09814453125 disk.sda3.in‚0! disk.nvme0n1.in‚ 16562.7421875! proc.memory.rssMB‚ 2943.484375( disk.nvme0c0n1.out‚77844.09814453125‰Uæå2Ù- _wandbruntime‚8300  _runtime‚8300ê{œtÇÚÁ & lr-AdamW/pg1‚0.0006020415925600081 & lr-AdamW/pg2‚0.0006020415925600081 trainer/global_step‚4999 % _timestamp‚1.7842981158767438e+09 _runtime‚8300.093917706 _step‚408B=ÌÛÆ & lr-AdamW/pg2‚0.0006020415925600081 trainer/global_step‚4999 % _timestamp‚1.7842981158767438e+09 _runtime‚8300.093917706 _step‚408 & lr-AdamW/pg1‚0.0006020415925600081˜ÌM:2Ü- _wandbruntime‚8300  _runtime‚8300Hgžÿ'ÝùO BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6061110051840 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚42618443 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1440100612171135 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14192032 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 % _timestamp‚1.7842981158778021e+09 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚254 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4702344935424 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚43289282 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14192286 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1440099242142845 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚42618437 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚388201136721920 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4700999545856 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1440727903392768 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚27018506 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚259 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4694300267903 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚8300.095324282 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚27018247 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚394262614864896 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚12826220 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1435406311903232 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 _step‚409 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1440727903392768 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚85907725 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚388201504813056 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6061075515392 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175™l½j(ÞþO 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1440100612171135 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14192032 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 % _timestamp‚1.7842981158778021e+09 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚254 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4702344935424 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚43289282 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14192286 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1440099242142845 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚42618437 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚388201136721920 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4700999545856 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1440727903392768 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚27018506 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚259 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4694300267903 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚8300.095324282 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚27018247 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚394262614864896 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚12826220 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1435406311903232 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 _step‚409 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1440727903392768 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚85907725 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚388201504813056 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6061075515392 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6061110051840 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚42618443™±°´b2ß- _wandbruntime‚8301  _runtime‚8301Ø ‡ëÆà‚N _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1436304930750464 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 ?6DeviceStatsMonit­qj]B&or.on_train_batch_end/num_device_alloc‚1995 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚42626643 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚85924423 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚12828763 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4701924581376 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4693879133827 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4695223734261 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1441006829158400 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚42626643 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14194755 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1436304904577024 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1435684168253440 _runtime‚8301.479196401 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚43297774 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚249 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚394340892262400 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1436304930750464 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚410 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚85914430 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚85924423 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6062303662080 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚27023762 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚42626649 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 $ _timestamp‚1.784298117261833e+09 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚85914430 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4703269978112 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚244 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚27023513 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚394341294882816 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚388278623129600 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚42626649 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4703269978112 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚43297774 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14194999 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1435684193681408 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚43287787 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚43287787 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1441008200728576 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1436304904577024 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1435684168253440 _runtime‚8301.479196401 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚43297774 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚249 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚394340892262400 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1436304930750464 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚410 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚85914430 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚85924423 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6062303662080 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚27023762 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚42626649 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 $ _timestamp‚1.784298117261833e+09 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚85914430 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4703269978112 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚244 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚27023513 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚394341294882816 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚388278623129600 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚42626649 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4703269978112 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚43297774 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14194999 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1435684193681408 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚43287787 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚43287787 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1441008200728576 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4701924581376 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4693879133827 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4695223734261šõ&In2â- _wandbruntime‚8301  _runtime‚8301ɽˆã‚ 2loss/SRqqt2/assignment_loss‚0.5210304329893606 /loss/FBt2/detection_loss‚0.4817526801606441 epoch‚12 _runtime‚8301.566129044 /loss/FRt1/assignment_loss‚1.527906807375626 2loss/SRqqt1/assignment_loss‚0.5273814377265367 2loss/FBt2/assignment_loss‚0.026259142108484422 0loss/SRqqt1/detection_loss‚0.942035483408042 %loss/total_loss‚8.218602489893271 2loss/FBt1/assignment_loss‚0.002461082123170232 /loss/FRt1/detection_loss‚0.5720963431605531 1loss/SRqqt2/detection_loss‚0.6663121121283356 /loss/FBt1/detection_loss‚0.7449395003290299 trainer/global_step‚4999 /loss/FRt2/detection_loss‚1.5611739364042516 % _timestamp‚1.7842981173491125e+09 _step‚411 0loss/FRt2/assignment_loss‚0.6452535319792383Ù)1À ä‡ /loss/FRt2/detection_loss‚1.5611739364042516 % _timestamp‚1.7842981173491125e+09 _step‚411 0loss/FRt2/assignment_loss‚0.6452535319792383 2loss/SRqqt2/assignment_loss‚0.5210304329893606 /loss/FBt2/detection_loss‚0.4817526801606441 epoch‚12 _runtime‚8301.566129044 /loss/FRt1/assignment_loss‚1.527906807375626 2loss/SRqqt1/assignment_loss‚0.5273814377265367 2loss/FBt2/assignment_loss‚0.026259142108484422 0loss/SRqqt1/detection_loss‚0.942035483408042 %loss/total_loss‚8.218602489893271 2loss/FBt1/assignment_loss‚0.002461082123170232 /loss/FRt1/detection_loss‚0.5720963431605531 1loss/SRqqt2/detection_loss‚0.6663121121283356 /loss/FBt1/detection_loss‚0.7449395003290299 trainer/global_step‚4999›¿¢è‰“å‚(:ˆ õèÒ÷Óûö gpu.0.gpu‚73.0 gpu.0.memory‚24, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚330.145' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚66.029 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‰„èÁÿæ‚(:ô õèÒ©¸Äù disk.dm-1.out‚ 392.46875! proc.memory.rssMB‚ 2943.484375$ disk./.usageGB‚549.1817932128906# disk.nvme0c0n1.in‚ 16563.1640625! disk.nvme0n1.in‚ 16563.1640625 disk.rbd1.in‚ 0.01171875 disk.sda.out‚ 392.46875$ memory_percent‚88.13416957855225) proc.memory.availableMB‚ 1944.09765625 cpu‚25.553381062506098 disk.rbd3.in‚0( disk.nvme0c0n1.out‚77854.26904296875 disk.rbd1.out‚ 20.1396484375 disk.sda.in‚0 disk.dm-1.in‚0 disk.sda3.out‚ 392.46875) proc.memory.percent‚17.96560287475586 network.sent‚ 9.485277e+06 disk.rbd3.out‚0 disk.rbd0.in‚0 disk.sda1.out‚0 disk.drbd2165.out‚0 disk.sda2.out‚0 disk.dm-0.in‚ 16563.1640625# disk.dm-0.out‚77854.26904296875 disk.sda3.in‚0 disk.drbd2165.in‚0 network.recv‚ 2.101366e+06) disk./.usagePercent‚36.85605093673362& disk.nvme0n1.out‚77854.26904296875 disk.sda2.in‚0 disk.rbd0.out‚0 disk.sda1.in‚0 proc.cpu.threads‚17u£qkŸç‚(:” œõèÒ×ïýö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚271.506' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚54.301199999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¬s(è‚(:ú œõèÒ“—­ù disk.sda1.out‚0 disk.drbd2165.out‚0 disk.rbd3.out‚0( disk.nvme0c0n1.out‚77866.57275390625 disk.rbd1.in‚ 0.01171875 disk.sda.in‚0 disk.rbd3.in‚0! disk.nvme0n1.in‚ 16563.3671875& disk.nvme0n1.out‚77866.57275390625 cpu‚25.553555834715247 disk.rbd1.out‚ 20.2021484375& proc.memory.availableMB‚ 1944.09375 disk.dm-0.in‚ 16563.3671875 disk.sda2.out‚0 proc.cpu.threads‚17# disk.dm-0.out‚77866.57275390625# disk.nvme0c0n1.in‚ 16563.3671875) proc.memory.percent‚17.96560287475586 network.sent‚ 9.548324e+06) disk./.usagePercent‚36.85632204871238 disk.rbd0.out‚0 disk.sda3.in‚0$ disk./.usageGB‚549.1858329772949 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.sda1.in‚0 disk.sda3.out‚ 392.65234375$ memory_percent‚88.13419342041016! proc.memory.rssMB‚ 2943.484375 network.recv‚ 2.105445e+06 disk.drbd2165.in‚0 disk.sda2.in‚0 disk.sda.out‚ 392.65234375 disk.dm-1.out‚ 392.65234375#]—_žé‚(:“ «õèÒ¯ðö gpu.0.gpu‚48.0 gpu.0.memory‚15, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚325.553' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚65.11059999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ª&÷yûê‚(:ð «õèÒóóù disk.rbd0.in‚0 disk.drbd2165.out‚0( proc.memory.availableMB‚ 1944.2421875 disk.sda2.out‚0! disk.dm-0.out‚77876.162109375& disk.nvme0c0n1.out‚77876.162109375 disk.rbd3.in‚0 disk.rbd3.out‚0# disk.nvme0c0n1.in‚ 16563.5859375! disk.nvme0n1.in‚ 16563.5859375 disk.sda2.in‚0 disk.rbd1.in‚ 0.01171875 disk.sda.in‚0 disk.sda.out‚ 392.921875 network.sent‚ 9.553343e+06 disk.dm-1.in‚0 disk.sda1.out‚0 disk.drbd2165.in‚0$ memory_percent‚88.13328742980957) proc.memory.percent‚17.96562671661377 network.recv‚ 2.108943e+06$ disk.nvme0n1.out‚77876.162109375 disk.dm-1.out‚ 392.921875 disk.rbd0.out‚0 disk.dm-0.in‚ 16563.5859375 disk.sda1.in‚0 disk.sda3.in‚0 cpu‚25.55393947458432( disk./.usagePercent‚36.8567782541384$ disk./.usageGB‚549.1926307678223 disk.rbd1.out‚ 20.2021484375 disk.sda3.out‚ 392.921875# proc.memory.rssMB‚ 2943.48828125 proc.cpu.threads‚17‹.žë‚(:“ ºõèÒö§‚÷ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚284.135' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚56.82699999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0\à úì‚(:÷ ºõèÒ—ÇŒù% disk.nvme0n1.out‚77884.0029296875 disk.sda.in‚0 disk.sda.out‚ 393.30078125 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda1.in‚0 disk.nvme0n1.in‚ 16648.234375) proc.memory.percent‚17.96562671661377 network.sent‚ 9.558505e+06" disk.dm-0.out‚77884.0029296875 disk.drbd2165.out‚0 disk.sda2.in‚0# proc.memory.rssMB‚ 2943.48828125 disk.sda1.out‚0 disk.sda3.out‚ 393.30078125 disk.rbd3.out‚0 disk.rbd1.in‚ 0.01171875( disk./.usagePercent‚36.8572590362877 disk.sda3.in‚0) proc.memory.availableMB‚ 1943.76953125 cpu‚25.553634231239254 proc.cpu.threads‚17 disk.rbd3.in‚0' disk.nvme0c0n1.out‚77884.0029296875 disk.dm-1.in‚0 disk.drbd2165.in‚0 disk.sda2.out‚0# memory_percent‚88.1361722946167 network.recv‚ 2.112249e+06" disk.nvme0c0n1.in‚ 16648.234375 disk.rbd1.out‚ 20.2802734375$ disk./.usageGB‚549.1997947692871 disk.dm-1.out‚ 393.30078125 disk.dm-0.in‚ 16648.234375=ºÔªží‚(:“ ÉõèÒ„³™ø gpu.0.gpu‚99.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚46.0 gpu.0.powerWatts‚334.292' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚66.85839999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚03M—¼úî‚(:ï ÉõèÒ—µù proc.cpu.threads‚17 disk.sda.out‚ 393.4921875 disk.dm-1.out‚ 393.4921875 disk.drbd2165.out‚0 disk.sda2.out‚0 disk.sda1.out‚0 disk.nvme0c0n1.in‚ 16648.4375' disk.nvme0c0n1.out‚77898.2744140625% disk.nvme0n1.out‚77898.2744140625 network.recv‚ 2.115806e+06 disk.rbd1.in‚ 0.01171875 disk.rbd0.in‚0 disk.sda3.in‚0 disk.rbd1.out‚ 20.2802734375 disk.sda3.out‚ 393.4921875 disk.sda2.in‚0# proc.memory.rssMB‚ 2943.48828125# disk./.usageGB‚549.203670501709) proc.memory.percent‚17.96562671661377 network.sent‚ 9.563517e+06) disk./.usagePercent‚36.85751913994248 disk.dm-1.in‚0 disk.rbd0.out‚0" disk.dm-0.out‚77898.2744140625 disk.rbd3.in‚0 disk.nvme0n1.in‚ 16648.4375$ memory_percent‚88.13765048980713 disk.rbd3.out‚0 disk.sda.in‚0 disk.dm-0.in‚ 16648.4375 disk.sda1.in‚0 disk.drbd2165.in‚0) proc.memory.availableMB‚ 1943.52734375 cpu‚25.553359629380736hœ£¥2ï- _wandbruntime‚8379  _runtime‚8379•>ZmÇðÁ _runtime‚8379.063985189 _step‚412 & lr-AdamW/pg1‚0.0006001698771934883 & lr-AdamW/pg2‚0.0006001698771934883 trainer/global_step‚5049 % _timestamp‚1.7842981948454108e+09©»—FÌñÆ & lr-AdamW/pg1‚0.0006001698771934883 & lr-AdamW/pg2‚0.0006001698771934883 trainer/global_step‚5049 % _timestamp‚1.7842981948454108e+09 _runtime‚8379.063985189 _step‚412œ—´PC2ò- _wandbruntime‚8379  _runtime‚8379‰Žzà óøO MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚86743777 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12953365 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14328760 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4740537381011 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚86733798 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚27282125 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚43028743 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 $ _timestamp‚1.784298194846212e+09 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1449926940352512 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404724736 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1449926966525952 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚43028743 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14329028 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚392075457105920 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚43028737 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚43705061 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚273 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚43715034 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1454674255828992 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4747315476480 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 aLDeviceStatsÉ4¼…Monitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1449300400812032 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4739192780689 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1454674255828992 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1454040938193043 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚398196385454080 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4748660866048 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4747315476480 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1449926966525952 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6120560257024 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚12953370 _runtime‚8379.064104284 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1454675627392000 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4748660866048 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚43715034 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚5049 _step‚413 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚43705061 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1454675627392000 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚43028737 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚398195980729344 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008™Lµµ(ôýO M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404724736 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1449926966525952 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚43028743 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14329028 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚392075457105920 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚43028737 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚43705061 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚273 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚43715034 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1454674255828992 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4747315476480 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1449300400812032 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4739192780689 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1454674255828992 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1454040938193043 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚398196385454080 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4748660866048 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4747315476480 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1449926966525952 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6120560257024 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚12953370 _runtime‚8379.064104284 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1454675627392000 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4748660866048 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚43715034 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚5049 _step‚413 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚43705061 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1454675627392000 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚43028737 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚398195980729344 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚86743777 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12953365 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14328760 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4740537381011 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚86733798 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚27282125 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚43028743 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 $ _timestamp‚1.784298194846212e+09 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1449926940352512 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820ݳ×2õ- _wandbruntime‚8380  _runtime‚8380%7à 'öƒN ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1454954553164800 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚27287194 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14331286 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚392152943513600 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1454318373409175 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚43723526 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚86760475 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚43036949 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 trainer/global_step‚5049 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚43723526 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1454953181594624 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6121748918272 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7842981962292426e+09 _step‚414 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚86750482 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚27287456 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚392153311604736 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚43713539 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4741460847369 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4749585908736 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12955908 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1449578257162240 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1450204967256064 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1450204967256064 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14331543 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚257 _runtime‚8380.446700359 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1454954553164800 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚86760475 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1450204941082624 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚43036943 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4749585908736 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚86750482 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚12955913 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4740116246935 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚43713539 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1454319743437577 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175/éw÷ˆN NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚43036943 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚27287194 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14331286 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚392152943513600 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1454318373409175 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚43723526 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚86760475 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚43036949 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 trainer/global_step‚5049 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚43723526 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1454953181594624 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6121748918272 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7842981962292426e+09 _step‚414 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚86750482 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚27287456 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚392153311604736 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚43713539 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4741460847369 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4749585908736 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12955908 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 LDDeviceStatsMonitor.on_train_batch_end/in•åîactive_split.small_pool.peak‚545 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1449578257162240 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1450204967256064 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1450204967256064 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14331543 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚257 _runtime‚8380.446700359 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1454954553164800 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚86760475 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1450204941082624 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚43036943 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4749585908736 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚86750482 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚12955913 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4740116246935 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚43713539 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1454319743437577 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1454954553164800 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 E/2ˆ- _wandbruntime‚8457  _runtime‚8457¡¾"ÿ'‰ùO ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1463827003031552 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚43439037 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚44130813 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚255 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚43439037 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1463827003031552 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14464085 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚43439043 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4793631407104 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6179961991168 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4786774494119 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 % _timestamp‚1.7842982733444588e+09 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4794976796672 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚395949777489920 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1463826976858112 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1467979894186661 _step‚417 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1463826976858112 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1468620608265216 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14464335 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1463194489720832 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚27544600 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚13080520 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1468621979828224 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚87579829 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1468620608265216 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚43439043 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚87569850 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4793631407104 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4794976796672 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚44140786 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚402129739481088 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚44140786 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008Ðu–¬"ŠþO F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4794976796672 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚395949777489920 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1463826976858112 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1467979894186661 _step‚417 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1463826976858112 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1468620608265216 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14464335 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1463194489720832 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚27544600 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚13080520 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1468621979828224 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚87579829 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1468620608265216 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚43439043 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚87569850 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4793631407104 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4794976796672 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚44140786 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚402129739481088 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚44140786 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1463827003031552 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚43439037 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚44130813 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚255 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚43439037 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1463827003031552 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.lar_’u,Xge_pool.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14464085 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚43439043 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4793631407104 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6179961991168 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4786774494119 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 % _timestamp‚1.7842982733444588e+09¡]C‡‰2‹- _wandbruntime‚8458  _runtime‚8458?}“ 'ŒƒN _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1464105003761664 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚87586534 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6181165938688 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1463472346071040 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚87596527 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1464105003761664 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1464104977588224 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚27549898 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13083058 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1468900905601024 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚87596527 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚44149278 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6181198370816 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1463472371499008 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4787697960477 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14466840 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4795901839360 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚44139291 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚239 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚13083063 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚234 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚44139291 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚44149278 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚396027263897600 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4794556442624 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚27550137 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚43447249 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4786353360043 _runtime‚8458.947357695 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 trainer/global_step‚5099 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926ĽÛ'ˆN REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14466840 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4795901839360 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚44139291 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚239 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚13083063 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚234 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚44139291 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚44149278 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚396027263897600 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4794556442624 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚27550137 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚43447249 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4786353360043 _runtime‚8458.947357695 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 trainer/global_step‚5099 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1464105003761664 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚87586534 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6181165938688 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1463472346071040 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚87596527 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1464105003761664 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1464104977588224 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚27549898 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13083058 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1468900905601024 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚87596527 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚44149278 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6181198370816 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1463472371499008 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4787697960477 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175¢ÑLD2Ž- _wandbruntime‚8459  _runtime‚8459õ· … % _timestamp‚1.7842982748169577e+09 _step‚419 0loss/FRt1/assignment_loss‚1.5963918990210484 3loss/FBt1/assignment_loss‚0.0005947479214388954 2loss/FBt2/assignment_loss‚0.006932474532146503 /loss/FRt1/detection_loss‚0.5463164218588105 /loss/FBt1/detection_loss‚1.0796574804739192 _runtime‚8459.033887592 0loss/FRt2/assignment_loss‚0.6644004055307907 2loss/SRqqt1/assignment_loss‚0.5111079928972158 2loss/SRqqt2/assignment_loss‚0.4766010568831023 /loss/FBt2/detection_loss‚0.4437316042183536 epoch‚12 /loss/FRt2/detection_loss‚1.4100916856420633 1loss/SRqqt1/detection_loss‚0.8569046202883168 1loss/SRqqt2/detection_loss‚0.6264943907272567 %loss/total_loss‚8.219224779994462 trainer/global_step‚5099œ}ZŠ trainer/global_step‚5099 % _timestamp‚1.7842982748169577e+09 _step‚419 0loss/FRt1/assignment_loss‚1.5963918990210484 3loss/FBt1/assignment_loss‚0.0005947479214388954 2loss/FBt2/assignment_loss‚0.006932474532146503 /loss/FRt1/detection_loss‚0.5463164218588105 /loss/FBt1/detection_loss‚1.0796574804739192 _runtime‚8459.033887592 0loss/FRt2/assignment_loss‚0.6644004055307907 2loss/SRqqt1/assignment_loss‚0.5111079928972158 2loss/SRqqt2/assignment_loss‚0.4766010568831023 /loss/FBt2/detection_loss‚0.4437316042183536 epoch‚12 /loss/FRt2/detection_loss‚1.4100916856420633 1loss/SRqqt1/detection_loss‚0.8569046202883168 1loss/SRqqt2/detection_loss‚0.6264943907272567 %loss/total_loss‚8.219224779994462£®ŽyŒ“‘‚(:ˆ £öèÒ†Ûÿö gpu.0.gpu‚91.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚244.025' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚48.805 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ó0•¾’‚(:ù £öèÒ„Ÿ´ù disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.rbd1.out‚ 20.4833984375$ memory_percent‚88.14222812652588& disk.nvme0n1.out‚77969.91650390625 cpu‚25.551583343114537 proc.cpu.threads‚17 network.sent‚ 9.652171e+06 network.recv‚ 2.136936e+06 disk.sda3.in‚0 disk.rbd1.in‚ 0.01171875 disk.dm-1.in‚0# disk.nvme0c0n1.in‚ 16748.0546875 disk.rbd3.out‚0# proc.memory.rssMB‚ 2943.48828125 disk.sda.in‚0 disk.dm-1.out‚ 398.703125 disk.sda1.out‚0 disk.sda3.out‚ 398.703125 disk.rbd3.in‚0( disk.nvme0c0n1.out‚77969.91650390625! disk.nvme0n1.in‚ 16748.0546875 disk.sda2.out‚0) disk./.usagePercent‚36.85967753946366$ disk./.usageGB‚549.2358322143555 disk.rbd0.in‚0# disk.dm-0.out‚77969.91650390625 disk.sda1.in‚0 disk.sda.out‚ 398.703125) proc.memory.availableMB‚ 1942.77734375) proc.memory.percent‚17.96562671661377 disk.rbd0.out‚0 disk.dm-0.in‚ 16748.0546875 disk.sda2.in‚0†`9äž“‚(:“ ²öèÒ­Ðø gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚297.386' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚59.47720000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0QOÑ”‚(:ü ²öèÒïòßù* disk./.usagePercent‚36.859982700444476 disk.dm-1.out‚ 399.390625 cpu‚25.551313842905767 disk.sda2.in‚0 disk.rbd3.in‚0$ disk.nvme0c0n1.in‚16883.77734375 disk.sda3.in‚0 network.sent‚ 9.715262e+06 disk.drbd2165.out‚0 disk.rbd1.out‚ 20.5615234375 disk.rbd0.out‚0$ disk./.usageGB‚549.2403793334961 disk.rbd3.out‚0 disk.rbd0.in‚0 disk.sda3.out‚ 399.390625( proc.memory.availableMB‚ 1943.2578125 disk.sda2.out‚0 disk.dm-1.in‚0 disk.dm-0.in‚16883.77734375# disk.dm-0.out‚77978.08935546875 disk.sda1.in‚0 disk.sda1.out‚0 network.recv‚ 2.141228e+06" disk.nvme0n1.in‚16883.77734375 disk.sda.in‚0# proc.memory.rssMB‚ 2943.48828125) proc.memory.percent‚17.96562671661377 disk.sda.out‚ 399.390625$ memory_percent‚88.13929557800293 proc.cpu.threads‚17& disk.nvme0n1.out‚77978.08935546875 disk.drbd2165.in‚0( disk.nvme0c0n1.out‚77978.08935546875 disk.rbd1.in‚ 0.01171875E!Ÿ•‚(:” ÁöèÒÎ…ø gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚276.599' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚55.319799999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0žJ5–‚(:õ ÁöèÒÆÆ®ù disk.rbd1.out‚ 20.5615234375 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.drbd2165.out‚0 disk.rbd3.out‚0) proc.memory.percent‚17.96562671661377 cpu‚25.55107881643314 proc.cpu.threads‚17 disk.sda2.out‚0& disk.nvme0n1.out‚77988.75048828125# proc.memory.rssMB‚ 2943.48828125 disk.sda.in‚0 disk.dm-0.in‚16884.19921875" disk.nvme0n1.in‚16884.19921875 disk.dm-1.in‚0 disk.sda1.in‚0 disk.sda1.out‚0$ disk.nvme0c0n1.in‚16884.19921875( disk.nvme0c0n1.out‚77988.75048828125 network.sent‚ 9.719788e+06) disk./.usagePercent‚36.86061171095611 disk.sda.out‚ 399.703125# proc.memory.availableMB‚1942.75 network.recv‚ 2.145039e+06 disk.sda2.in‚0# disk.dm-0.out‚77988.75048828125 disk.rbd3.in‚0 disk.rbd1.in‚ 0.01171875 disk.sda3.out‚ 399.703125$ disk./.usageGB‚549.2497520446777 disk.dm-1.out‚ 399.703125 disk.sda3.in‚0 disk.drbd2165.in‚0$ memory_percent‚88.14239501953125­Xn1”—‚(:‰ ÐöèÒ–Ê–÷ gpu.0.gpu‚93.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚344.528' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚68.9056 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0º¤ž˜‚(:÷ ÐöèÒ²Âù disk.sda1.in‚0) disk./.usagePercent‚36.86089434327391# disk./.usageGB‚549.253963470459' disk.nvme0c0n1.out‚78001.8447265625 disk.rbd0.out‚0 proc.cpu.threads‚17 disk.rbd3.out‚0 disk.dm-1.out‚ 400.109375$ memory_percent‚88.13860416412354) proc.memory.availableMB‚ 1943.37109375 cpu‚25.55081483550063 network.recv‚ 2.148307e+06 disk.sda3.out‚ 400.109375 disk.drbd2165.in‚0 disk.rbd3.in‚0 disk.rbd1.in‚ 0.01171875 disk.rbd1.out‚ 20.6240234375 network.sent‚ 9.724813e+06 disk.sda1.out‚0$ disk.nvme0c0n1.in‚16884.40234375" disk.nvme0n1.in‚16884.40234375% disk.nvme0n1.out‚78001.8447265625 disk.sda2.in‚0 disk.sda.in‚0" disk.dm-0.out‚78001.8447265625 disk.sda3.in‚0 disk.sda.out‚ 400.109375 disk.drbd2165.out‚0 disk.sda2.out‚0 disk.dm-1.in‚0# proc.memory.rssMB‚ 2943.48828125) proc.memory.percent‚17.96562671661377 disk.dm-0.in‚16884.40234375 disk.rbd0.in‚0éïtÛŸ™‚(:” ßöèÒ±Üàö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚264.778' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚52.955600000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ëjä>š‚(:ø ßöèÒ¨ïÈø disk.sda.out‚ 400.484375 disk.drbd2165.in‚0 network.recv‚ 2.151836e+06( disk./.usagePercent‚36.8612120486239 disk.dm-1.in‚0 disk.sda3.out‚ 400.484375 disk.rbd3.in‚0 disk.rbd3.out‚0$ memory_percent‚88.14144134521484 network.sent‚ 9.730406e+06 disk.sda2.in‚0 disk.rbd0.in‚0 disk.sda2.out‚0# proc.memory.rssMB‚ 2943.48828125 cpu‚25.550494748333513 proc.cpu.threads‚17 disk.dm-0.in‚16884.62109375 disk.sda1.out‚0$ disk.nvme0c0n1.in‚16884.62109375& proc.memory.availableMB‚ 1942.90625) proc.memory.percent‚17.96562671661377 disk.rbd1.out‚ 20.6240234375 disk.sda1.in‚0 disk.dm-1.out‚ 400.484375 disk.rbd0.out‚0 disk.sda3.in‚0" disk.nvme0n1.in‚16884.62109375# disk.dm-0.out‚78013.36865234375 disk.drbd2165.out‚0( disk.nvme0c0n1.out‚78013.36865234375$ disk./.usageGB‚549.2586975097656 disk.sda.in‚0& disk.nvme0n1.out‚78013.36865234375 disk.rbd1.in‚ 0.01171875°­Ÿ›‚(:” îöèÒ¦Áâ÷ gpu.0.gpu‚64.0 gpu.0.memory‚21, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚38.0 gpu.0.powerWatts‚109.588' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.917599999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0H×ùœ‚(:î îöèÒèŒÌø network.sent‚ 9.73547e+06$ disk./.usageGB‚549.2680358886719" disk.nvme0n1.out‚ 78023.1328125 cpu‚25.550230823487983 network.recv‚ 2.154833e+06 disk.sda2.in‚0 disk.sda2.out‚0$ disk.nvme0c0n1.in‚17000.53515625 disk.rbd0.in‚0 disk.drbd2165.in‚0# proc.memory.rssMB‚ 2943.48828125 disk.sda.out‚ 400.703125 disk.dm-1.in‚0 disk.sda1.in‚0 disk.sda3.out‚ 400.703125$ memory_percent‚88.13996315002441 disk.rbd3.in‚0$ disk.nvme0c0n1.out‚ 78023.1328125) disk./.usagePercent‚36.86183875506772 disk.rbd1.in‚ 0.01171875 disk.dm-0.in‚17000.53515625 disk.dm-0.out‚ 78023.1328125 disk.sda1.out‚0 disk.drbd2165.out‚0" disk.nvme0n1.in‚17000.53515625 disk.sda.in‚0 disk.rbd0.out‚0 disk.sda3.in‚0 disk.rbd3.out‚0( proc.memory.availableMB‚ 1943.1484375) proc.memory.percent‚17.96562671661377 disk.rbd1.out‚ 20.7021484375 disk.dm-1.out‚ 400.703125 proc.cpu.threads‚17»ÇÌ2- _wandbruntime‚8536  _runtime‚85360±~ÆžÀ & lr-AdamW/pg2‚0.0005963775785568074 trainer/global_step‚5149 $ _timestamp‚1.784298352093105e+09 _runtime‚8536.310247139 _step‚420 & lr-AdamW/pg1‚0.0005963775785568074Â!ËŸÅ $ _timestamp‚1.784298352093105e+09 _runtime‚8536.310247139 _step‚420 & lr-AdamW/pg1‚0.0005963775785568074 & lr-AdamW/pg2‚0.0005963775785568074 trainer/global_step‚5149¤[-:À2 - _wandbruntime‚8536  _runtime‚8536ªyu<Ê¡ùO TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚5149 PCDeviceSt¯kþ5&atsMonitor.on_train_batch_start/active.large_pool.allocated‚43849343 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1481920220208569 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4839947337728 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚13207665 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14600803 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚44556565 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 % _timestamp‚1.7842983520941746e+09 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6239871187968 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚44556565 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4839947337728 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚43849337 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4841292727296 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚44566538 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1477727039537152 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4841292727296 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚247 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1482566960701440 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1477727013363712 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1477727013363712 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4833011607227 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14600556 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4831667006905 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 13446003226.÷G(¢þO aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚399824097873920 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1482568332264448 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚406063934525440 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 _runtime‚8536.314952704 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4841292727296 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚247 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1482566960701440 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1477727013363712 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1477727013363712 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4833011607227 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14600556 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4831667006905 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚5149 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚43849343 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1481920220208569 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4839947337728 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚13207665 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14600803 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚44556565 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 % _timestamp‚1.7842983520941746e+09 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6239871187968 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚44556565 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4839947337728 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚43849337 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4841292727296 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚44566538 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1477727039537152 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 IDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1482200395481393 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚44575030 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚399901952372736 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4842217769984 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1482847258037248 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14603526 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4832590473151 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚27813494 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 % _timestamp‚1.7842983534768748e+09 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚88422586 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚240 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1477366434979840 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4840872373248 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚27813739 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚245 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 trainer/global_step‚5149 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1478005040267264 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚44565043 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚44565043 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚44575030 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚88432579 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4833935073585 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚88422586 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚13210213 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4842217769984 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1478005040267264 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚399901584281600 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 _step‚422 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1482199025452991 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13210208 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚406142622480384 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1477366460407808 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚43857549 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1478005014093824 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008®”“ti ¥ˆN EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚27813494 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 % _timestamp‚1.7842983534768748e+09 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚88422586 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚240 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1477366434979840 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4840872373248 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚27813739 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚245 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 trainer/global_step‚5149 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1478005040267264 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚44565043 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚44565K󫘥043 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚44575030 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚88432579 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4833935073585 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚88422586 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚13210213 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4842217769984 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1478005040267264 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚399901584281600 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 _step‚422 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1482199025452991 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13210208 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚406142622480384 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1477366460407808 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚43857549 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1478005014093824 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 _runtime‚8537.694375151 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1482847258037248 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1482200395481393 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚44575030 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚399901952372736 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4842217769984 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1482847258037248 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14603526 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4832590473151 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0¦cÈhG2¦- _wandbruntime‚8537  _runtime‚8537yM §„ 0loss/FRt2/assignment_loss‚0.6156015108419071 3loss/SRqqt2/assignment_loss‚0.45424036764023723 /loss/FRt1/detection_loss‚0.5169804375442789 1loss/SRqqt2/detection_loss‚0.5768957280573553 %loss/total_loss‚8.381322461200972 epoch‚12 trainer/global_step‚5149 _step‚423 0loss/FRt1/assignment_loss‚1.6339592901122035 2loss/SRqqt1/assignment_loss‚0.5159089372848092 1loss/FBt1/assignment_loss‚0.04369156328137924 2loss/FBt2/assignment_loss‚0.004940650926193322 /loss/FBt2/detection_loss‚0.5297447861304312 _runtime‚8537.780582855 /loss/FRt2/detection_loss‚1.4017476478793742 1loss/SRqqt1/detection_loss‚1.0183433090657767 /loss/FBt1/detection_loss‚1.0692682324370253 % _timestamp‚1.7842983535635955e+09éI¨‰ /loss/FBt2/detection_loss‚0.5297447861304312 _runtime‚8537.780582855 /loss/FRt2/detection_loss‚1.4017476478793742 1loss/SRqqt1/detection_loss‚1.0183433090657767 /loss/FBt1/detection_loss‚1.0692682324370253 % _timestamp‚1.7842983535635955e+09 0loss/FRt2/assignment_loss‚0.6156015108419071 3loss/SRqqt2/assignment_loss‚0.45424036764023723 /loss/FRt1/detection_loss‚0.5169804375442789 1loss/SRqqt2/detection_loss‚0.5768957280573553 %loss/total_loss‚8.381322461200972 epoch‚12 trainer/global_step‚5149 _step‚423 0loss/FRt1/assignment_loss‚1.6339592901122035 2loss/SRqqt1/assignment_loss‚0.5159089372848092 1loss/FBt1/assignment_loss‚0.04369156328137924 2loss/FBt2/assignment_loss‚0.004940650926193322§ê’›§“©‚(:ˆ ýöèÒîÖ’÷ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚289.215' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚57.843 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ò ±ýª‚(:ò ýöèÒ”šÞø proc.cpu.threads‚17 disk.dm-1.in‚0' disk.nvme0c0n1.out‚78038.9326171875 disk.sda2.out‚0 disk.rbd1.out‚ 20.7021484375$ memory_percent‚88.14435005187988$ disk./.usageGB‚549.2723083496094" disk.dm-0.out‚78038.9326171875" disk.nvme0n1.in‚17000.73828125 cpu‚25.549908840319834 disk.sda.in‚0 disk.dm-1.out‚401.1875 disk.rbd0.out‚0 disk.sda3.out‚401.1875 disk.drbd2165.out‚0 disk.rbd3.in‚0 disk.rbd3.out‚0) proc.memory.percent‚17.96562671661377 disk.sda2.in‚0 disk.rbd0.in‚0 network.recv‚ 2.158314e+06 disk.dm-0.in‚17000.73828125 disk.sda.out‚401.1875) disk./.usagePercent‚36.86212548350607 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd2165.in‚0$ disk.nvme0c0n1.in‚17000.73828125% disk.nvme0n1.out‚78038.9326171875( proc.memory.availableMB‚ 1942.4296875 network.sent‚ 9.740491e+06 disk.sda3.in‚0 disk.rbd1.in‚ 0.01171875# proc.memory.rssMB‚ 2943.48828125ÞÏ„”«‚(:‰ Œ÷èÒ°”ªø gpu.0.gpu‚47.0 gpu.0.memory‚15, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚343.177' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚68.6354 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0@‚Õò¬‚(:ü Œ÷èÒä¥Ðø cpu‚25.54964768945554 disk.sda.out‚ 401.515625" disk.nvme0n1.in‚17000.95703125 disk.rbd1.out‚ 20.7646484375 network.sent‚ 9.803681e+06$ disk./.usageGB‚549.2752494812012 disk.rbd0.out‚0 disk.dm-0.in‚17000.95703125* disk./.usagePercent‚36.862322865314965 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.drbd2165.out‚0( disk.nvme0c0n1.out‚78047.76318359375& disk.nvme0n1.out‚78047.76318359375 disk.sda2.out‚0# proc.memory.rssMB‚ 2943.48828125 disk.sda1.in‚0 disk.sda3.in‚0 disk.sda2.in‚0 disk.sda.in‚0 disk.dm-1.in‚0) proc.memory.availableMB‚ 1942.62109375 disk.rbd3.in‚0 disk.rbd0.in‚0# disk.dm-0.out‚78047.76318359375 proc.cpu.threads‚17 disk.sda3.out‚ 401.515625 disk.rbd3.out‚0$ disk.nvme0c0n1.in‚17000.95703125 disk.dm-1.out‚ 401.515625$ memory_percent‚88.14318180084229) proc.memory.percent‚17.96562671661377 network.recv‚ 2.162714e+06 disk.rbd1.in‚ 0.01171875MÛ\ž­‚(:“ ›÷èÒÒÑñö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚265.603' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚53.12060000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0UB»­ ®‚(:þ ›÷èÒÇèŠù$ disk./.usageGB‚549.2821311950684& disk.nvme0n1.out‚78062.45263671875 disk.sda2.out‚0 disk.dm-0.in‚17001.37890625 disk.drbd2165.in‚0 disk.sda2.in‚0 disk.rbd1.in‚ 0.01171875 disk.sda.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0# proc.memory.rssMB‚ 2943.48828125 network.recv‚ 2.166444e+06 disk.rbd3.in‚0 disk.sda3.out‚ 401.8984375 disk.drbd2165.out‚0 disk.rbd1.out‚ 20.7646484375 disk.dm-1.out‚ 401.8984375 disk.rbd3.out‚0$ disk.nvme0c0n1.in‚17001.37890625" disk.nvme0n1.in‚17001.37890625 disk.dm-1.in‚0) proc.memory.percent‚17.96562671661377# disk.dm-0.out‚78062.45263671875 disk.sda1.out‚0 disk.sda3.in‚0$ memory_percent‚88.14475536346436) proc.memory.availableMB‚ 1942.36328125 proc.cpu.threads‚17( disk.nvme0c0n1.out‚78062.45263671875 disk.sda.out‚ 401.8984375 network.sent‚ 9.808687e+06) disk./.usagePercent‚36.86278470290674 disk.sda1.in‚0 cpu‚25.54932575159362 *Êà•¯‚(:Š ª÷èÒð½ôö gpu.0.gpu‚100.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚299.516' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚59.9032 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¶¦Ã" °‚(:€ ª÷èÒ¬å‹ù* disk./.usagePercent‚36.863073223397826 disk.sda3.out‚ 402.20703125% disk.nvme0n1.out‚78070.2080078125 disk.sda.out‚ 402.20703125$ memory_percent‚88.14022541046143) proc.memory.percent‚17.96562671661377$ disk./.usageGB‚549.2864303588867 disk.drbd2165.in‚0 disk.rbd3.in‚0 disk.dm-1.out‚ 402.20703125 disk.rbd0.out‚0 disk.sda2.in‚0 disk.sda.in‚0 disk.dm-1.in‚0 disk.dm-0.in‚17068.79296875 disk.rbd1.in‚ 0.01171875 cpu‚25.549094803141628" disk.dm-0.out‚78070.2080078125 proc.cpu.threads‚17 disk.sda2.out‚0 disk.rbd1.out‚ 20.9052734375 network.sent‚ 9.812837e+06 network.recv‚ 2.168943e+06 disk.sda1.out‚0 disk.rbd3.out‚0" disk.nvme0n1.in‚17068.79296875) proc.memory.availableMB‚ 1943.10546875 disk.rbd0.in‚0 disk.sda1.in‚0 disk.drbd2165.out‚0$ disk.nvme0c0n1.in‚17068.79296875' disk.nvme0c0n1.out‚78070.2080078125# proc.memory.rssMB‚ 2943.48828125 disk.sda3.in‚0ù뇳”±‚(:‰ ¹÷èÒ¡Ÿð÷ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚281.671' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚56.3342 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÂEõƲ‚(:÷ ¹÷èÒ×ôîø disk.rbd1.in‚ 0.01171875 disk.dm-0.out‚78080.58203125$ disk.nvme0c0n1.in‚17069.01171875# proc.memory.rssMB‚ 2943.48828125" disk.nvme0n1.in‚17069.01171875 disk.rbd0.in‚0 network.sent‚ 9.817898e+06 network.recv‚ 2.172183e+06 disk.rbd3.out‚0$ memory_percent‚88.13803195953369* disk./.usagePercent‚36.863546581328634 disk.sda3.out‚ 402.5859375 disk.drbd2165.in‚0 disk.sda2.out‚0 disk.rbd0.out‚0) proc.memory.percent‚17.96562671661377 disk.sda1.out‚0 disk.rbd3.in‚0 disk.sda2.in‚0 disk.rbd1.out‚ 20.9052734375 disk.sda.out‚ 402.5859375 disk.dm-1.in‚0 disk.dm-1.out‚ 402.5859375 disk.dm-0.in‚17069.01171875$ disk./.usageGB‚549.2934837341309 disk.sda3.in‚0 disk.drbd2165.out‚0% disk.nvme0c0n1.out‚78080.58203125# disk.nvme0n1.out‚78080.58203125 disk.sda.in‚0 disk.sda1.in‚0 cpu‚25.548750272349242) proc.memory.availableMB‚ 1943.46484375 proc.cpu.threads‚17ÄêpÉ2³- _wandbruntime‚8615  _runtime‚8615©'d Æ´À trainer/global_step‚5199 % _timestamp‚1.7842984308301728e+09 _runtime‚ 8615.04729462 _step‚424 & lr-AdamW/pg1‚0.0005944572388589014 & lr-AdamW/pg2‚0.0005944572388589014~rK]˵Š_runtime‚ 8615.04729462 _step‚424 & lr-AdamW/pg1‚0.0005944572388589014 & lr-AdamW/pg2‚0.0005944572388589014 trainer/global_step‚5199 % _timestamp‚1.7842984308301728e+09¨XsÛ2¶- _wandbruntime‚8615  _runtime‚8615¯"’ÿ'·ùO H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚44992290 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚409998330048512 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 _step‚425 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 % _timestamp‚1.7842984308311856e+09 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1495860546230477 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚44259637 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4886263268352 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 _runtime‚8615.051937253 trainer/global_step‚5199 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1496513313137664 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14737873 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1496513313137664 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1491627076042752 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚44982317 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚44259637 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚403698786349056 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1495861916258767 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚89251933 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6299509163008 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚44259643 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4877904120013 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4887608657920 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚28072947 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚13334815 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚44992290 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14738127 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1490982667538432 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1490982642110464 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚409997927420928 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1491627076042752 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4886263268352 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 _runtime‚8615.051937253 trainer/global_step‚5199 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1496513313137664 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14737873 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1496513313137664 aLDeviceStatsMonito1Ñ›y|r.on_train_batch_start/allocated_bytes.large_pool.allocated‚1491627076042752 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚44982317 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚44259637 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚403698786349056 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1495861916258767 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚89251933 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6299509163008 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚44259643 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4877904120013 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4887608657920 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚28072947 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚13334815 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚44992290 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14738127 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1490982667538432 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1490982642110464 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚409997927420928 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1491627076042752 IDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚89268631 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚44990795 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚28078231 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 400523264 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _step‚426 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1496792238903296 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚410076617137152 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚410077017660416 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6300744903680 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1496139351474899 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 _runtime‚8616.432363265 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4887188303872 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1491905076772864 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4888533700608 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚45000782 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚403776272756736 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1496793610473472 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1491260549316608 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚242 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1491905050599424 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14740868 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1491260523888640 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚89268631 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚28077984 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 trainer/global_step‚5199 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚44267843 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚45000782 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434J´úÜ'»ˆN J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚44267843 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚45000782 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4878827586259 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13337358 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚89268631 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚44990795 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚28078231 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 400523264 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _step‚426 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1496792238903296 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚410076617137152 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚410077017660416 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6300744903680 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1496139351474899 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 _runtime‚8616.432363265 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4887188303872 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1491905076772864 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4888533700608 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚45000782 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚403776272756736 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1496793610473472 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1491260549316608 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚242 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1491905050599424 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14740868 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1491260523888640 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚89268631 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚28077984 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 trainer/global_step‚5199ªQZ]D2¼- _wandbruntime‚8616  _runtime‚8616uŇf ½ƒ %loss/total_loss‚7.685787357818322 2loss/FBt1/assignment_loss‚0.007452472569049826 2loss/FBt2/assignment_loss‚0.001702537016322413 /loss/FRt2/detection_loss‚1.4514052172136658 _step‚427 0loss/FRt2/assignment_loss‚0.6090767919725063 /loss/FRt1/detection_loss‚0.5550500853286117 0loss/FBt2/detection_loss‚0.39269689758558485 % _timestamp‚1.7842984323016405e+09 _runtime‚ 8616.51857892 epoch‚12 0loss/FRt1/assignment_loss‚1.5946255243918745 1loss/SRqqt2/assignment_loss‚0.504915922326645 1loss/SRqqt1/detection_loss‚0.8210633919889537 1loss/SRqqt2/detection_loss‚0.6498762421618667 /loss/FBt1/detection_loss‚0.5680184850755793 trainer/global_step‚5199 2loss/SRqqt1/assignment_loss‚0.5299037901876615 ýx˜¾ˆ 2loss/FBt1/assignment_loss‚0.007452472569049826 2loss/FBt2/assignment_loss‚0.001702537016322413 /loss/FRt2/detection_loss‚1.4514052172136658 _step‚427 0loss/FRt2/assignment_loss‚0.6090767919725063 /loss/FRt1/detection_loss‚0.5550500853286117 0loss/FBt2/detection_loss‚0.39269689758558485 % _timestamp‚1.7842984323016405e+09 _runtime‚ 8616.51857892 epoch‚12 0loss/FRt1/assignment_loss‚1.5946255243918745 1loss/SRqqt2/assignment_loss‚0.504915922326645 1loss/SRqqt1/detection_loss‚0.8210633919889537 1loss/SRqqt2/detection_loss‚0.6498762421618667 /loss/FBt1/detection_loss‚0.5680184850755793 trainer/global_step‚5199 2loss/SRqqt1/assignment_loss‚0.5299037901876615 %loss/total_loss‚7.685787357818322«h{ Ξ¿‚(:“ È÷èÒ¨òËø gpu.0.gpu‚98.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚333.764' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚66.75280000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¶A/ÀÀ‚(:ý È÷èÒýâÿù) disk./.usagePercent‚36.86399945865671 disk.rbd0.in‚0 disk.sda1.in‚0 disk.sda3.in‚0$ disk.nvme0c0n1.in‚17069.43359375$ memory_percent‚88.13924789428711' proc.memory.availableMB‚ 1943.265625 disk.sda.in‚0# disk.dm-0.out‚78093.59228515625 disk.sda1.out‚0& disk.nvme0n1.out‚78093.59228515625) proc.memory.percent‚17.96562671661377 network.sent‚ 9.823016e+06 disk.dm-1.out‚ 402.7265625 disk.drbd2165.out‚0( disk.nvme0c0n1.out‚78093.59228515625 cpu‚25.550769083057734 network.recv‚ 2.175671e+06 disk.sda.out‚ 402.7265625 disk.rbd0.out‚0 disk.dm-0.in‚17069.43359375 disk.rbd3.in‚0$ disk./.usageGB‚549.3002319335938 disk.sda2.out‚0 proc.cpu.threads‚17 disk.sda2.in‚0 disk.rbd1.out‚ 20.9677734375 disk.sda3.out‚ 402.7265625 disk.rbd3.out‚0" disk.nvme0n1.in‚17069.43359375 disk.dm-1.in‚0 disk.drbd2165.in‚0 disk.rbd1.in‚ 0.01171875# proc.memory.rssMB‚ 2943.48828125ýö xžÁ‚(:“ ×÷èÒ‰ì’÷ gpu.0.gpu‚100.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚329.85' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚65.97000000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚02Ýð‚(:å ×÷èÒ…ÿßû) proc.memory.percent‚17.96562671661377 cpu‚25.555460654067613 disk.dm-1.out‚ 403.03125 disk.rbd0.out‚0 disk.dm-0.out‚ 78104.390625 disk.sda1.out‚0 disk.sda3.out‚ 403.03125 disk.rbd3.out‚0" disk.nvme0n1.in‚17069.63671875 disk.rbd1.out‚ 20.9677734375 disk.sda1.in‚0 disk.rbd3.in‚0 disk.sda.out‚ 403.03125 network.sent‚ 9.88559e+06 disk.sda2.in‚0# proc.memory.rssMB‚ 2943.48828125$ disk./.usageGB‚549.3046340942383$ memory_percent‚88.13762664794922 proc.cpu.threads‚17 network.recv‚ 2.18015e+06 disk.dm-0.in‚17069.63671875 disk.drbd2165.out‚0! disk.nvme0n1.out‚ 78104.390625) disk./.usagePercent‚36.86429489135122 disk.sda3.in‚0 disk.sda2.out‚0 disk.rbd0.in‚0$ disk.nvme0c0n1.in‚17069.63671875 disk.dm-1.in‚0 disk.drbd2165.in‚0# disk.nvme0c0n1.out‚ 78104.390625 disk.rbd1.in‚ 0.01171875 disk.sda.in‚0& proc.memory.availableMB‚ 1943.53125e³Ù;2Ã- _wandbruntime‚8642  _runtime‚8642<Çèº1ÄÑz =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.18043175339698792 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03044423647224903 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.11538461595773697 _runtime‚8642.702984141 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.20297439396381378 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.10659056901931763 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.1934100091457367 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7502436637878418 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.012393198907375336 particle/accuracy_2_of_4‚1 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7870370149612427 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 *jet/accuracy_1_of_4‚0.9694589972496033 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9958943128585815 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8549129962921143 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 +particle/specificity‚0.9071701765060425 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.5735924243927002 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.5715094804763794 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6839272379875183 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.11063550412654877 9"Purity/2FRt0SRqqt*FBt/event_purity‚0.0390494279563427 jet/accuracy_5_of_5‚0 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 +jet/accuracy_3_of_3‚0.07288771122694016 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8888888955116272 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6840091943740845 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5566666722297668 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6269943714141846 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6256366968154907 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6223457455635071 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.2970660924911499 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9931229948997498 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7430022358894348 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.10060611367225647 (particle/accuracy‚0.8497681617736816 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.625 6 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.999627947807312 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3379969000816345 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9181712865829468 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.2033979594707489 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 particle/accuracy_2_of_3‚1 epoch‚12 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.24481603503227234 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8516628742218018 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4365406930446625 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9965608716011047 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 7 Purity/2FRt1SRqqt1FBt/FRt_puritåw/ø&/y‚0.4791666567325592 particle/accuracy_4_of_5‚1 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6283897161483765 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 particle/accuracy_1_of_1‚1 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9960343241691589 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7142857313156128 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2035014033317566 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6233124136924744 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9987789392471313 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6056565046310425 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.1448977142572403 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.23135405778884888 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6269317269325256 particle/accuracy_1_of_3‚1 7validation_average_jet_accuracy‚0.34358078241348267 6 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.942307710647583 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6218229532241821 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.13583333790302277 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8016560673713684 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6431962251663208 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.18112562596797943 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6830226182937622 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7798563838005066 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 +jet/accuracy_2_of_2‚0.13362960517406464 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.11538461595773697 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.6732631325721741 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.6769721508026123 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8016560673713684 *jet/accuracy_1_of_3‚0.9175475239753723 particle/accuracy_5_of_5‚0 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9418967366218567 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.6443848609924316 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5739902853965759 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 $ _timestamp‚1.784298458484908e+09 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8634631037712097 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5546196699142456 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5622521638870239 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 'particle/f_score‚0.6465438008308411 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8988833427429199 9"Purity/*FRt2SRqqt*FBt/event_purity‚0.1329851597547531 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.5728873014450073 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.041462548077106476 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9930664896965027 6 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.193946972489357 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 *jet/accuracy_2_of_3‚0.5202583074569702 particle/accuracy_3_of_5‚1 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9994588494300842 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8590325117111206 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.8125 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.13538360595703125 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5739902853965759 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 particle/accuracy_1_of_2‚1 particle/accuracy_3_of_3‚1 +particle/sensitivity‚0.6398667097091675 trainer/global_step‚5199 8"Purity/0FRt0SRqqt*FBt/event_purity‚0.995245099067688 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2844180464744568 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7346718907356262 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.2889980375766754 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 *jet/accuracy_1_of_2‚0.7736880779266357 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.8571428656578064 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.5685833096504211 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9869047999382019 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.02057807706296444 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.30535969138145447 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4296875 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.1905878782272339 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 8"Purity/1FRt0SRqqt2FBt/event_purity‚0.852985680103302 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.21283137798309326 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9877882599830627 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.1905878782272339 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5751280784606934 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.15641814470291138 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 +jet/accuracy_3_of_4‚0.09510649740695953 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6170641779899597 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9931973218917847 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.18926085531711578 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.34223589301109314 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.10753705352544785 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5737058520317078 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5926783680915833 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9755765199661255 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.1751333326101303 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7004534006118774 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9305555820465088 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.11233457177877426 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3020833134651184 particle/accuracy_1_of_5‚1 /particle/accuracy_3_of_4‚0.9965986609458923 _step‚428 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.2568989396095276 8"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.703903079032898 *jet/accuracy_1_of_1‚0.2959635257720947 *jet/accuracy_2_of_4‚0.8113952875137329 jet/accuracy_3_of_5‚1 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.10060611367225647 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9850649237632751 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.18341058492660522 particle/accuracy_2_of_5‚1 )loss/val_total_loss‚8.217098572369764 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5596802234649658 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8795779943466187 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 particle/accuracy_2_of_2‚1 /particle/accuracy_4_of_4‚0.9436200261116028 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.48249495029449463 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7629496455192566 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 jet/accuracy_1_of_5‚1 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 jet/accuracy_2_of_5‚1 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.1865050196647644 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 6 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.852985680103302 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9966895580291748 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9953312873840332 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.17429372668266296 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.48608407378196716 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.16403530538082123 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.1938541978597641 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.42129629850387573 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 6 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.998015820980072 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.6400446891784668 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6862736344337463 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7870370149612427 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9897873401641846 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9845883250236511 8"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.750041663646698 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.17120686173439026 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6696920990943909 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5460994243621826 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4801265299320221 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.023300400003790855 particle/accuracy_1_of_4‚1 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6354166269302368 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.1788042187690735 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 +jet/accuracy_4_of_4‚0.01628045178949833 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.1970338076353073 jet/accuracy_4_of_5‚0 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344ñ«‰Z\=ÅÖz 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.1788042187690735 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 +jet/accuracy_4_of_4‚0.01628045178949833 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.1970338076353073 jet/accuracy_4_of_5‚0 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.18043175339698792 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03044423647224903 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.11538461595773697 _runtime‚8642.702984141 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.20297439396381378 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.10659056901931763 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.1934100091457367 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7502436637878418 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.012393198907375336 particle/accuracy_2_of_4‚1 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7870370149612427 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 *jet/accuracy_1_of_4‚0.9694589972496033 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9958943128585815 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8549129962921143 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 +particle/specificity‚0.9071701765060425 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.5735924243927002 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.5715094804763794 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6839272379875183 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.11063550412654877 9"Purity/2FRt0SRqqt*FBt/event_purity‚0.0390494279563427 jet/accuracy_5_of_5‚0 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 +jet/accuracy_3_of_3‚0.07288771122694016 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8888888955116272 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6840091943740845 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5566666722297668 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6269943714141846 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6256366968154907 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6223457455635071 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.2970660924911499 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9931229948997498 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7430022358894348 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.10060611367225647 (particle/accuracy‚0.8497681617736816 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.625 6 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.999627947807312 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3379969000816345 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9181712865829468 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.2033979594707489 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 particle/accuracy_2_of_3‚1 epoch‚12 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.24481603503227234 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8516628742218018 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4365406930446625 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9965608716011047 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4791666567325592 particle/accuracy_4_of_5‚1 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6283897161483765 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 particle/accuracy_1_of_1‚1 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9960343241691589 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7142857313156128 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2035014033317566 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6233124136924744 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9987789392471313 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6056565046310425 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.1448977142572403 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.23135405778884888 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6269317269325256 particle/accuracy_1_of_3‚1 7validation_average_jet_accuracy‚0.34358078241348267 6 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.942307710647583 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6218229532241821 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.13583333790302277 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8016560673713684 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6431962251663208 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.18112562596797943 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6830226182937622 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7798563838005066 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 +jet/accuracy_2_of_2‚0.13362960517406464 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.11538461595773697 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.6732631325721741 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.6769721508026123 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8016560673713684 *jet/accuracy_1_of_3‚0.9175475239753723 particle/accuracy_5_of_5‚0 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9418967366218567 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.6443848609924316 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5739902853965759 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 $ _timestamp‚1.784298458484908e+09 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8634631037712097 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5546196699142456 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5622521638870239 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 'particle/f_score‚0.6465438008308411 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8988833427429199 9"Purity/*FRt2SRqqt*FBt/event_purity‚0.1329851597547531 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.5728873014450073 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.041462548077106476 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9930664896965027 6 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.193946972489357 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 *jet/accuracy_2_of_3‚0.5202583074569702 particle/accuracy_3_of_5‚1 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9994588494300842 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8590325117111206 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.8125 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.13538360595703125 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5739902853965759 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 particle/accuracy_1_of_2‚1 particle/accuracy_3_of_3‚1 +particle/sensitivity‚0.6398667097091675 trainer/global_step‚5199 8"Purity/0FRt0SRqqt*FBt/event_purity‚0.995245099067688 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2844180464744568 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7346718907356262 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.2889980375766754 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 *jet/accuracy_1_of_2‚0.7736880779266357 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.8571428656578064 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.5685833096504211 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9869047999382019 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.02057807706296444 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.30535969138145447 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4296875 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.1905878782272339 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 8"Purity/1FRt0SRqqt2FBt/event_purity‚0.852985680103302 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.21283137798309326 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9877882599830627 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.1905878782272339 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5751280784606934 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.15641814470291138 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 +jet/accuracy_3_of_4‚0.09510649740695953 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6170641779899597 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9931973218917847 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.18926085531711578 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.34223589301109314 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.10753705352544785 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5737058520317078 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5926783680915833 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9755765199661255 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.1751333326101303 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7004534006118774 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9305555820465088 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.11233457177877426 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3020833134651184 particle/accuracy_1_of_5‚1 /particle/accuracy_3_of_4‚0.9965986609458923 _step‚428 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.2568989396095276 8"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.703903079032898 *jet/accuracy_1_of_1‚0.2959635257720947 *jet/accuracy_2_of_4‚0.8113952875137329 jet/accuracy_3_of_5‚1 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.10060611367225647 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9850649237632751 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.18341058492660522 particle/accuracy_2_of_5‚1 )loss/val_total_loss‚8.217098572369764 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5596802234649658 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8795779943466187 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 particle/accuracy_2_of_2‚1 /particle/accuracy_4_of_4‚0.9436200261116028 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.48249495029449463 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7629496455192566 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 jet/accuracy_1_of_5‚1 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 jet/accuracy_2_of_5‚1 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.1865050196647644 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 6 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.852985680103302 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9966895580291748 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9953312873840332 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.17429372668266296 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.48608407378196716 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.16403530538082123 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.1938541978597641 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.42129629850387573 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 6 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.998015820980072 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.6400446891784668 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6862736344337463 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7870370149612427 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9897873401641846 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9845883250236511 8"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.750041663646698 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.17120686173439026 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6696920990943909 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5460994243621826 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4801265299320221 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.023300400003790855 particle/accuracy_1_of_4‚1 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6354166269302368¬±V ǞƂ(:“ æ÷èÒÈ—ÿö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚264.47' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚52.894000000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‡¬_(Ç‚(:ù æ÷èÒ£Á¯ù disk.dm-0.in‚17176.63671875$ memory_percent‚88.13924789428711 network.recv‚ 2.18335e+06$ disk./.usageGB‚549.3138580322266" disk.nvme0n1.in‚17176.63671875 disk.rbd1.out‚ 21.0458984375 disk.sda3.in‚0) proc.memory.percent‚17.96562671661377 disk.dm-1.in‚0" disk.dm-0.out‚78112.0380859375 disk.sda3.out‚ 403.2578125 disk.drbd2165.out‚0 disk.sda1.out‚0 disk.rbd0.out‚0 network.sent‚ 9.891205e+06 disk.sda.out‚ 403.2578125 disk.rbd0.in‚0 disk.rbd3.in‚0 disk.rbd3.out‚0$ disk.nvme0c0n1.in‚17176.63671875% disk.nvme0n1.out‚78112.0380859375# proc.memory.rssMB‚ 2943.48828125 disk.rbd1.in‚ 0.01171875 disk.drbd2165.in‚0' disk.nvme0c0n1.out‚78112.0380859375 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda1.in‚0' proc.memory.availableMB‚ 1943.265625 cpu‚25.556013192761423 proc.cpu.threads‚17) disk./.usagePercent‚36.86491391756901 disk.sda.in‚0 disk.dm-1.out‚ 403.25781255ÎLœÈ‚(:‘ õ÷èÒ‚õö gpu.0.gpu‚5.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚329.225' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚65.84500000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0–-ôv É‚(:ÿ õ÷èÒͽ‰ù disk.sda3.out‚ 403.59765625 proc.cpu.threads‚17" disk.dm-0.out‚78126.3525390625' disk.nvme0c0n1.out‚78126.3525390625 disk.sda2.out‚0 disk.sda.in‚0$ memory_percent‚88.13931941986084 disk.sda3.in‚0" disk.nvme0n1.in‚17176.85546875 disk.dm-1.out‚ 403.59765625 network.sent‚ 9.948666e+06$ disk.nvme0c0n1.in‚17176.85546875% disk.nvme0n1.out‚78126.3525390625 disk.rbd1.out‚ 21.0458984375$ disk./.usageGB‚549.3182487487793 disk.rbd1.in‚ 0.01171875# proc.memory.rssMB‚ 2943.48828125 cpu‚25.55574585459647 disk.drbd2165.out‚0 disk.rbd3.in‚0 disk.sda.out‚ 403.59765625) proc.memory.availableMB‚ 1943.25390625 network.recv‚ 2.187946e+06* disk./.usagePercent‚36.865208582240925 disk.dm-0.in‚17176.85546875 disk.rbd3.out‚0 disk.sda2.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0) proc.memory.percent‚17.96562671661377 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.dm-1.in‚0 disk.sda1.in‚0ŽN*-žÊ‚(:“ „øèÒÊÒ†÷ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚275.487' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚55.09740000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ùOÌË‚(:ú „øèÒŽ÷…ù# proc.memory.rssMB‚ 2943.48828125 network.recv‚ 2.191493e+06 disk.rbd1.out‚ 21.1083984375 disk.sda.in‚0 disk.rbd3.out‚0$ disk.nvme0n1.out‚78133.619140625 network.sent‚ 9.95373e+06 disk.sda1.out‚0$ disk.nvme0c0n1.in‚17177.05859375 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-1.in‚0 disk.rbd0.out‚0" disk.nvme0n1.in‚17177.05859375! disk.dm-0.out‚78133.619140625 disk.sda3.out‚ 403.91796875 disk.drbd2165.out‚0 disk.rbd1.in‚ 0.01171875) proc.memory.percent‚17.96562671661377$ disk./.usageGB‚549.3253860473633 disk.rbd3.in‚0) proc.memory.availableMB‚ 1943.23828125 cpu‚25.55542038514111 proc.cpu.threads‚17 disk.sda1.in‚0 disk.sda3.in‚0 disk.sda.out‚ 403.91796875 disk.dm-1.out‚ 403.91796875 disk.rbd0.in‚0$ memory_percent‚88.13941478729248) disk./.usagePercent‚36.86568757233749 disk.drbd2165.in‚0& disk.nvme0c0n1.out‚78133.619140625 disk.dm-0.in‚17177.05859375pÆf”Ì‚(:‰ “øèÒË’úö gpu.0.gpu‚92.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚241.606' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚48.3212 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÉöÊó¹Í‚(:ó “øèÒëÞãù disk.sda2.in‚0# proc.memory.rssMB‚ 2943.48828125 disk.rbd0.out‚0" disk.dm-0.out‚78143.7841796875 disk.rbd3.in‚0$ disk.nvme0c0n1.in‚17 ‡&E177.48046875" disk.nvme0n1.in‚17177.48046875$ disk./.usageGB‚549.3323287963867 disk.sda.out‚ 404.34375 disk.dm-0.in‚17177.48046875 disk.drbd2165.in‚0 disk.drbd2165.out‚0$ memory_percent‚88.14105987548828 proc.cpu.threads‚17% disk.nvme0n1.out‚78143.7841796875 disk.dm-1.out‚ 404.34375 cpu‚25.55514872770217 disk.sda2.out‚0 disk.rbd1.in‚ 0.01171875 disk.rbd1.out‚ 21.1083984375 network.recv‚ 2.195082e+06 disk.rbd0.in‚0 disk.sda3.in‚0& proc.memory.availableMB‚ 1942.96875) proc.memory.percent‚17.96562671661377 network.sent‚ 9.958802e+06 disk.sda.in‚0 disk.dm-1.in‚0 disk.sda1.out‚0 disk.sda3.out‚ 404.34375 disk.rbd3.out‚0' disk.nvme0c0n1.out‚78143.7841796875* disk./.usagePercent‚36.866153506049805 disk.sda1.in‚0zn?ŸÎ‚(:” ¢øèÒ¼Àüö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚269.062' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚53.812400000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Cn(éúÏ‚(:ï ¢øèÒ§²Ùø disk.dm-0.in‚17263.66796875 disk.sda3.in‚0 disk.drbd2165.in‚0 disk.sda2.out‚0) proc.memory.percent‚17.96562671661377 disk.sda.in‚0 disk.rbd0.in‚0 disk.dm-0.out‚ 78156.8515625$ disk.nvme0c0n1.in‚17263.66796875$ disk.nvme0c0n1.out‚ 78156.8515625# proc.memory.rssMB‚ 2943.48828125$ disk./.usageGB‚549.3368034362793 disk.sda1.in‚0 disk.drbd2165.out‚0$ memory_percent‚88.14170360565186 network.sent‚ 9.963817e+06 disk.dm-1.out‚ 404.578125 disk.rbd0.out‚0" disk.nvme0n1.in‚17263.66796875" disk.nvme0n1.out‚ 78156.8515625) proc.memory.availableMB‚ 1942.86328125 proc.cpu.threads‚17) disk./.usagePercent‚36.86645380288747 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.rbd1.out‚ 21.1865234375 network.recv‚ 2.198479e+06 disk.sda.out‚ 404.578125 disk.dm-1.in‚0 disk.sda1.out‚0 disk.sda3.out‚ 404.578125 disk.sda2.in‚0 disk.rbd1.in‚ 0.01171875 cpu‚25.55483163946291 'áª2Ð- _wandbruntime‚8719  _runtime‚8719Y/oÇÑÁ _runtime‚8719.998717597 _step‚429 & lr-AdamW/pg1‚0.0005925209363499975 & lr-AdamW/pg2‚0.0005925209363499975 trainer/global_step‚5249 % _timestamp‚1.7842985357815561e+09ÍÙ«#ÌÒÆ & lr-AdamW/pg1‚0.0005925209363499975 & lr-AdamW/pg2‚0.0005925209363499975 trainer/global_step‚5249 % _timestamp‚1.7842985357815561e+09 _runtime‚8719.998717597 _step‚429­Ô+12Ó- _wandbruntime‚8720  _runtime‚8720åûÜ<ÿ'ÔùO C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14910742 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚239 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1508702665379840 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚407621218979840 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚407620850888704 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 trainer/global_step‚5249 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1509355093983232 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4939767200256 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 % _timestamp‚1.7842985357826345e+09 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚13470883 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1509355093983232 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4929970364558 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6367160335360 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚45467051 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚32439296 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1513632610316430 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1514294861183488 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚90268994 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1514293489620480 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1508702639951872 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚28381391 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1513633980344720 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4939767200256 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 _step‚430 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚28381630 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1514293489620480 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚44801943 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚45477024 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1509355067809792 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 _runtime‚8719.999984877 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚45467051 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚234 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4938421810688 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚44801943 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4931314964880 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14910508 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚413988411754496 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0È£Í(ÕþO 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚44801949 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6367192774656 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4938421810688 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚44801943 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4931314964880 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14910508 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚413988411754496 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14910742 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚239 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1508702665379840 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚407621218979840 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚407620850888704 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 trainer/global_step‚5249 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1509355093983232 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4939767200256 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 % _timestamp‚1.7842985357826345e+09 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚13470883 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1509355093983232 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4929970364558 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6367160335360 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚45467051 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚32439296 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1513632610316430 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1514294861183488 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚90268994 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1514293489620480 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1508702639951872 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚28381391 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1513633980344720 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4939767200256 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 _step‚430 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚28381630 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1514293489620480 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚44801943 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚45477024 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1509355067809792 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 _runtime‚8719.999984877 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚45467051 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚234 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1509633068539904 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚90295671 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1509633094713344 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚414066711589888 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1509633094713344 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚44810155 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚45475529 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 400523264 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚45485516 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚45475529 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚28386742 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚407698337296384 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚44810149 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4939346846208 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚229 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚407698705387520 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1513912785589254 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1508980547158016 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚224 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚90295671 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚32432128 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚44810149 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚28386971 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1513911415560852 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚90285678 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1508980521730048 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14913540 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6368374293504 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4930893830804 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1509633068539904 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4939346846208 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4932238431238 $ _timestamp‚1.784298537169263e+09 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6368406725632 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1514572415386112 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚90285678 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4940692242944 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _step‚431 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 QBDeviceStatsMonitor.on_train_batch_end/activeȆ‰g^_bytes.small_pool.peak‚ 1680116736 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚44810155 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚13473431 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚45485516 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736Ò_‘ '؆N BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4939346846208 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚229 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚407698705387520 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1513912785589254 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1508980547158016 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚224 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚90295671 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚32432128 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚44810149 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚28386971 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1513911415560852 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚90285678 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1508980521730048 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14913540 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6368374293504 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4930893830804 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1509633068539904 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4939346846208 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4932238431238 $ _timestamp‚1.784298537169263e+09 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6368406725632 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1514572415386112 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚90285678 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4940692242944 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _step‚431 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚44810155 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚13473431 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚45485516 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1509633068539904 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚90295671 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1509633094713344 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚414066711589888 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1509633094713344 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚44810155 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚45475529 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 400523264 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚45485516 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚45475529 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚28386742 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚407698337296384 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚44810149¯¦êmG2Ù- _wandbruntime‚8721  _runtime‚8721¬ß†jÚ /loss/FRt1/detection_loss‚0.5405509442183971 1loss/SRqqt2/detection_loss‚0.5862335013079958 /loss/FBt1/detection_loss‚0.6164101433974726 /loss/FBt2/detection_loss‚0.4919781537912086 trainer/global_step‚5249 0loss/FRt2/assignment_loss‚0.6063255320868504 2loss/SRqqt2/assignment_loss‚0.4237866881169317 0loss/FBt2/assignment_loss‚0.0162175166483461 .loss/FRt2/detection_loss‚1.496981323361209 1loss/SRqqt1/detection_loss‚0.9785772928842466 _step‚432 2loss/SRqqt1/assignment_loss‚0.5595778623086753 _runtime‚8721.473113353 2loss/FBt1/assignment_loss‚0.022175107821025126 %loss/total_loss‚7.952676414075535 epoch‚13 % _timestamp‚1.7842985372561495e+09 0loss/FRt1/assignment_loss‚1.6138623481331758Rsgq Û† 2loss/FBt1/assignment_loss‚0.022175107821025126 %loss/total_loss‚7.952676414075535 epoch‚13 % _timestamp‚1.7842985372561495e+09 0loss/FRt1/assignment_loss‚1.6138623481331758 /loss/FRt1/detection_loss‚0.5405509442183971 1loss/SRqqt2/detection_loss‚0.5862335013079958 /loss/FBt1/detection_loss‚0.6164101433974726 /loss/FBt2/detection_loss‚0.4919781537912086 trainer/global_step‚5249 0loss/FRt2/assignment_loss‚0.6063255320868504 2loss/SRqqt2/assignment_loss‚0.4237866881169317 0loss/FBt2/assignment_loss‚0.0162175166483461 .loss/FRt2/detection_loss‚1.496981323361209 1loss/SRqqt1/detection_loss‚0.9785772928842466 _step‚432 2loss/SRqqt1/assignment_loss‚0.5595778623086753 _runtime‚8721.473113353°EP“Ü‚(:ˆ ±øèÒæŽÓø gpu.0.gpu‚91.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚289.215' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚57.843 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0©”¯4ëÝ‚(:à ±øèÒ£·õù" disk.nvme0n1.in‚17263.87109375 disk.sda2.in‚0 disk.rbd0.out‚0 disk.sda1.out‚0) proc.memory.availableMB‚ 1942.75390625 network.sent‚ 9.968879e+06 disk.rbd1.in‚ 0.01171875 disk.dm-1.in‚0 disk.sda1.in‚0) proc.memory.percent‚17.96562671661377$ disk.nvme0n1.out‚78167.365234375 disk.sda2.out‚0 disk.rbd0.in‚0 disk.dm-0.in‚17263.87109375 disk.drbd2165.in‚0 disk.rbd3.in‚0 proc.cpu.threads‚17 network.recv‚ 2.202132e+06) disk./.usagePercent‚36.86690309611006$ disk./.usageGB‚549.3434982299805# proc.memory.rssMB‚ 2943.48828125 cpu‚25.55458749591522$ disk.nvme0c0n1.in‚17263.87109375 disk.dm-1.out‚405 disk.sda.out‚405! disk.dm-0.out‚78167.365234375& disk.nvme0c0n1.out‚78167.365234375 disk.rbd1.out‚ 21.1865234375 disk.sda3.in‚0 disk.drbd2165.out‚0 disk.rbd3.out‚0 disk.sda.in‚0 disk.sda3.out‚405$ memory_percent‚88.14237117767334“Nˆ-žÞ‚(:“ ÀøèÒý¿Ùö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚292.743' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚58.54859999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0·˜ÞR ß‚(:€ ÀøèÒÊ÷öø disk.sda2.in‚0$ memory_percent‚88.14558982849121 disk.dm-1.out‚ 405.12890625 disk.rbd0.out‚0# disk.dm-0.out‚78176.87548828125 disk.rbd3.in‚0 disk.rbd1.in‚ 0.01171875# proc.memory.rssMB‚ 2943.48828125 disk.rbd0.in‚0 disk.sda1.out‚0 disk.rbd3.out‚0$ disk.nvme0c0n1.in‚17264.29296875( proc.memory.availableMB‚ 1942.2265625 proc.cpu.threads‚17 disk.sda.in‚0 disk.dm-1.in‚0 disk.sda3.in‚0 disk.rbd1.out‚ 21.3115234375 cpu‚25.55427195620659 network.sent‚ 1.0031324e+07$ disk./.usageGB‚549.3505325317383" disk.nvme0n1.in‚17264.29296875) proc.memory.percent‚17.96562671661377 disk.dm-0.in‚17264.29296875( disk.nvme0c0n1.out‚78176.87548828125 disk.sda2.out‚0 disk.sda.out‚ 405.12890625 disk.sda1.in‚0 disk.drbd2165.out‚0& disk.nvme0n1.out‚78176.87548828125 network.recv‚ 2.206365e+06( disk./.usagePercent‚36.8673751740032 disk.sda3.out‚ 405.12890625 disk.drbd2165.in‚0 ï kà‚(:’ ÏøèÒ‹µ÷ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚283.03' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚56.60599999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ìTˆ-á‚(:ö ÏøèÒÏîëø! disk.dm-0.out‚78193.123046875 disk.rbd3.in‚0$ disk.nvme0n1.out‚78193.123046875 disk.sda.in‚0 network.recv‚ 2.210096e+06 disk.sda2.in‚0 disk.rbd1.out‚ 21.3115234375 disk.sda1.in‚0$ disk.nvme0c0n1.in‚17264.51171875) proc.memory.availableMB‚ 1942.21484375 proc.cpu.threads‚17( disk./.usagePercent‚36.8670917736628 disk.dm-1.out‚ 405.359375 disk.rbd0.in‚0 network.sent‚ 1.0036958e+07$ disk./.usageGB‚549.3463096618652 disk.rbd0.out‚0 disk.sda1.out‚0$ memory_percent‚88.14566135406494" disk.nvme0n1.in‚17264.51171875 disk.dm-1.in‚0 disk.drbd2165.in‚0 disk.drbd2165.out‚0 cpu‚25.553980567910408 disk.sda2.out‚0 disk.rbd1.in‚ 0.01171875 disk.dm-0.in‚17264.51171875# proc.memory.rssMB‚ 2943.48828125) proc.memory.percent‚17.96562671661377 disk.sda.out‚ 405.359375 disk.sda3.in‚0 disk.sda3.out‚ 405.359375 disk.rbd3.out‚0& disk.nvme0c0n1.out‚78193.123046875¯Zê”â‚(:‰ ÞøèÒÂáš÷ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚38.0 gpu.0.powerWatts‚110.527' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.1054 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚00©üêøã‚(:í ÞøèÒòóèø network.sent‚ 1.004203e+07) disk./.usagePercent‚36.86755693935251 disk.dm-1.in‚0" disk.dm-0.out‚78200.8896484375 disk.sda3.out‚ 405.78125$ memory_percent‚88.14575672149658# proc.memory.rssMB‚ 2943.48828125 disk.rbd3.out‚0) proc.memory.availableMB‚ 1942.19921875 network.recv‚ 2.213494e+06$ disk./.usageGB‚549.3532409667969% disk.nvme0n1.out‚78200.8896484375 disk.rbd1.in‚ 0.01171875 disk.sda3.in‚0 disk.sda.out‚ 405.78125 disk.dm-1.out‚ 405.78125 disk.sda1.out‚0 disk.drbd2165.in‚0! disk.nvme0c0n1.in‚ 17308.71875 disk.nvme0n1.in‚ 17308.71875 disk.rbd0.out‚0 cpu‚25.553661353701965 disk.dm-0.in‚ 17308.59375) proc.memory.percent‚17.96562671661377 proc.cpu.threads‚17 disk.sda2.in‚0 disk.rbd1.out‚ 21.3896484375 disk.sda.in‚0 disk.rbd0.in‚0' disk.nvme0c0n1.out‚78200.8896484375 disk.sda2.out‚0 disk.sda1.in‚0 disk.drbd2165.out‚0 disk.rbd3.in‚0ek’!”ä‚(:‰ íøèÒá®ýö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚270.729' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.1458 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ÿ „å‚(:û íøèÒ” ¤ù disk.sda1.in‚0 cpu‚25.55339699356968 disk.sda1.out‚0 disk.sda3.in‚0 disk.rbd3.out‚0 disk.drbd2165.in‚0# proc.memory.rssMB‚ 2943.48828125) proc.memory.percent‚17.96562671661377 network.sent‚ 1.0047185e+07" disk.dm-0.out‚78214.4208984375$ memory_percent‚88.14444541931152 proc.cpu.threads‚17) disk./.usagePercent‚36.86805077788606 disk.rbd0.in‚0 disk.rbd0.out‚0' disk.nvme0c0n1.out‚78214.4208984375% disk.nvme0n1.out‚78214.4208984375 network.recv‚ 2.217033e+06 disk.dm-0.in‚ 17359.1484375! disk.nvme0n1.in‚ 17359.1484375 disk.sda2.out‚0 disk.sda.out‚ 406.44140625# disk.nvme0c0n1.in‚ 17359.1484375 disk.sda2.in‚0$ disk./.usageGB‚549.3605995178223 disk.sda3.out‚ 406.44140625 disk.drbd2165.out‚0 disk.rbd3.in‚0 disk.rbd1.in‚ 0.01171875 disk.rbd1.out‚ 21.3896484375( proc.memory.availableMB‚ 1942.4140625 disk.sda.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 406.44140625‹#)²2æ- _wandbruntime‚8798  _runtime‚8798º)›»ÇçÁ & lr-AdamW/pg1‚0.0005905687953951876 & lr-AdamW/pg2‚0.0005905687953951876 trainer/global_step‚5299 % _timestamp‚1.7842986142911003e+09 _runtime‚8798.508275264 _step‚433@‹5ÌèÆ % _timestamp‚1.7842986142911003e+09 _runtime‚8798.508275264 _step‚433 & lr-AdamW/pg1‚0.0005905687953951876 & lr-AdamW/pg2‚0.0005905687953951876 trainer/global_step‚5299±n¤ 2é- _wandbruntime‚8798  _runtime‚8798¨˜Ñÿ'êùO LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1528239842056704 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _step‚434 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 % _timestamp‚1.7842986142921524e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15048839 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚417922330744832 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _runtime‚8798.512948465 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚274 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4986083130880 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚91105046 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚45892803 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4984737741312 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1527574306366628 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6427159472128 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1522596728860672 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1528241213619712 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚269 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4977552077988 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6427196105728 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1522596754288640 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚45212249 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1528241213619712 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1523255130488832 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚45902776 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1528239842056704 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚28647146 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚45212243 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚91115025 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1527572936338338 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4976207477666 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚45212243 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚13598038 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚15049108 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚417922735469568 trainer/global_step‚5299 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚411495539363840 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1523255130488832 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404724736 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚45892803 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440÷T/ÕÝ ëþO QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚28647146 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚45212243 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚91115025 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1527572936338338 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4976207477666 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚45212243 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚13598038 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚15049108 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚417922735469568 trainer/global_step‚5299 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚411495539363840 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1523255130488832 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404724736 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚45892803 TEDeviceStatsMonitor.on_train_batch_start/inactive_spl®ÁÙˆ'it_bytes.all.peak‚ 1341808128 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1528239842056704 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _step‚434 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 % _timestamp‚1.7842986142921524e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15048839 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚417922330744832 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _runtime‚8798.512948465 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚274 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4986083130880 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚91105046 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚45892803 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4984737741312 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1527574306366628 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6427159472128 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1522596728860672 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1528241213619712 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚269 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4977552077988 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6427196105728 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1522596754288640 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚45212249 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1528241213619712 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1523255130488832 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚45902776 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1528239842056704²"oÄ2ì- _wandbruntime‚8799  _runtime‚87999åƒ 'íƒN G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚91131723 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚45901281 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1523533105045504 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚418001405750272 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1527851741582760 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1528518767822336 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚91121730 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚28652445 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚411572657680384 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚13600581 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1523533131218944 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚45220455 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13600576 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 _runtime‚8799.896737941 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚45220449 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1527853111611162 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1522874610638848 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4977130943912 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚45220449 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4987008173568 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1523533131218944 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚91131723 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36626432 % _timestamp‚1.7842986156793482e+09 _step‚435 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15051605 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6428379978752 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚28652181 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1522874636066816 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4985662776832 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚45220455 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1523533105045504 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4985662776832 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚411573025771520 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚418001001032704 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6428343352320 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15051864 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404717568 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1528520139392512 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚91121730 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚45901281 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4987008173568 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4978475544346 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚45911268 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119çDŽF'îˆN ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15051864 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404717568 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1528520139392512 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚91121730 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚45901281 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4987008173568 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4978475544346 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚45911268 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚91131723 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚45901281 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1523533105045504 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚418001405750272 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1527851741582760 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1528518767822336 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚91121730 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚28652445 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚411572657680384 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚13600581 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1523533131218944 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚45220455 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13600576 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 _runtime‚8799.896737941 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚45220449 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1527853111611162 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1522874610638848 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4977130943912 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚45220449 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4987008173568 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1523533131218944 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚91131723 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36626432 % _timestamp‚1.7842986156793482e+09 _step‚435 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15051605 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6428379978752 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚28652181 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1522874636066816 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4985662776832 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚45220455 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1523533105045504 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4985662776832 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚411573025771520 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚418001001032704 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6428343352320 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640³VjöŽ2ï- _wandbruntime‚8799  _runtime‚8799ëË=ð 0loss/FRt1/assignment_loss‚1.6436468725767055 epoch‚13 trainer/global_step‚5299 2loss/FBt1/assignment_loss‚0.003850863063953674 .loss/FRt2/detection_loss‚1.492236903124482 % _timestamp‚1.7842986157648113e+09 2loss/SRqqt1/assignment_loss‚0.5230965962837456 2loss/SRqqt2/assignment_loss‚0.4466025986080964 /loss/FRt1/detection_loss‚0.5311180120911086 1loss/SRqqt1/detection_loss‚0.7999157977885887 /loss/FBt1/detection_loss‚1.2986813947558402 %loss/total_loss‚8.569401789180407 _runtime‚ 8799.98179023 0loss/FRt2/assignment_loss‚0.6179622567025367 2loss/FBt2/assignment_loss‚0.005934614254255585 1loss/SRqqt2/detection_loss‚0.5800432567678424 .loss/FBt2/detection_loss‚0.626312623163252 _step‚436‡ÂP ñ† % _timestamp‚1.7842986157648113e+09 2loss/SRqqt1/assignment_loss‚0.5230965962837456 2loss/SRqqt2/assignment_loss‚0.4466025986080964 /loss/FRt1/detection_loss‚0.5311180120911086 1loss/SRqqt1/detection_loss‚0.7999157977885887 /loss/FBt1/detection_loss‚1.2986813947558402 %loss/total_loss‚8.569401789180407 _runtime‚ 8799.98179023 0loss/FRt2/assignment_loss‚0.6179622567025367 2loss/FBt2/assignment_loss‚0.005934614254255585 1loss/SRqqt2/detection_loss‚0.5800432567678424 .loss/FBt2/detection_loss‚0.626312623163252 _step‚436 0loss/FRt1/assignment_loss‚1.6436468725767055 epoch‚13 trainer/global_step‚5299 2loss/FBt1/assignment_loss‚0.003850863063953674 .loss/FRt2/detection_loss‚1.492236903124482´8¿ÛŸò‚(:” üøèÒœ‚ž÷ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚38.0 gpu.0.powerWatts‚111.772' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.354400000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0£òÕŒþó‚(:ó üøèÒ£¬¨ù network.recv‚ 2.220525e+06 disk.sda2.out‚0( proc.memory.availableMB‚ 1942.3515625) proc.memory.percent‚17.96562671661377$ disk./.usageGB‚549.4209747314453 disk.sda3.in‚0$ disk.nvme0c0n1.out‚ 78244.0859375 network.sent‚ 1.0052187e+07 disk.dm-1.out‚ 409.74609375 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.rbd3.in‚0 disk.sda.in‚0) disk./.usagePercent‚36.87210260913048 disk.rbd3.out‚0 disk.sda2.in‚0 disk.rbd1.out‚ 21.4521484375# proc.memory.rssMB‚ 2943.48828125 cpu‚25.553136206770635 disk.sda.out‚ 409.74609375 disk.dm-0.out‚ 78244.0859375$ memory_percent‚88.14482688903809 proc.cpu.threads‚17 disk.rbd1.in‚ 0.01171875 disk.dm-0.in‚ 17359.3671875 disk.sda1.in‚0 disk.sda3.out‚ 409.74609375 disk.drbd2165.in‚0 disk.drbd2165.out‚0# disk.nvme0c0n1.in‚ 17359.3671875! disk.nvme0n1.in‚ 17359.3671875 disk.dm-1.in‚0 disk.sda1.out‚0" disk.nvme0n1.out‚ 78244.0859375q™ú8Ÿô‚(:” ‹ùèÒ½ˆàö gpu.0.gpu‚89.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚298.392' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚59.678399999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0„>÷Êõ‚(:÷ ‹ùèÒþèÆù$ disk.nvme0n1.out‚78260.986328125 disk.dm-1.out‚ 410.8671875# proc.memory.rssMB‚ 2943.48828125 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda3.out‚ 410.8671875 disk.sda2.out‚0! disk.dm-0.out‚78260.986328125 disk.sda3.in‚0 proc.cpu.threads‚17 network.sent‚ 1.0115385e+07 disk.rbd1.out‚ 21.4521484375 disk.dm-1.in‚0$ memory_percent‚88.14480304718018 cpu‚25.552818541477567 network.recv‚ 2.225304e+06& disk.nvme0c0n1.out‚78260.986328125! disk.nvme0n1.in‚ 17359.5703125 disk.sda2.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.dm-0.in‚ 17359.5703125 disk.drbd2165.out‚0 disk.rbd3.in‚0# disk.nvme0c0n1.in‚ 17359.5703125 disk.rbd1.in‚ 0.01171875 disk.sda.in‚0 disk.drbd2165.in‚0 disk.rbd3.out‚0 disk.sda.out‚ 410.8671875) proc.memory.availableMB‚ 1942.35546875) proc.memory.percent‚17.96562671661377) disk./.usagePercent‚36.87009423002437$ disk./.usageGB‚549.39104843139659;Ž”ö‚(:‰ šùèÒ­Šîö gpu.0.gpu‚92.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚326.524' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚65.3048 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0²ÝÂ÷‚(:ÿ šùèÒ•¥„ù nÑMìetwork.sent‚ 1.0120447e+07 network.recv‚ 2.228828e+06 disk.dm-1.out‚ 413.65234375$ disk./.usageGB‚549.4100761413574 disk.rbd1.in‚ 0.01171875 disk.sda.in‚0 disk.sda3.out‚ 413.65234375 disk.drbd2165.in‚0) proc.memory.percent‚17.96562671661377 cpu‚25.552562804135718 disk.rbd1.out‚ 21.5302734375 disk.dm-0.in‚ 17359.9921875# disk.dm-0.out‚78274.66259765625 disk.sda3.in‚0 disk.drbd2165.out‚0! disk.nvme0n1.in‚ 17359.9921875 disk.sda2.out‚0 disk.dm-1.in‚0 disk.sda1.in‚0 disk.rbd3.in‚0 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.rbd3.out‚0& disk.nvme0n1.out‚78274.66259765625$ memory_percent‚88.14492225646973# proc.memory.rssMB‚ 2943.48828125) disk./.usagePercent‚36.87137119560516 disk.sda.out‚ 413.65234375( disk.nvme0c0n1.out‚78274.66259765625 disk.sda2.in‚0 disk.sda1.out‚0# disk.nvme0c0n1.in‚ 17359.9921875( proc.memory.availableMB‚ 1942.3359375ɤ”ø‚(:‰ ©ùèÒ¶óóö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚282.783' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚56.5566 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0·ZœÍ ù‚(:þ ©ùèÒÓÿ˜ù disk.rbd1.in‚ 0.01171875 disk.dm-1.out‚ 414.1015625 network.recv‚ 2.232265e+06) disk./.usagePercent‚36.87189498702021 disk.drbd2165.in‚0 disk.rbd3.out‚0 disk.dm-1.in‚0( proc.memory.availableMB‚ 1942.0859375 disk.drbd2165.out‚0 disk.sda2.in‚0 disk.rbd1.out‚ 21.5302734375 disk.rbd0.in‚0 disk.dm-0.in‚17499.63671875 cpu‚25.55224760139035$ disk.nvme0c0n1.in‚17499.63671875" disk.nvme0n1.in‚17499.63671875 disk.sda2.out‚0) proc.memory.percent‚17.96562671661377$ disk./.usageGB‚549.4178810119629 disk.sda3.out‚ 414.1015625 disk.sda3.in‚0 disk.sda.in‚0# disk.dm-0.out‚78293.57177734375$ memory_percent‚88.14644813537598# proc.memory.rssMB‚ 2943.48828125 disk.rbd3.in‚0 disk.sda.out‚ 414.1015625 disk.rbd0.out‚0 proc.cpu.threads‚17( disk.nvme0c0n1.out‚78293.57177734375 network.sent‚ 1.0125464e+07 disk.sda1.in‚0 disk.sda1.out‚0& disk.nvme0n1.out‚78293.571777343759Æ4e”ú‚(:‰ ¸ùèÒ»Çýö gpu.0.gpu‚91.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚46.0 gpu.0.powerWatts‚340.338' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚68.0676 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÄQ¶û‚(:ý ¸ùèÒ‹Øšù disk.rbd0.out‚0 disk.sda2.in‚0 proc.cpu.threads‚17 disk.sda.in‚0 disk.sda1.in‚0% proc.memory.availableMB‚ 1941.9375 disk.dm-0.in‚17499.83984375 disk.sda3.out‚ 414.39453125 disk.rbd3.in‚0$ disk.nvme0c0n1.in‚17499.83984375$ memory_percent‚88.14735412597656 disk.rbd1.out‚ 21.5927734375( disk.nvme0c0n1.out‚78322.42822265625) proc.memory.percent‚17.96562671661377 cpu‚25.55199058259757 network.recv‚ 2.23485e+06$ disk./.usageGB‚549.4241142272949 disk.dm-1.in‚0 disk.sda1.out‚0) disk./.usagePercent‚36.87231330333116 disk.dm-1.out‚ 414.39453125# disk.dm-0.out‚78322.42822265625 disk.rbd3.out‚0 disk.sda2.out‚0 disk.drbd2165.out‚0& disk.nvme0n1.out‚78322.42822265625 network.sent‚ 1.0130457e+07 disk.rbd1.in‚ 0.01171875 disk.sda.out‚ 414.39453125 disk.sda3.in‚0 disk.drbd2165.in‚0" disk.nvme0n1.in‚17499.83984375# proc.memory.rssMB‚ 2943.48828125 disk.rbd0.in‚0|Ù82ü- _wandbruntime‚8877  _runtime‚8877Ã9ÆýÀ trainer/global_step‚5349 $ _timestamp‚1.784298693164192e+09 _runtime‚8877.381354105 _step‚437 & lr-AdamW/pg1‚0.0005886009413768375 & lr-AdamW/pg2‚0.0005886009413768375 ÿ‡ËþÅ & lr-AdamW/pg2‚0.0005886009413768375 trainer/global_step‚5349 $ _timestamp‚1.784298693164192e+09 _runtime‚8877.381354105 _step‚437 & lr-AdamW/pg1‚0.0005886009413768375µ®ƒr2ÿ- _wandbruntime‚8877  _runtime‚8877­j ÿ'€ùO [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5032399061504 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1536490817769472 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 % _timestamp‚1.7842986931652648e+09 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5031053671936 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚415369491656704 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚45622543 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6486211062272 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1541514632388536 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1537155140820992 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚45622549 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1536490843197440 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1542186194492928 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1542186194492928 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1537155166994432 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚46318555 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1537155140820992 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚46318555 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5022444590774 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚45622543 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚292 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚13725188 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚13725183 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚421856109540864 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚28912482 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚421855702718976 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚46328528 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1537155166994432 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚15187294 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 406821888 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5032399061504 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1542187566055936 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚46328528 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640vŠ:ˆ(þO JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚46318555 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5022444590774 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚45622543 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚292 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚13725188 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚13725183 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚421856109540864 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚28912482 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚421855702718976 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚46328528 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1537155166994432 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚15187294 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 406821888 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5032399061504 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1542187566055936 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚46328528 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5032399061504 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1536490817769472 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 % _timestamp‚1.7842986931652648e+09 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5031053671936 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚415369491656704 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚45622543 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6486211062272 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1541514632388536 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1537155140820992 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚45622549 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1536490843197440 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1542186194492928 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1542186194492928 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1537155166994432 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚46318555 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1537155140820992 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1542466491828736 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚91967775 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚45630749 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚421934366612992 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5031978707456 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚91957782 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚45630749 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 406814720 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚415447346155520 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚91967775 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚421934773427712 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚28917460 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6487427272192 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6487388548608 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1542466491828736 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚13727731 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 JDDeviceStatsMonitor.on_train_batch_end/oversize_à$pÀ³allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1536768724975616 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 % _timestamp‚1.7842986945527477e+09 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚46337020 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚28917738 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚38723584 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5024712657454 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5033324104192 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚273 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚415446978064384 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1537433141551104 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1541793437633070 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5031978707456 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚278 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15190007 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175š ¢Ã'„ˆN E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚46327033 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚91967775 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚421934773427712 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚28917460 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6487427272192 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6487388548608 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1542466491828736 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚13727731 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1536768724975616 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 % _timestamp‚1.7842986945527477e+09 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚46337020 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚28917738 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚38723584 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5024712657454 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5033324104192 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚273 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚415446978064384 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1537433141551104 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1541793437633070 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5031978707456 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚278 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15190007 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚91957782 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚45630755 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5023368057020 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1542466491828736 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚91967775 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚45630749 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚421934366612992 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5031978707456 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚91957782 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚45630749 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 406814720 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚415447346155520 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚29176546 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚425789361982976 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1551055177326592 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚92787129 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1556132546929152 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5077369602560 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚287 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 % _timestamp‚1.7842987724433434e+09 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚425788955161088 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚419243812040704 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1556132546929152 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚13852333 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1550384906678272 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚92777150 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚46032849 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5078714992128 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5070026304204 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚46744307 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚282 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚46032843 _step‚442 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6545181851136 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6545143120384 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 406821888 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1556133918492160 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 _runtime‚8956.660921112 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5068681703882 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚46032843 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚46754280 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚46754280 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1551055177326592 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚46032849 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚29176546 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚425789361982976 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1551055177326592 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚92787129 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1556132546929152 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5077369602560 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚287 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 % _timestamp‚1.7842987724433434e+09 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚425788955161088 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚419243812040704 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1556132546929152 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚13852333 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1550384906678272 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚92777150 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚46032849 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5078714992128 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5070026304204 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚46744307 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚282 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚46032843 _step‚442 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6545181851136 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6545143120384 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 406821888 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1556133918492160 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 _runtime‚8956.660921112 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5068681703882 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚46032843 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚46754280 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚46754280 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1551055177326592 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚46032849 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 I8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1550662788456448 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5070949770562 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1556412844264960 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚276 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1555733763654978 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1551333204230144 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6546325190144 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚46041055 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚46752785 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 _step‚443 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5079640034816 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13854876 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 _runtime‚8958.059286277 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1551333204230144 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚29181514 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚46752785 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1551333178056704 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚92803827 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚271 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚46041049 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15326909 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1556412844264960 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5078294638080 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚29181790 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚419321666539520 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚92793834 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚13854881 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1555732393626576 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5069605170128 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚419321298448384 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5079640034816 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚46762772 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚425867623638528 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚425868030453248 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚46041055 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640œš}±'œˆN \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚425868030453248 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚46041055 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1550662788456448 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5070949770562 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1556412844264960 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚276 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1555733763654978 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1551333204230144 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6546325190144 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚46041055 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚46752785 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 _step‚443 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5079640034816 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13854876 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 _runtime‚8958.059286277 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1551333204230144 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚29181514 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚46752785 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1551333178056704 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚92803827 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚271 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚46041049 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15326909 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1556412844264960 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5078294638080 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚29181790 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚419321666539520 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚92793834 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚13854881 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1555732393626576 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5069605170128 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚419321298448384 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5079640034816 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚46762772 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚425867623638528 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0»îÕóø2- _wandbruntime‚8958  _runtime‚8958·5¯ž‚ % _timestamp‚1.7842987739222465e+09 _runtime‚8958.139408191 2loss/SRqqt2/assignment_loss‚0.4861438863367106 2loss/FBt2/assignment_loss‚0.011227028575335364 /loss/FBt2/detection_loss‚0.4977554454839319 %loss/total_loss‚7.996988611133877 epoch‚13 _step‚444 1loss/FBt1/assignment_loss‚0.20531999485440422 /loss/FRt2/detection_loss‚1.4762644967973304 1loss/SRqqt2/detection_loss‚0.5420444158417513 /loss/FBt1/detection_loss‚0.5689625779050402 trainer/global_step‚5399 1loss/SRqqt1/assignment_loss‚0.586253549732279 1loss/SRqqt1/detection_loss‚0.8056882574760622 0loss/FRt1/assignment_loss‚1.6362070875292978 0loss/FRt2/assignment_loss‚0.6546839445686796 /loss/FRt1/detection_loss‚0.5264379260330544í¬e< Ÿ‡ _step‚444 1loss/FBt1/assignment_loss‚0.20531999485440422 /loss/FRt2/detection_loss‚1.4762644967973304 1loss/SRqqt2/detection_loss‚0.5420444158417513 /loss/FBt1/detection_loss‚0.5689625779050402 trainer/global_step‚5399 1loss/SRqqt1/assignment_loss‚0.586253549732279 1loss/SRqqt1/detection_loss‚0.8056882574760622 0loss/FRt1/assignment_loss‚1.6362070875292978 0loss/FRt2/assignment_loss‚0.6546839445686796 /loss/FRt1/detection_loss‚0.5264379260330544 % _timestamp‚1.7842987739222465e+09 _runtime‚8958.139408191 2loss/SRqqt2/assignment_loss‚0.4861438863367106 2loss/FBt2/assignment_loss‚0.011227028575335364 /loss/FBt2/detection_loss‚0.4977554454839319 %loss/total_loss‚7.996988611133877 epoch‚13¼ÓŽð ‚(:‰ ¡úèÒ¾œäö gpu.0.gpu‚99.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚336.569' gpu.0.enforcedPowerLimitWattsþmÈÀ¤‚500.0 gpu.0.powerPercent‚67.3138 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ï§G¡‚(:ù ¡úèÒ¹å”ù disk.rbd3.out‚0 disk.sda3.in‚0 disk.sda.in‚0# disk.dm-0.out‚78440.55224609375 disk.sda1.out‚0 disk.sda3.out‚ 428.84765625 disk.rbd1.in‚ 0.01171875 disk.rbd3.in‚0 disk.nvme0n1.in‚ 17748.15625& disk.nvme0n1.out‚78440.55224609375 disk.dm-1.out‚ 428.84765625 disk.dm-0.in‚ 17748.15625 network.recv‚ 2.259613e+06) proc.memory.percent‚17.96562671661377 disk.drbd2165.out‚0 disk.sda2.in‚0 disk.sda2.out‚0$ disk./.usageGB‚549.4774284362793 disk.dm-1.in‚0 disk.rbd0.out‚0$ memory_percent‚88.14222812652588) proc.memory.availableMB‚ 1942.77734375 disk.drbd2165.in‚0# proc.memory.rssMB‚ 2943.48828125 cpu‚25.55004949664237! disk.nvme0c0n1.in‚ 17748.15625( disk.nvme0c0n1.out‚78440.55224609375 disk.sda.out‚ 428.84765625 disk.rbd0.in‚0) disk./.usagePercent‚36.87589126462971 disk.rbd1.out‚ 21.8740234375 disk.sda1.in‚0 proc.cpu.threads‚17 network.sent‚ 1.0223363e+07œ*h³Ÿ¢‚(:” °úèÒŒùóö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚272.324' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚54.464800000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0½öí1䣂(:Ù °úèÒËá“ù disk.sda2.in‚0# proc.memory.rssMB‚ 2943.48828125 disk.rbd1.out‚ 21.9365234375 network.sent‚ 1.0286295e+07 disk.dm-1.in‚0 disk.sda3.out‚429 disk.drbd2165.in‚0 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.nvme0c0n1.in‚ 17748.375) disk./.usagePercent‚36.87621844225846 disk.rbd0.out‚0# disk.dm-0.out‚78453.00146484375( disk.nvme0c0n1.out‚78453.00146484375 disk.nvme0n1.in‚ 17748.375 disk.rbd1.in‚ 0.01171875$ memory_percent‚88.13836574554443) proc.memory.percent‚17.96562671661377 disk.sda.out‚429 disk.dm-0.in‚ 17748.375 disk.sda2.out‚0 network.recv‚ 2.263784e+06 disk.rbd0.in‚0& disk.nvme0n1.out‚78453.00146484375) proc.memory.availableMB‚ 1943.41015625 cpu‚25.549744176916068 disk.sda.in‚0 proc.cpu.threads‚17$ disk./.usageGB‚549.4823036193848 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd2165.out‚0 disk.dm-1.out‚429 disk.sda3.in‚0Zÿ„•Ÿ¤‚(:” ¿úèÒµžÞö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚292.983' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚58.596599999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¿qÿÌ¥‚(:ü ¿úèÒ…‚õø disk.sda1.in‚0 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.rbd1.out‚ 21.9365234375 network.recv‚ 2.267121e+06 disk.dm-0.in‚ 17748.578125 disk.rbd3.in‚0 disk.rbd1.in‚ 0.01171875 disk.dm-1.out‚ 430.54296875) disk./.usagePercent‚36.87644577694887 disk.sda3.in‚0( disk.nvme0c0n1.out‚78466.48486328125 disk.sda2.out‚0" disk.nvme0c0n1.in‚ 17748.578125 disk.sda2.in‚0 disk.sda.in‚0 cpu‚25.54946896647921 disk.rbd0.in‚0) proc.memory.availableMB‚ 1943.16015625# proc.memory.rssMB‚ 2943.48828125) proc.memory.percent‚17.96562671661377 disk.sda3.out‚ 430.54296875 disk.rbd3.out‚0 disk.nvme0n1.in‚ 17748.578125 disk.dm-1.in‚0 disk.rbd0.out‚0 network.sent‚ 1.0291356e+07$ disk./.usageGB‚549.4856910705566& disk.nvme0n1.out‚78466.48486328125 disk.sda.out‚ 430.54296875$ memory_percent‚88.13989162445068 proc.cpu.threads‚17# disk.dm-0.out‚78466.48486328125 disk.sda1.out‚0t¤Ý”¦‚(:‰ ÎúèÒ›¨òö gpu.0.gpu‚90.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚278.198' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.6396 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ˆŸ ‘í§‚(:â ÎúèÒ瓧ù) proc.memory.availableMB‚ 1943.16015625( disk./.usagePercent‚36.8769058224879 disk.rbd0.out‚0 disk.drbd2165.in‚0 disk.sda2.out‚0 disk.dm-1.in‚0 disk.sda1.out‚0 disk.rbd3.in‚0 disk.rbd1.in‚ 0.01171875 network.recv‚ 2.270551e+06 disk.nvme0c0n1.in‚17749 disk.sda2.in‚0# proc.memory.rssMB‚ 2943.48828125 cpu‚25.54924624341603 network.sent‚ 1.0296341e+07 disk.dm-1.out‚ 433.9140625 disk.dm-0.in‚17749 disk.rbd3.out‚0 proc.cpu.threads‚17 disk.sda.in‚0 disk.sda.out‚ 433.9140625 disk.rbd0.in‚0 disk.sda1.in‚0 disk.drbd2165.out‚0& disk.nvme0n1.out‚78485.31591796875 disk.rbd1.out‚ 22.0146484375$ memory_percent‚88.13989162445068# disk.dm-0.out‚78485.31591796875 disk.sda3.in‚0 disk.sda3.out‚ 433.9140625 disk.nvme0n1.in‚17749) proc.memory.percent‚17.96562671661377# disk./.usageGB‚549.492546081543( disk.nvme0c0n1.out‚78485.31591796875^³ GŸ¨‚(:” ÝúèÒÔè€÷ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚274.743' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚54.948600000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚04Y©‚(:÷ ÝúèÒ¾Ž™ù disk.dm-1.out‚ 435.40625 disk.sda1.in‚0" disk.nvme0n1.in‚17885.07421875% disk.nvme0n1.out‚78505.1513671875 disk.rbd1.in‚ 0.01171875$ disk./.usageGB‚549.4938545227051 disk.dm-0.in‚17885.07421875) proc.memory.availableMB‚ 1943.16015625 disk.sda3.in‚0# proc.memory.rssMB‚ 2943.48828125 disk.sda1.out‚0$ memory_percent‚88.13989162445068 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda3.out‚ 435.40625 disk.sda2.in‚0 disk.rbd1.out‚ 22.0224609375) disk./.usagePercent‚36.87699363307214 disk.dm-1.in‚0 disk.drbd2165.out‚0 disk.rbd3.out‚0) proc.memory.percent‚17.96562671661377 disk.sda.out‚ 435.40625" disk.dm-0.out‚78505.1513671875 disk.drbd2165.in‚0$ disk.nvme0c0n1.in‚17885.07421875 disk.sda2.out‚0 cpu‚25.548971979975057 proc.cpu.threads‚17 network.sent‚ 1.0301353e+07 network.recv‚ 2.274296e+06 disk.rbd3.in‚0' disk.nvme0c0n1.out‚78505.1513671875 disk.sda.in‚0 FzL2ª- _wandbruntime‚9036  _runtime‚9036<]ùAÇ«Á & lr-AdamW/pg1‚0.0005846186007169646 & lr-AdamW/pg2‚0.0005846186007169646 trainer/global_step‚5449 % _timestamp‚1.7842988520500762e+09 _runtime‚9036.267247914 _step‚445 *ûÛÌ¬Æ & lr-AdamW/pg2‚0.0005846186007169646 trainer/global_step‚5449 % _timestamp‚1.7842988520500762e+09 _runtime‚9036.267247914 _step‚445 & lr-AdamW/pg1‚0.0005846186007169646½ö‡V2­- _wandbruntime‚9036  _runtime‚9036ç‰9èÿ'®ùO aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1564955240005632 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚46443143 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚15461414 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚47170059 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚13979488 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 _runtime‚9036.268459331 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚266 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚47170059 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚46443149 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1570080270928384 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚271 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚29440631 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1570078899365376 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1570078899365376 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚423118500515840 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1564955240005632 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36633600 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚46443149 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚29440902 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1564278995587072 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5116263417312 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 _runtime‚9036.268459331 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚266 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚47170059 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚46443149 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1570080270928384 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚271 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚29440631 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1570078899365376 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1570078899365376 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚423118500515840 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1564955240005632 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36633600 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚46443149 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚29440902 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1564278995587072 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5116263417312 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 IDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6605526736384 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5125955965440 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚29446167 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚29445908 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36626432 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1570359196701184 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 trainer/global_step‚5449 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚46451355 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1565233214562304 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚93639879 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚423195986923520 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5117186883670 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚47178537 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13982026 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚93639879 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚46451355 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚93629886 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚429801145568768 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1564556877365248 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1564556902793216 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1565233240735744 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15463882 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚47178537 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚254 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚429801550286336WçT*'²ˆN \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1565233240735744 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚9037.654243095 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6605526736384 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5125955965440 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚29446167 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚29445908 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36626432 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1570359196701184 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 trainer/global_step‚5449 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚46451355 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1565233214562304 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚93639879 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚423195986923520 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5117186883670 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚47178537 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13982026 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚93639879 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚46451355 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚93629886 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚429801145568768 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1564556877365248 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1564556902793216 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1565233240735744 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15463882 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚47178537 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚254 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚429801550286336 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 _step‚447 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚259 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15464136 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚423195618832384 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1570359196701184 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404717568 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1569672719648484 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154¿­Óè2³- _wandbruntime‚9037  _runtime‚9037+š'™´ /loss/FRt2/detection_loss‚1.5373499208746761 /loss/FBt1/detection_loss‚0.6308636638005415 %loss/total_loss‚8.236441999252825 epoch‚13 0loss/FRt2/assignment_loss‚0.7816601308228458 2loss/SRqqt2/assignment_loss‚0.5140412818689813 /loss/FBt2/detection_loss‚0.5068129598372435 % _timestamp‚1.7842988535240526e+09 _runtime‚9037.741037853 0loss/FRt1/assignment_loss‚1.6561522943474525 2loss/FBt1/assignment_loss‚0.015464080952716173 1loss/SRqqt2/detection_loss‚0.6356883530807986 1loss/SRqqt1/assignment_loss‚0.615241592850232 /loss/SRqqt1/detection_loss‚0.82139557057226 trainer/global_step‚5449 _step‚448 2loss/FBt2/assignment_loss‚0.008674924851813292 /loss/FRt1/detection_loss‚0.5130972253932647ë‘Š µ† 1loss/SRqqt1/assignment_loss‚0.615241592850232 /loss/SRqqt1/detection_loss‚0.82139557057226 trainer/global_step‚5449 _step‚448 2loss/FBt2/assignment_loss‚0.008674924851813292 /loss/FRt1/detection_loss‚0.5130972253932647 /loss/FRt2/detection_loss‚1.5373499208746761 /loss/FBt1/detection_loss‚0.6308636638005415 %loss/total_loss‚8.236441999252825 epoch‚13 0loss/FRt2/assignment_loss‚0.7816601308228458 2loss/SRqqt2/assignment_loss‚0.5140412818689813 /loss/FBt2/detection_loss‚0.5068129598372435 % _timestamp‚1.7842988535240526e+09 _runtime‚9037.741037853 0loss/FRt1/assignment_loss‚1.6561522943474525 2loss/FBt1/assignment_loss‚0.015464080952716173 1loss/SRqqt2/detection_loss‚0.6356883530807986À±?ƒ½ž¶‚(:“ ìúèÒ‘¼ãö gpu.0.gpu‚90.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚279.255' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚55.85099999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¥)W ·‚(: ìúèÒìøŽù disk.rbd0.in‚0 disk.sda1.in‚0# disk.dm-0.out‚78518.28369140625 network.recv‚ 2.277708e+06" disk.nvme0n1.in‚17885.27734375 disk.rbd1.in‚ 0.01171875 cpu‚25.548670909433817& disk.nvme0n1.out‚78518.28369140625 disk.sda2.in‚0 network.sent‚ 1.0306396e+07 disk.dm-0.in‚17885.27734375 disk.sda3.in‚0 disk.sda3.out‚ 437.1328125( disk.nvme0c0n1.out‚78518.28369140625* disk./.usagePercent‚36.877444462339945 disk.dm-1.out‚ 437.1328125 disk.rbd0.out‚0 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.sda2.out‚0$ memory_percent‚88.14146518707275) proc.memory.percent‚17.96562671661377 disk.sda.in‚0 disk.sda.out‚ 437.1328125$ disk.nvme0c0n1.in‚17885.27734375) proc.memory.availableMB‚ 1942.90234375 proc.cpu.threads‚17$ disk./.usageGB‚549.5005722045898 disk.drbd2165.out‚0# proc.memory.rssMB‚ 2943.48828125 disk.rbd1.out‚ 22.0849609375 disk.dm-1.in‚0 disk.sda1.out‚0 disk.drbd2165.in‚0 p딸‚(:‰ ûúèÒÖÜö gpu.0.gpu‚70.0 gpu.0.memory‚21, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚39.0 gpu.0.powerWatts‚110.527' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.1054 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚09ý¹‚(:ò ûúèÒ‘Äýù disk.rbd1.out‚ 22.0849609375 disk.rbd0.in‚0 disk.sda.in‚0 disk.sda.out‚437.6875 disk.dm-1.out‚437.6875 disk.rbd0.out‚0 disk.drbd2165.in‚0$ disk.nvme0c0n1.in‚17885.69921875) proc.memory.percent‚17.96562671661377$ memory_percent‚88.13815116882324 network.recv‚ 2.28221e+06$ disk./.usageGB‚549.5073432922363% disk.nvme0n1.out‚78534.2158203125 disk.sda1.in‚0 disk.sda1.out‚0' disk.nvme0c0n1.out‚78534.2158203125 disk.drbd2165.out‚0" disk.nvme0n1.in‚17885.69921875 disk.rbd1.in‚ 0.01171875 disk.dm-1.in‚0" disk.dm-0.out‚78534.2158203125 cpu‚25.54839308398669 network.sent‚ 1.0369344e+07 disk.sda3.in‚0# proc.memory.rssMB‚ 2943.48828125 proc.cpu.threads‚17* disk./.usagePercent‚36.877898875713214 disk.rbd3.in‚0( proc.memory.availableMB‚ 1943.4453125 disk.dm-0.in‚17885.69921875 disk.sda3.out‚437.6875 disk.rbd3.out‚0 disk.sda2.in‚0 disk.sda2.out‚0ù ái”º‚(:‰ ŠûèÒ¦Ççö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚296.751' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚59.3502 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Gø½»‚(:ö ŠûèÒš¶‹ù network.recv‚ 2.285616e+06$ disk.nvme0c0n1.in‚17885.91796875 disk.dm-1.out‚ 440.6328125 disk.dm-0.in‚17885.91796875! disk.dm-0.out‚78559.537109375 disk.sda3.in‚0# proc.memory.rssMB‚ 2943.48828125) proc.memory.percent‚17.96562671661377( disk./.usagePercent‚36.8786269611406 disk.rbd3.out‚0& disk.nvme0c0n1.out‚78559.537109375 disk.sda.in‚0 disk.dm-1.in‚0 cpu‚25.548153375458106$ disk.nvme0n1.out‚78559.537109375 disk.rbd1.in‚ 0.01171875 disk.sda2.out‚0 disk.rbd1.out‚ 22.2568359375 disk.sda.out‚ 440.6328125 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda3.out‚ 440.6328125' proc.memory.availableMB‚ 1942.921875$ disk./.usageGB‚549.5181922912598 disk.drbd2165.in‚0# memory_percent‚88.1413459777832" disk.nvme0n1.in‚17885.91796875 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd2165.out‚0 proc.cpu.threads‚17 network.sent‚ 1.0374461e+07 disk.rbd3.in‚0 disk.sda2.in‚0M 4Ÿ¼‚(:” ™ûèÒ…ÃÆù gpu.0.gpu‚65.0 gpu.0.memory‚20, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚38.0 gpu.0.powerWatts‚110.286' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.057199999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0O °ÿ½‚(:ô ™ûèÒÀ‡áû disk.rbd0.out‚0# proc.memory.rssMB‚ 2943.48828125 disk.dm-0.in‚ 17974.015625 disk.sda1.in‚0 disk.sda.in‚0# disk.dm-0.out‚78579.60888671875 disk.sda3.out‚ 443.21875 disk.nvme0n1.in‚ 17974.015625 disk.sda2.in‚0 disk.sda1.out‚0 network.sent‚ 1.0379505e+07* disk./.usagePercent‚36.878625681102925 disk.rbd0.in‚0 disk.drbd2165.in‚0( disk.nvme0c0n1.out‚78579.60888671875 disk.rbd1.out‚ 22.2568359375 disk.rbd3.out‚0$ memory_percent‚88.14365863800049 network.recv‚ 2.289138e+06 disk.sda.out‚ 443.21875 disk.dm-1.out‚ 443.21875" disk.nvme0c0n1.in‚ 17974.015625) proc.memory.percent‚17.96562671661377 cpu‚25.54789385220026 proc.cpu.threads‚17$ disk./.usageGB‚549.5181732177734 disk.dm-1.in‚0 disk.rbd3.in‚0 disk.sda2.out‚0 disk.rbd1.in‚ 0.01171875) proc.memory.availableMB‚ 1942.54296875 disk.sda3.in‚0 disk.drbd2165.out‚0& disk.nvme0n1.out‚78579.60888671875§8’¾‚(:‡ ¨ûèÒç›àö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚276.31' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.262 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0±Ýaú¿‚(:ï ¨ûèÒ‰¦Œù network.recv‚ 2.292255e+06 disk.rbd1.in‚ 0.01171875 disk.dm-0.in‚ 17974.4375 disk.sda1.in‚0 disk.rbd3.in‚0 disk.sda2.in‚0 disk.rbd1.out‚ 22.3193359375 disk.rbd0.in‚0$ disk./.usageGB‚549.5219688415527 disk.rbd3.out‚0& disk.nvme0c0n1.out‚78602.521484375) proc.memory.percent‚17.96562671661377 network.sent‚ 1.0383984e+07 disk.sda3.in‚0 disk.sda2.out‚0 disk.sda.out‚ 445.19921875 disk.sda1.out‚0 disk.sda3.out‚ 445.19921875 disk.dm-1.out‚ 445.19921875! disk.dm-0.out‚78602.521484375 cpu‚25.547611649443613 disk.drbd2165.out‚0 disk.nvme0n1.in‚ 17974.4375 disk.rbd0.out‚0' proc.memory.availableMB‚ 1942.390625) disk./.usagePercent‚36.87888040859949 disk.drbd2165.in‚0 disk.dm-1.in‚0# proc.memory.rssMB‚ 2943.48828125 proc.cpu.threads‚17 disk.nvme0c0n1.in‚ 17974.4375$ disk.nvme0n1.out‚78602.521484375 disk.sda.in‚0$ memory_percent‚88.14458847045898€Ç†;2À- _wandbruntime‚9115  _runtime‚9115‚*çWÆÁÀ _step‚449 & lr-AdamW/pg1‚0.0005826043698537409 & lr-AdamW/pg2‚0.0005826043698537409 trainer/global_step‚5499 $ _timestamp‚1.784298930970541e+09 _runtime‚9115.187705997̽C’ËÂÅ _runtime‚9115.187705997 _step‚449 & lr-AdamW/pg1‚0.0005826043698537409 & lr-AdamW/pg2‚0.0005826043698537409 trainer/global_step‚5499 $ _timestamp‚1.784298930970541e+09Áô°q2Ã- _wandbruntime‚9115  _runtime‚9115¾•%óŒÄùO A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 _step‚450 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚47595811 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚433656615967232 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1578173109923840 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚46853443 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚29705381 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1578855250337792 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1584026623364608 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1584026623364608 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚47595811 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5162500530420 _runtime‚9115.191971981 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1584025251801600 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚94459233 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚15598995 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 trainer/global_step‚5499 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚426992452808704 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1583335610454260 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1578855276511232 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚14106633 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6664163158528 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚46853449 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6664197694976 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1578855276511232 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚247 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5170001463808 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5170001463808 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚433657018594816 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5171346853376 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15598748 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1583334240425970 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚46853443 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚426992820899840 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚47605784 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_gÃömspool.peak‚12097 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚252 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 % _timestamp‚1.7842989309716904e+09 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1578855250337792 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚29705633 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚14106638 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚94449254 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚29705381 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1578855250337792 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1584026623364608 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1584026623364608 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚47595811 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5162500530420 _runtime‚9115.191971981 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1584025251801600 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚94459233 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚15598995 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 trainer/global_step‚5499 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚426992452808704 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1583335610454260 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1578855276511232 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚14106633 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6664163158528 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚46853449 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6664197694976 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1578855276511232 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚247 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5170001463808 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5170001463808 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚433657018594816 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5171346853376 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15598748 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1583334240425970 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚46853443 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚426992820899840 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚47605784 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚252 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 % _timestamp‚1.7842989309716904e+09 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1578855250337792 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚29705633 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚14106638 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚94449254 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15601731 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚47604289 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1579133277241344 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚433735700406784 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5162079396344 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1584305549137408 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5172271896064 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚94475931 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5172271896064 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1578450991702016 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚47604289 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1583614415698794 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1584304177567232 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1578450966274048 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚47614276 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1579133251067904 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚427069939216384 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚239 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚427070307307520 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚94475931 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚14109181 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15601497 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5170926499328 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1583613045670392 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚46861655 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚433735297786368 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚47614276 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚94465938 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚29710673 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚94465938 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1584305549137408 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 % _timestamp‚1.7842989323574183e+09 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚46861649 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 trainer/global_step‚5499 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1579133251067904 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5163423996778 _step‚451 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚46861649 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 _runtime‚9116.574894872 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0‡Km'ȈN =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚29710673 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚94465938 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1584305549137408 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 % _timestamp‚1.7842989323574183e+09 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚46861649 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 trainer/global_step‚5499 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1579133251067904 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5163423996778 _step‚451 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚46861649 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 _runtime‚9116.574894872 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15601731 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚47604289 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1579133277241344 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚433735700406784 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5162079396344 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1584305549137408 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5172271896064 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚94475931 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5172271896064 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1578450991702016 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚47604289 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1583614415698794 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1584304177567232 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1578450966274048 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚47614276 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1579133251067904 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚427069939216384 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚239 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚427070307307520 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚94475931 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚14109181 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15601497 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5170926499328 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1583613045670392 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚46861655 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚433735297786368 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚47614276 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚94465938 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0Ãý™™#2É- _wandbruntime‚9116  _runtime‚9116ošàDÝʆ 3loss/FBt1/assignment_loss‚0.0007835484860510702 1loss/SRqqt2/detection_loss‚0.6234129782916988 0loss/FRt2/assignment_loss‚0.7107617052532824 3loss/FBt2/assignment_loss‚0.0004653637228784757 /losst„Ê //FRt1/detection_loss‚0.5147029740576847 epoch‚13 % _timestamp‚1.7842989324441583e+09 /loss/FRt2/detection_loss‚1.5275021170732779 /loss/FBt1/detection_loss‚0.7429863858575884 3loss/SRqqt2/assignment_loss‚0.48927718330339354 1loss/SRqqt1/detection_loss‚1.0106790164041766 /loss/FBt2/detection_loss‚0.3123720873518323 %loss/total_loss‚7.950405432683889 trainer/global_step‚5499 _runtime‚9116.661100988 _step‚452 /loss/FRt1/assignment_loss‚1.487494951936434 2loss/SRqqt1/assignment_loss‚0.5299671209455903ƒ¦×^Ë‹ % _timestamp‚1.7842989324441583e+09 /loss/FRt2/detection_loss‚1.5275021170732779 /loss/FBt1/detection_loss‚0.7429863858575884 3loss/SRqqt2/assignment_loss‚0.48927718330339354 1loss/SRqqt1/detection_loss‚1.0106790164041766 /loss/FBt2/detection_loss‚0.3123720873518323 %loss/total_loss‚7.950405432683889 trainer/global_step‚5499 _runtime‚9116.661100988 _step‚452 /loss/FRt1/assignment_loss‚1.487494951936434 2loss/SRqqt1/assignment_loss‚0.5299671209455903 3loss/FBt1/assignment_loss‚0.0007835484860510702 1loss/SRqqt2/detection_loss‚0.6234129782916988 0loss/FRt2/assignment_loss‚0.7107617052532824 3loss/FBt2/assignment_loss‚0.0004653637228784757 /loss/FRt1/detection_loss‚0.5147029740576847 epoch‚13Ä{@žÌ‚(:“ ·ûèÒÕ—…÷ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚309.24' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚61.848000000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÎÒ.öÍ‚(:ë ·ûèÒ­ÿ¼ù disk.rbd1.out‚ 22.3193359375' proc.memory.availableMB‚ 1942.328125 disk.sda3.in‚0# disk.nvme0n1.out‚78627.55859375 disk.sda2.out‚0$ memory_percent‚88.14496994018555 cpu‚25.54736726233626 disk.sda2.in‚0 disk.sda1.out‚0 disk.sda.out‚ 448.4921875 disk.dm-0.out‚78627.55859375 disk.sda3.out‚ 448.4921875 disk.rbd3.in‚0 disk.nvme0n1.in‚ 17974.65625) disk./.usagePercent‚36.87928771658647 disk.sda.in‚0 disk.dm-0.in‚ 17974.65625# proc.memory.rssMB‚ 2943.48828125 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.drbd2165.out‚0! disk.nvme0c0n1.in‚ 17974.65625% disk.nvme0c0n1.out‚78627.55859375 disk.rbd1.in‚ 0.01171875 network.sent‚ 1.0389458e+07 disk.dm-1.out‚ 448.4921875 disk.sda1.in‚0 disk.drbd2165.in‚0 disk.rbd3.out‚0) proc.memory.percent‚17.96562671661377 proc.cpu.threads‚17 network.recv‚ 2.295371e+06$ disk./.usageGB‚549.5280380249023 disk.dm-1.in‚0¢Ëo’΂(:‡ ÆûèÒ‰®éö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚287.91' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚57.582 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚02 Ï‚(:ù ÆûèÒÂÎù network.recv‚ 2.299778e+06$ disk./.usageGB‚549.5351676940918 disk.dm-0.in‚ 17974.859375 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda.in‚0 disk.dm-1.out‚ 452.5546875$ memory_percent‚88.14458847045898# disk.dm-0.out‚78647.77001953125 disk.rbd3.out‚0& disk.nvme0n1.out‚78647.77001953125 disk.sda2.in‚0 disk.dm-1.in‚0 disk.drbd2165.out‚0 disk.rbd3.in‚0 cpu‚25.547073531174753 network.sent‚ 1.0451697e+07* disk./.usagePercent‚36.879766194667965 disk.sda3.in‚0" disk.nvme0c0n1.in‚ 17974.859375 disk.sda.out‚ 452.5546875) proc.memory.percent‚17.96562671661377 disk.rbd0.in‚0 disk.sda3.out‚ 452.5546875 disk.drbd2165.in‚0( disk.nvme0c0n1.out‚78647.77001953125 disk.nvme0n1.in‚ 17974.859375 disk.sda2.out‚0 disk.rbd1.out‚ 22.3974609375' proc.memory.availableMB‚ 1942.390625 disk.rbd1.in‚ 0.01171875# proc.memory.rssMB‚ 2943.48828125 proc.cpu.threads‚17 disk.rbd0.out‚0‚‰qx”Ђ(:‰ ÕûèÒþŸòö gpu.0.gpu‚93.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚336.329' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚67.2658 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ˆ ã-ÿÑ‚(:ô ÕûèÒó‰ú# disk./.usageGB‚549.538402557373 disk.drbd2165.in‚0" disk.nvme0c0n1.in‚ 18069.671875) proc.memory.availableMB‚ 1942.44140625 disk.sda1.in‚0 network.sent‚ 1.0456693e+07$ disk.nvme0n1.out‚78673.083984375# proc.memory.rssMB‚ 2943.48828125 disk.dm-1.in‚0 disk.sda3.in‚0$ memory_percent‚88.14427852630615 proc.cpu.threads‚17 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd1.out‚ 22.3974609375' disk./.usagePercent‚36.879983289057 disk.dm-0.in‚ 18069.671875 disk.drbd2165.out‚0 disk.rbd3.in‚0 cpu‚25.546827422192706 disk.sda.out‚ 454.12890625 disk.rbd0.out‚0 disk.sda1.out‚0 disk.sda3.out‚ 454.12890625& disk.nvme0c0n1.out‚78673.083984375 disk.nvme0n1.in‚ 18069.671875 disk.sda.in‚0 disk.rbd0.in‚0! disk.dm-0.out‚78673.083984375) proc.memory.percent‚17.96562671661377 disk.dm-1.out‚ 454.12890625 disk.rbd3.out‚0 disk.rbd1.in‚ 0.01171875 network.recv‚ 2.303171e+06¬|‰—”Ò‚(:‰ äûèÒ›®Ùö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚270.414' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.0828 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÈÆÿÓ‚(:ô äûèÒª Ûø disk.drbd2165.out‚0% disk.nvme0n1.out‚78682.5146484375) proc.memory.percent‚17.96562671661377 disk.dm-0.in‚ 18069.890625 disk.sda1.in‚0 network.sent‚ 1.0461803e+07 disk.rbd0.out‚0 disk.sda.out‚ 455.8984375" disk.nvme0c0n1.in‚ 18069.890625 disk.nvme0n1.in‚ 18069.890625' proc.memory.availableMB‚ 1942.359375 cpu‚25.546541151801268 disk.rbd1.in‚ 0.01171875( disk./.usagePercent‚36.8803025304522$ disk./.usageGB‚549.5431594848633 disk.dm-1.in‚0 disk.sda3.out‚ 455.8984375 disk.rbd3.out‚0 disk.sda2.in‚0 disk.rbd3.in‚0 disk.sda2.out‚0$ memory_percent‚88.14477920532227 disk.sda1.out‚0 disk.sda.in‚0 disk.rbd0.in‚0 disk.drbd2165.in‚0' disk.nvme0c0n1.out‚78682.5146484375 proc.cpu.threads‚17 disk.dm-1.out‚ 455.8984375 disk.sda3.in‚0 disk.rbd1.out‚ 22.4599609375" disk.dm-0.out‚78682.5146484375 network.recv‚ 2.306622e+06# proc.memory.rssMB‚ 2943.48828125¬ÓžÔ‚(:“ óûèÒÖ¢Þö gpu.0.gpu‚92.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚339.413' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚67.88260000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0­¨ \öÕ‚(:ë óûèÒ¡Ìïø disk.sda.in‚0 disk.sda3.in‚0 disk.rbd3.out‚0 disk.rbd1.in‚ 0.01171875' proc.memory.availableMB‚ 1942.109375) disk./.usagePercent‚36.88075540778027 disk.sda1.out‚0 disk.nvme0n1.in‚ 18070.09375 disk.rbd0.in‚0 disk.dm-0.in‚ 18070.09375 disk.drbd2165.out‚0! disk.nvme0c0n1.in‚ 18070.09375$ memory_percent‚88.14630508422852# proc.memory.rssMB‚ 2943.48828125 cpu‚25.54630160266997 disk.sda2.in‚0 disk.rbd1.out‚ 22.4599609375 disk.sda.out‚ 456.4453125 disk.dm-1.out‚ 456.4453125 disk.dm-0.out‚78694.36328125 disk.sda3.out‚ 456.4453125 disk.drbd2165.in‚0# disk.nvme0n1.out‚78694.36328125 network.sent‚ 1.0466848e+07) proc.memory.percent‚17.96562671661377 proc.cpu.threads‚17 disk.rbd0.out‚0 disk.sda1.in‚0 disk.rbd3.in‚0% disk.nvme0c0n1.out‚78694.36328125 disk.sda2.out‚0$ disk./.usageGB‚549.5499076843262 disk.dm-1.in‚0 network.recv‚ 2.310252e+06‹N`¶2Ö- _wandbruntime‚9193  _runtime‚9193¾‘[îÇ×Á % _timestamp‚1.7842990094970896e+09 _runtime‚9193.714259311 _step‚453 & lr-AdamW/pg1‚0.0005805749374671467 & lr-AdamW/pg2‚0.0005805749374671467 trainer/global_step‚5549n¡ÌØÆ % _timestamp‚1.7842990094970896e+09 _runtime‚9193.714259311 _step‚453 & lr-AdamW/pg1‚0.0005805749374671467 & lr-AdamW/pg2‚0.0005805749374671467 trainer/global_step‚5549Ånâí 2Ù- _wandbruntime‚9193  _runtime‚9193ºéb+ÿ'ÚùO JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚95295285 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15736384 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1597275936476168 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚48021563 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1597972975800832 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚437591411757568 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚95285306 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚48031536 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚235 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 400530432 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1597972975800832 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1592755313016832 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚14233788 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7842990094982326e+09 _step‚454 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5217662784000 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚48021563 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1592755286843392 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1597274566447878 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _runtime‚9193.715637313 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1592755286843392 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚47263743 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚47263749 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚47263743 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1592067173404672 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5217662784000 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 trainer/global_step‚5549 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1597971604237824 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚14233783 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1592755313016832 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚437591011227136 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1597971604237824 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5216317394432 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1597971604237824 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚14233783 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1592755313016832 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚437591011227136 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1597971604237824 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5216317394432 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15736384 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1597275936476168 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚48021563 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1597972975800832 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚437591411757568 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚95285306 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚48031536 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚235 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 400530432 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1597972975800832 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1592755313016832 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚14233788 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7842990094982326e+09 _step‚454 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5217662784000 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚48021563 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1592755286843392 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1597274566447878 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _runtime‚9193.715637313 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1592755286843392 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚47263743 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚47263749 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚47263743 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1592067173404672 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5217662784000 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 trainer/global_step‚5549Æpnü`žÜ‚(:“ ‚üèÒ•ûåö gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚283.318' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚56.66359999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0,¬EÁûÝ‚(:ð ‚üèÒ¤§òø) disk./.usagePercent‚36.88111049023026 disk.dm-1.in‚0 disk.rbd0.out‚0! disk.dm-0.out‚78707.658203125 disk.sda3.out‚ 457.3046875 disk.rbd3.out‚0 disk.nvme0n1.in‚ 18070.515625 cpu‚25.546008939111907 disk.rbd1.out‚ 22.5380859375 network.sent‚ 1.0471862e+07 disk.drbd2165.out‚0$ disk.nvme0n1.out‚78707.658203125 disk.sda.out‚ 457.3046875) proc.memory.percent‚17.96562671661377$ disk./.usageGB‚549.5551986694336 disk.rbd0.in‚0 disk.dm-0.in‚ 18070.515625 disk.sda1.out‚0 disk.sda3.in‚0 disk.sda.in‚0 disk.drbd2165.in‚0& disk.nvme0c0n1.out‚78707.658203125 disk.sda2.out‚0# memory_percent‚88.1418228149414 proc.cpu.threads‚17 disk.dm-1.out‚ 457.3046875 disk.rbd1.in‚ 0.01171875 network.recv‚ 2.313577e+06 disk.sda1.in‚0 disk.rbd3.in‚0" disk.nvme0c0n1.in‚ 18070.515625 disk.sda2.in‚0& proc.memory.availableMB‚ 1942.84375# proc.memory.rssMB‚ 2943.48828125uÇz2Þ- _wandbruntime‚9195  _runtime‚9195@È‚0}߃N NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚47271955 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚48030041 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5218587826688 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1598251901573632 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚95301990 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚430944627691520 trainer/global_step‚5549 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚47271949 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚48030041 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocatiI<}©Œ ons.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚95311983 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚437669703015936 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1593033313746944 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚29975419 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15739093 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5208316509452 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 _step‚455 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚217 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1597553371692300 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5217242429952 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚437670103539200 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚48040028 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚29975641 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚14236331 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6725475847680 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚222 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1592345080610816 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1592345055182848 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1593033313746944 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚430944259600384 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1597554741720702 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5209661109886 % _timestamp‚1.7842990108809855e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5218587826688 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚14236326 _runtime‚9195.098352145 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚29975641 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚14236331 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6725475847680 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚222 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1592345080610816 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1592345055182848 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1593033313746944 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚430944259600384 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1597554741720702 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5209661109886 % _timestamp‚1.7842990108809855e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5218587826688 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚14236326 _runtime‚9195.098352145 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1598251901573632 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚95301990 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚430944627691520 trainer/global_step‚5549 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚47271949 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚48030041 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚95311983 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚437669703015936 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1593033313746944 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚29975419 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15739093 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5208316509452 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 _step‚455 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736ÇVÕû³2á- _wandbruntime‚9195  _runtime‚9195ÿMñLâ /loss/FBt2/detection_loss‚0.5553472636635902 %loss/total_loss‚7.998114119871882 1loss/SRqqt1/detection_loss‚0.8497885934430911 epoch‚13 _runtime‚9195.184887405 _step‚456 3loss/SRqqt1/assignment_loss‚0.49628218703434807 2loss/FBt1/assignment_loss‚0.010073800036804498 /loss/FRt1/detection_loss‚0.4769041551904673 0loss/SRqqt2/detection_loss‚0.639167950110105 trainer/global_step‚5549 $ _timestamp‚1.784299010967894e+09 2loss/SRqqt2/assignment_loss‚0.4799596567651522 2loss/FBt2/assignment_loss‚0.001949200456153721 .loss/FRt2/detection_loss‚1.536331516872718 /loss/FBt1/detection_loss‚0.5560977531445993 /loss/FRt1/assignment_loss‚1.577940883029121 0loss/FRt2/assignment_loss‚0.8182711601257324ÑÁXÍ ã† 3loss/SRqqt1/assignment_loss‚0.49628218703434807 2loss/FBt1/assignment_loss‚0.010073800036804498 /loss/FRt1/detection_loss‚0.4769041551904673 0loss/SRqqt2/detection_loss‚0.639167950110105 trainer/global_step‚5549 $ _timestamp‚1.784299010967894e+09 2loss/SRqqt2/assignment_loss‚0.4799596567651522 2loss/FBt2/assignment_loss‚0.001949200456153721 .loss/FRt2/detection_loss‚1.536331516872718 /loss/FBt1/detection_loss‚0.5560977531445993 /loss/FRt1/assignment_loss‚1.577940883029121 0loss/FRt2/assignment_loss‚0.8182711601257324 /loss/FBt2/detection_loss‚0.5553472636635902 %loss/total_loss‚7.998114119871882 1loss/SRqqt1/detection_loss‚0.8497885934430911 epoch‚13 _runtime‚9195.184887405 _step‚456ÈLJ6ý”ä‚(:‰ ‘üèÒ©’Ÿ÷ gpu.0.gpu‚69.0 gpu.0.memory‚22, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚335.937' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚67.1874 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¶'€ù å‚(:þ ‘üèÒá³ïø) disk./.usagePercent‚36.88170903584532 disk.drbd2165.in‚0& disk.nvme0n1.out‚78718.94189453125) proc.memory.percent‚17.96562671661377 proc.cpu.threads‚17 disk.sda.out‚ 457.83984375 disk.rbd0.in‚0 disk.rbd3.in‚0 disk.dm-0.in‚ 18215.5390625 cpu‚25.5457989628981 disk.sda3.out‚ 457.83984375 network.sent‚ 1.0524065e+07$ disk./.usageGB‚549.5641174316406 disk.sda1.out‚0$ memory_percent‚88.14160823822021) proc.memory.availableMB‚ 1942.87890625 disk.sda2.out‚0 disk.drbd2165.out‚0! disk.nvme0n1.in‚ 18215.5390625 disk.rbd1.out‚ 22.5380859375 disk.dm-1.out‚ 457.83984375# proc.memory.rssMB‚ 2943.48828125 network.recv‚ 2.318134e+06# disk.dm-0.out‚78718.94189453125 disk.sda3.in‚0# disk.nvme0c0n1.in‚ 18215.5390625 disk.sda2.in‚0 disk.rbd0.out‚0 disk.sda1.in‚0 disk.rbd3.out‚0( disk.nvme0c0n1.out‚78718.94189453125 disk.rbd1.in‚ 0.01171875 disk.sda.in‚0 disk.dm-1.in‚0ŠtÁ“æ‚(:ˆ  üèÒ Íš÷ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚287.615' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚57.523 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0<øDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚47674043 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚14360938 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1605961262313472 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚15874330 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1611919328237056 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚48447315 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚47674049 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5253630156314 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1606655349522432 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚30235268 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5254974756636 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚47674049 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1606655349522432 D;DeviceStatsMonitor.on_train_batch„ŸwéY_start/segment.all.current‚1995 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5262633325056 _runtime‚9272.397629532 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1606655323348992 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚434741093576704 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1611919328237056 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚30235017 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚434741461667840 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 $ _timestamp‚1.784299088180287e+09 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5262633325056 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚14360933 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚246 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1611216262498076 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1611917956674048 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15874084 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6784575523328 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚48457288 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6784610059776 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5263978714624 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚47674049 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1606655349522432 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5262633325056 _runtime‚9272.397629532 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1606655323348992 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚434741093576704 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1611919328237056 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚30235017 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚434741461667840 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 $ _timestamp‚1.784299088180287e+09 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5262633325056 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚14360933 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚246 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1611216262498076 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1611917956674048 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15874084 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6784575523328 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚48457288 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6784610059776 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5263978714624 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚47674043 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚14360938 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1605961262313472 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097ÊpšÝï2ô- _wandbruntime‚9273  _runtime‚9273M©y 'õƒN D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚47682249 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚48465780 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚48455793 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚47682249 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚48465780 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1606239144091648 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1611495067742610 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚441604759839232 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚5599 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5263558360576 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚441604357218816 _step‚459 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚434818948075520 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚30240270 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚47682255 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚96148035 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚48455793 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚231 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1611493697714208 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚96148035 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚47682255 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚434818579984384 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5255898222994 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5264903757312 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚434818948075520 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚30240270 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚47682255 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚96148035 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚48455793 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚231 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1611493697714208 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚96148035 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚47682255 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚434818579984384 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5255898222994 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5264903757312 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚47682249 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚48465780 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1606239144091648 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 XCDeviceSt?¿ójÈ atsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1611495067742610 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚441604759839232 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚5599 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5263558360576 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚441604357218816 _step‚459 E&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6419439315795898 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.03568877652287483 particle/accuracy_1_of_2‚1 jet/accuracy_4_of_5‚0 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9925665259361267 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.025561364367604256 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9982653856277466 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.16971004009246826 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 (particle/accuracy‚0.8530932068824768 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.05946573242545128 +"Purity/2FRt*SRqqt2FBt/event_purity‚0.75 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9984205961227417 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8883976936340332 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.03281654790043831 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9898792505264282 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.04696737974882126 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.035231661051511765 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.1538461595773697 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 6"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.7103311419487 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6736106276512146 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9099808931350708 8"Purity/1FRt0SRqqt1FBt/event_purity‚0.650144100189209 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.19651606678962708 *jet/accuracy_3_of_3‚0.0780334547162056 8"Purity/1FRt0SRqqt2FBt/event_purity‚0.886715829372406 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.1288406252861023 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.47453704476356506 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9230769276618958 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5488664507865906 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9987687468528748 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8975813984870911 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.21067054569721222 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.03253466635942459 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.8047856688499451 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 /particle/accuracy_4_of_4‚0.9590263366699219 6 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.997677743434906 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9976905584335327 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7359312772750854 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4296875 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.03061528690159321 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.025561364367604256 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 % _timestamp‚1.7842991157463942e+09 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.39739227294921875 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.42592594027519226 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9659565687179565 particle/accuracy_2_of_2‚1 7validation_average_jet_accuracy‚0.37613746523857117 particle/accuracy_5_of_5‚0 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9937704801559448 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 jet/accuracy_2_of_5‚1 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.12477137893438339 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8446038365364075 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6455510258674622 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7582095265388489 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.20703376829624176 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8446038365364075 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.20525573194026947 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.19006730616092682 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.3261154294013977 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4786078929901123 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9032845497131348 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7600043416023254 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7679396271705627 particle/accuracy_2_of_5‚1 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.7176960706710815 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8832109570503235 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 )jet/accuracy_2_of_3‚0.569678783416748 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.14166668057441711 /particle/accuracy_3_of_4‚0.9977324604988098 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 +jet/accuracy_3_of_4‚0.10172148793935776 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7611581087112427 particle/accuracy_1_of_1‚1 jet/accuracy_1_of_5‚1 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5458333492279053 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9285714030265808 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6671752333641052 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7494276165962219 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9982653856277466 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 8"Purity/0FRt2SRqqt0FBt/event_purity‚0.482925146818161 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.19109980762004852 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.16316330432891846 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 particle/accuracy_1_of_4‚1 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9950533509254456 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6753987073898315 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.8571428656578064 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 particle/accuracy_1_of_3‚1 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9921822547912598 6 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.992766261100769 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6552014350891113 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.1577349603176117 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 -"Purity/2FRt0SRqqt1FBt/event_purity‚0.3125 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3200412094593048 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 *jet/accuracy_1_of_2‚0.8000361323356628 particle/accuracy_4_of_5‚1 _runtime‚9299.964445079 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6159067153930664 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6809888482093811 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7317711114883423 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 *jet/accuracy_1_of_3‚0.9229243993759155 particle/accuracy_2_of_3‚1 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 * Purity/2FRt*SRqqt2FBt/FRt_purity‚0.875 - Purity/2FRt0SRqqt1FBt/FRt_purity‚0.609375 jet/accuracy_5_of_5‚0 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6852255463600159 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9821432828903198 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.22018344700336456 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.04207504913210869 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.18570177257061005 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.034581057727336884 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.1538461595773697 particle/accuracy_3_of_5‚1 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8194444179534912 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9943838715553284 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.2828874886035919 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9963935017585754 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8611656427383423 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6881343722343445 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.5128552913665771 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 6 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.986477792263031 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6724440455436707 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.20703376829624176 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.20857727527618408 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.44649168848991394 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8862056732177734 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6543790698051453 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7529729008674622 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9351851344108582 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9790050983428955 *jet/accuracy_2_of_4‚0.8003683686256409 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 particle/accuracy_2_of_4‚1 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6774096488952637 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7601891756057739 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8194444179534912 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7653886079788208 particle/accuracy_1_of_5‚1 (loss/val_total_loss‚8.21152075254523 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 *jet/accuracy_1_of_4‚0.9629429578781128 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.7145294547080994 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.12962964177131653 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6753987073898315 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 'particle/f_score‚0.6520504951477051 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.46940481662750244 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.1778191328048706 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.8111489415168762 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.18200834095478058 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6579378843307495 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.12962964177131653 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9451196789741516 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.2200814187526703 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5743929743766785 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 6 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.886715829372406 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 trainer/global_step‚5599 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 *jet/accuracy_1_of_1‚0.3326077163219452 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.39908966422080994 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6943397521972656 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 +jet/accuracy_2_of_2‚0.15468060970306396 jet/accuracy_3_of_5‚1 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7582248449325562 epoch‚13 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7300432920455933 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.21461832523345947 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293â£lq+þùz & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7300432920455933 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.21461832523345947 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.17619502544403076 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.21917679905891418 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 particle/accuracy_3_of_3‚1 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.1999092996120453 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5676952004432678 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9829781651496887 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7585585713386536 +jet/accuracy_4_of_4‚0.01628045178949833 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.14169606566429138 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.8048352003097534 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.021654298529028893 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.2722361981868744 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 +particle/specificity‚0.9110966920852661 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.7125721573829651 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.11231066286563873 +particle/sensitivity‚0.6409980058670044 _step‚461 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4742012917995453 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.21457026898860931 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6419439315795898 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.03568877652287483 particle/accuracy_1_of_2‚1 jet/accuracy_4_of_5‚0 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9925665259361267 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.025561364367604256 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9982653856277466 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.16971004009246826 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 (particle/accuracy‚0.8530932068824768 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.05946573242545128 +"Purity/2FRt*SRqqt2FBt/event_purity‚0.75 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9984205961227417 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8883976936340332 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.03281654790043831 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9898792505264282 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.04696737974882126 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.035231661051511765 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.1538461595773697 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 6"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.7103311419487 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6736106276512146 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9099808931350708 8"Purity/1FRt0SRqqt1FBt/event_purity‚0.650144100189209 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.19651606678962708 *jet/accuracy_3_of_3‚0.0780334547162056 8"Purity/1FRt0SRqqt2FBt/event_purity‚0.886715829372406 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.1288406252861023 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.47453704476356506 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9230769276618958 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5488664507865906 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9987687468528748 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8975813984870911 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.21067054569721222 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.03253466635942459 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.8047856688499451 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 /particle/accuracy_4_of_4‚0.9590263366699219 6 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.997677743434906 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9976905584335327 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7359312772750854 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4296875 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.03061528690159321 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.025561364367604256 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 % _timestamp‚1.7842991157463942e+09 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.39739227294921875 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.42592594027519226 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9659565687179565 particle/accuracy_2_of_2‚1 7validation_average_jet_accuracy‚0.37613746523857117 particle/accuracy_5_of_5‚0 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9937704801559448 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 jet/accuracy_2_of_5‚1 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.12477137893438339 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8446038365364075 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6455510258674622 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7582095265388489 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.20703376829624176 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8446038365364075 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.20525573194026947 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.19006730616092682 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.3261154294013977 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4786078929901123 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9032845497131348 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7600043416023254 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7679396271705627 particle/accuracy_2_of_5‚1 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.7176960706710815 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8832109570503235 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 )jet/accuracy_2_of_3‚0.569678783416748 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.14166668057441711 /particle/accuracy_3_of_4‚0.9977324604988098 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 +jet/accuracy_3_of_4‚0.10172148793935776 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7611581087112427 particle/accuracy_1_of_1‚1 jet/accuracy_1_of_5‚1 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5458333492279053 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9285714030265808 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6671752333641052 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7494276165962219 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9982653856277466 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 8"Purity/0FRt2SRqqt0FBt/event_purity‚0.482925146818161 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.19109980762004852 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.16316330432891846 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 particle/accuracy_1_of_4‚1 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9950533509254456 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6753987073898315 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.8571428656578064 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 particle/accuracy_1_of_3‚1 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9921822547912598 6 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.992766261100769 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6552014350891113 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.1577349603176117 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 -"Purity/2FRt0SRqqt1FBt/event_purity‚0.3125 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3200412094593048 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 *jet/accuracy_1_of_2‚0.8000361323356628 particle/accuracy_4_of_5‚1 _runtime‚9299.964445079 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6159067153930664 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6809888482093811 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7317711114883423 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 *jet/accuracy_1_of_3‚0.9229243993759155 particle/accuracy_2_of_3‚1 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 * Purity/2FRt*SRqqt2FBt/FRt_purity‚0.875 - Purity/2FRt0SRqqt1FBt/FRt_purity‚0.609375 jet/accuracy_5_of_5‚0 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6852255463600159 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9821432828903198 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.22018344700336456 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.04207504913210869 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.18570177257061005 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.034581057727336884 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.000279017869615927Ÿ÷nŠ34 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.1538461595773697 particle/accuracy_3_of_5‚1 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8194444179534912 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9943838715553284 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.2828874886035919 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9963935017585754 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8611656427383423 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6881343722343445 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.5128552913665771 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 6 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.986477792263031 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6724440455436707 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.20703376829624176 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.20857727527618408 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.44649168848991394 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8862056732177734 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6543790698051453 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7529729008674622 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9351851344108582 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9790050983428955 *jet/accuracy_2_of_4‚0.8003683686256409 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 particle/accuracy_2_of_4‚1 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6774096488952637 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7601891756057739 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8194444179534912 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7653886079788208 particle/accuracy_1_of_5‚1 (loss/val_total_loss‚8.21152075254523 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 *jet/accuracy_1_of_4‚0.9629429578781128 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.7145294547080994 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.12962964177131653 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6753987073898315 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 'particle/f_score‚0.6520504951477051 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.46940481662750244 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.1778191328048706 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.8111489415168762 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.18200834095478058 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6579378843307495 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.12962964177131653 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9451196789741516 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.2200814187526703 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5743929743766785 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 6 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.886715829372406 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 trainer/global_step‚5599 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 *jet/accuracy_1_of_1‚0.3326077163219452 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.39908966422080994 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6943397521972656 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 +jet/accuracy_2_of_2‚0.15468060970306396 jet/accuracy_3_of_5‚1 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7582248449325562 epoch‚13 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067͇¥½,”ÿ‚(:‰ ëüèÒ”ÜÁö gpu.0.gpu‚97.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚36.0 gpu.0.powerWatts‚110.656' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.1312 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¥+þ#ÿ€‚(:ô ëüèÒÒˆÄú disk.dm-1.in‚0 disk.sda2.in‚0 disk.rbd1.out‚ 22.8037109375 disk.sda1.out‚0 cpu‚25.551676371560657 disk.sda.out‚ 459.66796875 disk.dm-0.in‚18312.35546875 disk.dm-0.out‚ 78756.984375 disk.sda3.in‚0$ disk.nvme0c0n1.in‚18312.35546875# disk.nvme0c0n1.out‚ 78756.984375 disk.sda2.out‚0 disk.rbd3.out‚0" disk.nvme0n1.in‚18312.35546875) proc.memory.availableMB‚ 1942.14453125) disk./.usagePercent‚36.88247321833502 disk.sda3.out‚ 459.66796875 disk.drbd2165.out‚0) proc.memory.percent‚17.96562671661377 disk.rbd0.in‚0 disk.rbd0.out‚0 network.sent‚ 1.0659699e+07 network.recv‚ 2.339949e+06 disk.rbd1.in‚ 0.01171875 disk.dm-1.out‚ 459.66796875# proc.memory.rssMB‚ 2943.48828125 disk.sda.in‚0 disk.sda1.in‚0 disk.drbd2165.in‚0! disk.nvme0n1.out‚ 78756.984375$ memory_percent‚88.14609050750732 proc.cpu.threads‚17$ disk./.usageGB‚549.5755043029785 disk.rbd3.in‚0ÂÞaÕž‚(:“ úüèÒ¦³ôö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚41.0 gpu.0.powerWatts‚279.255' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚55.85099999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ãG_O‚‚(:û úüèÒÇ—«ù# disk.nvme0c0n1.in‚ 18312.5703125 disk.sda2.out‚0) proc.memory.percent‚17.96562671661377$ disk./.usageGB‚549.6333961486816 disk.rbd0.out‚0% disk.nvme0n1.out‚80890.5634765625# proc.memory.rssMB‚ 2943.48828125 proc.cpu.threads‚17" disk.dm-0.out‚80890.5634765625! disk.nvme0n1.in‚ 18312.5703125 disk.dm-1.in‚0 disk.rbd0.in‚0 network.sent‚ 1.0716855e+07 disk.dm-1.out‚ 459.85546875 disk.drbd2165.out‚0' disk.nvme0c0n1.out‚80890.5634765625) disk./.usagePercent‚36.88635838867465 disk.dm-0.in‚ 18312.5703125 disk.sda1.out‚0 disk.sda.out‚ 459.85546875 disk.sda1.in‚0 disk.sda3.out‚ 459.85546875 disk.drbd2165.in‚0 disk.sda2.in‚0 disk.sda.in‚0' proc.memory.availableMB‚ 2717.671875 cpu‚25.551251392782657 network.recv‚ 2.343864e+06 disk.rbd3.out‚0 disk.rbd1.in‚ 0.01171875 disk.rbd1.out‚ 22.8818359375$ memory_percent‚83.41264724731445 disk.sda3.in‚0 disk.rbd3.in‚0‰Àוƒ‚(:Š ‰ýèÒˆã«÷ gpu.0.gpu‚100.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚331.878' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚66.3756 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Écéï „‚(:ÿ ‰ýèÒ‚Šµù disk.dm-1.in‚0) disk./.usagePercent‚36.88679948965615" disk.nvme0n1.in‚18312.98046875 disk.rbd1.out‚ 22.8818359375( proc.memory.availableMB‚ 2717.5234375 proc.cpu.threads‚17 disk.drbd2165.in‚0 disk.rbd3.in‚0$ memory_percent‚83.41355323791504 disk.sda3.out‚ 460.16015625 disk.sda.out‚ 460.16015625 cpu‚25.551035110429957$ disk./.usageGB‚549.6399688720703 disk.sda2.in‚0 disk.dm-1.out‚ 460.16015625 network.sent‚ 1.0721869e+07 disk.sda3.in‚0 disk.rbd0.out‚0$ disk.nvme0c0n1.in‚18312.98046875% disk.nvme0n1.out‚80906.0224609375 disk.rbd1.in‚ 0.01171875" disk.dm-0.out‚80906.0224609375# proc.memory.rssMB‚ 2943.48828125) proc.memory.percent‚17.96562671661377 network.recv‚ 2.347228e+06 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda.in‚0 disk.rbd0.in‚0 disk.dm-0.in‚18312.98046875 disk.drbd2165.out‚0 disk.rbd3.out‚0' disk.nvme0c0n1.out‚80906.0224609375 disk.sda2.out‚0¢lÏì…‚(:’ ˜ýèÒÔÜ™÷ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚302.16' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚60.43200000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0J Ìü†‚(:ñ ˜ýèÒºÛÍø disk.rbd1.out‚ 22.9443359375 network.sent‚ 1.0726926e+07# disk./.usageGB‚549.644645690918 disk.dm-1.in‚0 disk.drbd2165.in‚0 disk.rbd3.out‚0) proc.memory.availableMB‚ 2717.44921875 network.recv‚ 2.350291e+06 disk.sda3.in‚0 disk.rbd1.in‚ 0.01171875# proc.memory.rssMB‚ 2943.48828125 cpu‚25.550717841861637 disk.sda1.out‚0 disk.rbd3.in‚0% disk.nvme0c0n1.out‚80915.55078125 disk.sda1.in‚0 disk.drbd2165.out‚0) proc.memory.percent‚17.96562671661377$ disk.nvme0c0n1.in‚18380.60546875" disk.nvme0n1.in‚18380.60546875 disk.rbd0.out‚0 disk.dm-0.in‚18380.60546875 disk.sda3.out‚ 460.71875$ memory_percent‚83.41400623321533 disk.sda.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-0.out‚80915.55078125 disk.sda.out‚ 460.71875 disk.dm-1.out‚ 460.71875 disk.rbd0.in‚0 proc.cpu.threads‚17* disk./.usagePercent‚36.887113354893124# disk.nvme0n1.out‚80915.550781256(ž”‡‚(:‰ §ýèÒ‚ Ž÷ gpu.0.gpu‚90.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚292.788' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚58.5576 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Þ¥püˆ‚(:ù §ýèÒ®ÍÉù network.sent‚ 1.0731949e+07 network.recv‚ 2.353716e+06) disk./.usagePercent‚36.88775874988694 disk.dm-1.in‚0# proc.memory.rssMB‚ 2943.48828125) proc.memory.percent‚17.96562671661377 proc.cpu.threads‚17 disk.dm-1.out‚ 461.265625 disk.rbd0.out‚0" disk.nvme0n1.in‚18381.02734375 disk.sda2.in‚0 disk.sda3.out‚ 461.265625 disk.rbd1.out‚ 22.9443359375 disk.sda.out‚ 461.265625 disk.dm-0.in‚18381.02734375 disk.rbd3.in‚0$ disk.nvme0c0n1.in‚18381.02734375 disk.sda1.in‚0 disk.drbd2165.in‚0$ memory_percent‚83.41388702392578& proc.memory.availableMB‚ 2717.46875# disk.dm-0.out‚80926.56201171875 disk.sda2.out‚0 disk.sda.in‚0 disk.drbd2165.out‚0 cpu‚25.55049800296283$ disk./.usageGB‚549.6542625427246 disk.rbd1.in‚ 0.01171875 disk.rbd0.in‚0 disk.sda1.out‚0 disk.sda3.in‚0 disk.rbd3.out‚0( disk.nvme0c0n1.out‚80926.56201171875& disk.nvme0n1.out‚80926.56201171875_,GŸ‰‚(:” ¶ýèÒ¿ƒöö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚292.983' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚58.596599999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 y ³Š‚(:ý ¶ýèÒŸ«þø" disk.dm-0.out‚80938.8310546875 disk.rbd0.out‚0 disk.rbd1.out‚ 23.0068359375) proc.memory.availableMB‚ 2717.46484375) proc.memory.percent‚17.96562671661377 disk.dm-1.out‚ 461.4921875' disk.nvme0c0n1.out‚80938.8310546875 network.recv‚ 2.356792e+06 disk.rbd0.in‚0$ disk.nvme0c0n1.in‚18381.23046875 disk.sda2.out‚0 proc.cpu.threads‚17 disk.sda.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 network.sent‚ 1.0736465e+07 disk.dm-1.in‚0 disk.dm-0.in‚18381.23046875# proc.memory.rssMB‚ 2943.48828125$ disk./.usageGB‚549.6583938598633 disk.sda3.out‚ 461.4921875 disk.rbd3.out‚0% disk.nvme0n1.out‚80938.8310546875 disk.rbd1.in‚ 0.01171875$ memory_percent‚83.41391086578369) disk./.usagePercent‚36.88803600604653 disk.sda.out‚ 461.4921875 disk.drbd2165.out‚0" disk.nvme0n1.in‚18381.23046875 disk.sda2.in‚0 cpu‚25.550181371574265 disk.sda3.in‚0 disk.drbd2165.in‚0 disk.rbd3.in‚0~U;Û2‹- _wandbruntime‚9382  _runtime‚9382E•ÆŒÀ & lr-AdamW/pg2‚0.0005764709904784398 trainer/global_step‚5649 $ _timestamp‚1.784299198610545e+09 _runtime‚9382.827727967 _step‚462 & lr-AdamW/pg1‚0.0005764709904784398Í/ËÅ _step‚462 & lr-AdamW/pg1‚0.0005764709904784398 & lr-AdamW/pg2‚0.0005764709904784398 trainer/global_step‚5649 $ _timestamp‚1.784299198610545e+09 _runtime‚9382.827727967ÎâÌ2Ž- _wandbruntime‚9382  _runtime‚9382ú½w°ý'÷O IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1624383341289472 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚30541605 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1623681285582848 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 trainer/global_step‚5649 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚14497001 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16044604 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚48216349 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚48942027 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16044832 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5305696400864 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1629698133159424 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5316137259520 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6852186739712 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚48216355 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚228 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5316137259520 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1624383367462912 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚48932054 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1629699504722432 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1624383367462912 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚14497006 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5307041001186 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1629698133159424 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1624383341289472 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚48216349 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1629699504722432 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1628988326584034 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚445515712947200 _step‚463 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚97148403 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6852219179008 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5314791869952 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚438663526207488 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚32439296 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 $ _timestamp‚1.784299198611742e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚438663894298624 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚48942027 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚30541838fbMÍ!üO XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16044832 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5305696400864 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1629698133159424 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5316137259520 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6852186739712 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚48216355 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚228 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5316137259520 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1624383367462912 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚48932054 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1629699504722432 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1624383367462912 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚14497006 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5307041001186 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1629698133159424 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1624383341289472 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚48216349 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1629699504722432 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1628988326584034 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚445515712947200 _step‚463 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚97148403 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6852219179008 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5314791869952 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚438663526207488 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚32439296 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 $ _timestamp‚1.784299198611742e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚438663894298624 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚48942027 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚30541838 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1624383341289472 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚30541605 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1623681285582848 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 trainer/global_step‚5649 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚14497001 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16044604 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚48216349 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚48942027Ïš„âJ2‘- _wandbruntime‚9384  _runtime‚9384‘þ¶' '’ƒN TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1623959141933056 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1624661342019584 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚97165087 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚445594814090240 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5307964467544 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚48950519 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚30546907 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16047363 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚14499544 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚438741380706304 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5317062302208 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚48224561 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16047590 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6853433383936 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚30547139 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚48940532 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1624661368193024 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚438741012615168 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1629265761800166 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 % _timestamp‚1.7842991999993834e+09 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚97175080 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1629978430495232 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1629977058925056 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚48224555 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚32432128 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚97175080 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5315716905472 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚227 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚97165087 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 _runtime‚9384.216853472 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 400523264 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5306619867110 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1629978430495232 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16047363 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚14499544 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚438741380706304 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5317062302208 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚48224561 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16047590 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6853433383936 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚30547139 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚48940532 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1624661368193024 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚438741012615168 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1629265761800166 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 % _timestamp‚1.7842991999993834e+09 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚97175080 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1629978430495232 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1629977058925056 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚48224555 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚32432128 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚97175080 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5315716905472 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚227 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚97165087 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 _runtime‚9384.216853472 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 400523264 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5306619867110 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1629978430495232 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚48950519 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚30546907 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627ÐI•2”- _wandbruntime‚9384  _runtime‚9384Û³Ÿn•‚ 0loss/FRt2/assignment_loss‚0.6087966233568118 2loss/SRqqt1/assignment_loss‚0.5238506856892161 3loss/SRqqt2/assignment_loss‚0.49487205384927585 3loss/FBt2/assignment_loss‚0.0011353354471747678 .loss/FRt1/detection_loss‚0.531790813782539 1loss/SRqqt1/detection_loss‚0.8713697181541532 %loss/total_loss‚8.022141259698007 _runtime‚9384.303096054 epoch‚14 trainer/global_step‚5649 _step‚465 .loss/FRt2/detection_loss‚1.502750780774028 .loss/FBt2/detection_loss‚0.516128757455486 0loss/FRt1/assignment_loss‚1.6053963546336099 1loss/FBt1/assignment_loss‚0.00944091149851399 1loss/SRqqt2/detection_loss‚0.6543269337932829 /loss/FBt1/detection_loss‚0.7022822912639151 % _timestamp‚1.7842992000861564e+09e˜£ –‡ %loss/total_loss‚8.022141259698007 _runtime‚9384.303096054 epoch‚14 trainer/global_step‚5649 _step‚465 .loss/FRt2/detection_loss‚1.502750780774028 .loss/FBt2/detection_loss‚0.516128757455486 0loss/FRt1/assignment_loss‚1.6053963546336099 1loss/FBt1/assignment_loss‚0.00944091149851399 1loss/SRqqt2/detection_loss‚0.6543269337932829 /loss/FBt1/detection_loss‚0.7022822912639151 % _timestamp‚1.7842992000861564e+09 0loss/FRt2/assignment_loss‚0.6087966233568118 2loss/SRqqt1/assignment_loss‚0.5238506856892161 3loss/SRqqt2/assignment_loss‚0.49487205384927585 3loss/FBt2/assignment_loss‚0.0011353354471747678 .loss/FRt1/detection_loss‚0.531790813782539 1loss/SRqqt1/detection_loss‚0.8713697181541532ѯq”—‚(:‰ ÅýèÒöêö gpu.0.gpu‚90.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚259.638' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚51.9276 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‰¼• ˜‚(:þ ÅýèÒ÷Þïø) disk./.usagePercent‚36.88849297949515 disk.sda1.in‚0# proc.memory.rssMB‚ 2943.48828125 disk.rbd1.in‚ 0.01171875 disk.rbd1.out‚ 23.0068359375" disk.dm-0.out‚80948.4794921875$ memory_percent‚83.41555595397949 disk.sda2.out‚0 disk.dm-1.out‚ 461.86328125 disk.dm-0.in‚18381.44921875 disk.sda3.in‚0 network.recv‚ 2.359915e+06 disk.rbd3.out‚0) proc.memory.percent‚17.96562671661377 cpu‚25.54994276534921 proc.cpu.threads‚17 network.sent‚ 1.0741204e+07 disk.sda.out‚ 461.86328125 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.drbd2165.out‚0$ disk./.usageGB‚549.6652030944824 disk.sda3.out‚ 461.86328125 disk.rbd3.in‚0$ disk.nvme0c0n1.in‚18381.44921875% disk.nvme0n1.out‚80948.4794921875 disk.sda2.in‚0( proc.memory.availableMB‚ 2717.1953125 disk.sda.in‚0' disk.nvme0c0n1.out‚80948.4794921875" disk.nvme0n1.in‚18381.44921875 Ïz”™‚(:‰ ÔýèÒ¨Ôïö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚278.198' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.6396 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0aˆ“§š‚(:ø ÔýèÒÆáýø disk.dm-1.in‚0 disk.dm-0.in‚18464.94140625 disk.sda1.in‚0% disk.nvme0n1.out‚80957.0166015625 disk.sda2.out‚0# proc.memory.rssMB‚ 2943.48828125 proc.cpu.threads‚17 disk.sda.in‚0 disk.rbd0.out‚0 disk.sda3.out‚ 462.078125$ disk.nvme0c0n1.in‚18464.94140625' disk.nvme0c0n1.out‚80957.0166015625 disk.sda2.in‚0 disk.rbd1.out‚ 23.0068359375 disk.sda1.out‚0 network.recv‚ 2.364145e+06$ disk./.usageGB‚549.6723594665527 disk.rbd3.in‚0 disk.rbd0.in‚0 disk.drbd2165.in‚0 disk.rbd3.out‚0# memory_percent‚83.4155797958374" disk.dm-0.out‚80957.0166015625 cpu‚25.549730041743285 disk.sda.out‚ 462.078125 disk.dm-1.out‚ 462.078125 disk.sda3.in‚0 disk.drbd2165.out‚0" disk.nvme0n1.in‚18464.94140625) proc.memory.percent‚17.96562671661377 network.sent‚ 1.080418e+07) disk./.usagePercent‚36.88897324962938 disk.rbd1.in‚ 0.01171875) proc.memory.availableMB‚ 2717.19140625Š_ß6”›‚(:‰ ãýèÒÃã÷ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚278.726' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.7452 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚09 ä œ‚(:‚ ãýèÒ›Ûåø disk.sda3.out‚ 462.41796875" disk.nvme0n1.in‚18465.14453125 disk.rbd1.in‚ 0.01171875 cpu‚25.549440309941428 disk.rbd1.out‚ 23.0849609375) disk./.usagePercent‚36.88924768970609# disk.dm-0.out‚80972.90673828125 disk.rbd3.in‚0 network.sent‚ 1.0809234e+07 disk.sda1.in‚0 disk.drbd2165.in‚0) proc.memory.percent‚17.96562671661377 disk.rbd0.out‚0& disk.nvme0n1.out‚80972.90673828125( proc.memory.availableMB‚ 2717.4296875 disk.sda3.in‚0 disk.sda2.in‚0$ disk./.usageGB‚549.6764488220215 disk.sda.in‚0 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.drbd2165.out‚0 disk.sda2.out‚0 proc.cpu.threads‚17 network.recv‚ 2.367609e+06 disk.sda.out‚ 462.41796875 disk.dm-0.in‚18465.14453125( disk.nvme0c0n1.out‚80972.90673828125# proc.memory.rssMB‚ 2943.48828125 disk.dm-1.out‚ 462.41796875 disk.sda1.out‚0 disk.rbd3.out‚0$ disk.nvme0c0n1.in‚18465.14453125$ memory_percent‚83.41412544250488=ƒzÅ“‚(:ˆ òýèÒÛ·âö gpu.0.gpu‚96.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚335.265' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚67.053 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0D Jž‚(:÷ òýèÒÖ¶‹ù disk.sda1.in‚0 disk.rbd0.out‚0) disk./.usagePercent‚36.88971823155402$ disk./.usageGB‚549.6834602355957 disk.rbd1.out‚ 23.0849609375 disk.sda.out‚ 462.796875 disk.drbd2165.out‚0( disk.nvme0c0n1.out‚80982.72607421875 disk.sda2.in‚0 network.sent‚ 1.0814343e+07 disk.sda2.out‚0$ proc.memory.availableMB‚2719.625 cpu‚25.54920163643544 disk.rbd0.in‚0 network.recv‚ 2.370847e+06& disk.nvme0n1.out‚80982.72607421875# disk.dm-0.out‚80982.72607421875 disk.sda3.out‚ 462.796875$ disk.nvme0c0n1.in‚18465.36328125 disk.sda.in‚0 disk.dm-1.out‚ 462.796875 disk.sda3.in‚0 disk.rbd3.in‚0$ memory_percent‚83.40072631835938# proc.memory.rssMB‚ 2943.48828125) proc.memory.percent‚17.96562671661377 disk.dm-1.in‚0 disk.dm-0.in‚18465.36328125" disk.nvme0n1.in‚18465.36328125 disk.rbd1.in‚ 0.01171875 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.rbd3.out‚0 proc.cpu.threads‚17Éç–ŸŸ‚(:” þèÒ”ÏÚö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚288.103' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚57.620599999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0G6-¹ ‚(:õ þèÒÞµþø network.recv‚ 2.374043e+06 disk.dm-1.in‚0# proc.memory.rssMB‚ 2943.48828125 disk.sda3.in‚0) proc.memory.availableMB‚ 1747.96484375 proc.cpu.threads‚17$ disk./.usageGB‚549.6856536865234 disk.rbd1.in‚ 0.01171875 disk.sda3.out‚ 463.06640625$ disk.nvme0c0n1.out‚ 80997.9765625$ memory_percent‚89.33126926422119) disk./.usagePercent‚36.88986543588621 disk.sda.in‚0 disk.dm-0.out‚ 80997.9765625 disk.drbd2165.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda.out‚ 463.06640625 disk.sda1.out‚0 disk.rbd3.in‚0 disk.rbd3.out‚0 cpu‚25.5489132130578 network.sent‚ 1.0819402e+07 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.dm-0.in‚18465.78515625 disk.sda1.in‚0$ disk.nvme0c0n1.in‚18465.78515625 disk.drbd2165.out‚0" disk.nvme0n1.out‚ 80997.9765625) proc.memory.percent‚17.96562671661377 disk.rbd1.out‚ 23.1474609375 disk.dm-1.out‚ 463.06640625" disk.nvme0n1.in‚18465.785156257ËiN2¡- _wandbruntime‚9461  _runtime‚9461ð²ÿ5Ç¢Á % _timestamp‚1.7842992774096773e+09 _runtime‚9461.626941936 _step‚466 & lr-AdamW/pg1‚0.0005743967394651735 & lr-AdamW/pg2‚0.0005743967394651735 trainer/global_step‚5699O&ÒÌ£Æ & lr-AdamW/pg1‚0.0005743967394651735 & lr-AdamW/pg2‚0.0005743967394651735 trainer/global_step‚5699 % _timestamp‚1.7842992774096773e+09 _runtime‚9461.626941936 _step‚466Ò«R‘2¤- _wandbruntime‚9461  _runtime‚9461w«B)¡¥ùO _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6912416543744 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚449450263135232 _step‚467 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚48626655 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1643645857158656 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1643644485595648 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚14624156 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚14624151 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚449450665762816 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 P@DeviceStatsMonitor.on_train_baœÁÝ^!tch_start/requested_bytes.all.peak‚ 47499471119 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 _runtime‚9461.628208166 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1638283403968512 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1638283377795072 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚49357806 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1642928652605942 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5351933513972 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚48626655 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1637575374491648 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚442537846591488 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1637575349063680 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5361107800576 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚30806058 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚97984455 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1638283377795072 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚49367779 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820®^5Í(¦þO [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5362453190144 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1643644485595648 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5362453190144 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16181902 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚49357806 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5353278114294 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚48626649 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16181656 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 % _timestamp‚1.7842992774108546e+09 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚48626655 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1637575374491648 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚442537846591488 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1637575349063680 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5361107800576 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚30806058 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚97984455 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1638283377795072 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚49367779 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6912416543744 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚449450263135232 _step‚467 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚48626655 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1643645857158656 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1643644485595648 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚14624156 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚14624151 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚449450665762816 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 _runtime‚9461.628208166 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1638283403968512 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1638283377795072 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚49357806 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1642928652605942 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5351933513972 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 IDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚30811079 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚233 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5354201580652 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5352856980218 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1643923411361280 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚98001139 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚442615701090304 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚98011132 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16184385 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1638561404698624 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚30811317 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6913660244992 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1637853256269824 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚449529361335296 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5362032836096 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚49376271 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5363378232832 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚449528958714880 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚48634861 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1637853230841856 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚48634855 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5363378232832 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1643924782931456 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16184618 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚49366284 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 trainer/global_step‚5699 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1638561404698624 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1638561378525184 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚48634855 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6913625715712 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5362032836096 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 eA©‡N SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚49366284 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 trainer/global_step‚5699 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1638561404698624 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1638561378525184 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚48634855 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6913625715712 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5362032836096 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚98001139 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚14626699 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 _step‚468 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚30811079 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚233 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5354201580652 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5352856980218 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1643923411361280 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚98001139 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚442615701090304 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚98011132 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16184385 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1638561404698624 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚30811317 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6913660244992 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1637853256269824 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚449529361335296 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5362032836096 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚49376271 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5363378232832 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚449528958714880 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚48634861 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1637853230841856 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚48634855 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5363378232832 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1643924782931456 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16184618 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0ÔýÆ×2ª- _wandbruntime‚9463  _runtime‚9463Á™½„ «„ %loss/total_loss‚7.744733217374586 trainer/global_step‚5699 /loss/FBt2/detection_loss‚0.4064450017744895 epoch‚14 0loss/FRt1/assignment_loss‚1.6551692614121074 2loss/FBt1/assignment_loss‚0.009879081578727949 /loss/FRt1/detection_loss‚0.5015529548462767 % _timestamp‚1.7842992788839777e+09 _runtime‚9463.100979604 _step‚469 3loss/SRqqt1/assignment_loss‚0.44280025237687526 .loss/FRt2/detection_loss‚1.525700294097598 1loss/SRqqt1/detection_loss‚0.8856019057385531 0loss/FRt2/assignment_loss‚0.7093021125342716 2loss/SRqqt2/assignment_loss‚0.4834359339191086 1loss/FBt2/assignment_loss‚0.03511726036427544 1loss/SRqqt2/detection_loss‚0.6154687300633712 0loss/FBt1/detection_loss‚0.47426042866893114r¾™é¬‰ /loss/FBt2/detection_loss‚0.4064450017744895 epoch‚14 0loss/FRt1/assignment_loss‚1.6551692614121074 2loss/FBt1/assignment_loss‚0.009879081578727949 /loss/FRt1/detection_loss‚0.5015529548462767 % _timestamp‚1.7842992788839777e+09 _runtime‚9463.100979604 _step‚469 3loss/SRqqt1/assignment_loss‚0.44280025237687526 .loss/FRt2/detection_loss‚1.525700294097598 1loss/SRqqt1/detection_loss‚0.8856019057385531 0loss/FRt2/assignment_loss‚0.7093021125342716 2loss/SRqqt2/assignment_loss‚0.4834359339191086 1loss/FBt2/assignment_loss‚0.03511726036427544 1loss/SRqqt2/detection_loss‚0.6154687300633712 0loss/FBt1/detection_loss‚0.47426042866893114 %loss/total_loss‚7.744733217374586 trainer/global_step‚5699Õ烑­‚(:† þèÒƒ”’÷ gpu.0.gpu‚15.0 gpu.0.memory‚5, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚41.0 gpu.0.powerWatts‚300.52' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚60.104 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0±t®‚(:ý þèÒݘ–ù disk.rbd0.out‚0) proc.memory.availableMB‚ 1747.70703125 proc.cpu.threads‚17 disk.rbd1.out‚ 23.1474609375 disk.sda1.out‚0% disk.nvme0n1.out‚81005.5185546875 disk.dm-0.in‚19147.08984375 disk.drbd2165.in‚0 disk.drbd2165.out‚0 network.sent‚ 1.0824419e+07 disk.rbd1.in‚ 0.01171875 disk.rbd0.in‚0 disk.sda1.in‚0 disk.sda3.in‚0 disk.rbd3.in‚0) proc.memory.percent‚17.96562671661377 cpu‚25.548676762612843) disk./.usagePercent‚36.89018006914579 disk.sda.out‚ 463.2578125" disk.nvme0n1.in‚19147.08984375$ memory_percent‚89.33284282684326# proc.memory.rssMB‚ 2943.48828125 disk.sda.in‚0 disk.dm-1.out‚ 463.2578125' disk.nvme0c0n1.out‚81005.5185546875 disk.rbd3.out‚0$ disk.nvme0c0n1.in‚19147.08984375 network.recv‚ 2.377557e+06$ disk./.usageGB‚549.6903419494629 disk.sda2.out‚0 disk.dm-1.in‚0" disk.dm-0.out‚81005.5185546875 disk.sda3.out‚ 463.2578125 disk.sda2.in‚0vcØÍ”¯‚(:‰ ŸþèÒ£ðáö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚271.478' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.2956 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‘Åmy°‚(:ù ŸþèÒæÚÉø disk.rbd3.out‚0 cpu‚25.548392847987785 proc.cpu.threads‚17 disk.sda3.out‚ 463.546875$ disk.nvme0c0n1.in‚19344.95703125" disk.nvme0n1.in‚19344.95703125 disk.sda2.out‚0 disk.sda.out‚ 463.546875 disk.drbd2165.in‚0' disk.nvme0c0n1.out‚81015.9189453125 disk.dm-1.out‚ 463.546875 disk.rbd3.in‚0 disk.sda2.in‚0# proc.memory.rssMB‚ 2943.48828125) proc.memory.percent‚17.96562671661377 network.sent‚ 1.0887433e+07( disk./.usagePercent‚36.8907939752129 disk.sda3.in‚0 disk.drbd2165.out‚0 disk.rbd1.in‚ 0.01171875 disk.dm-1.in‚0) proc.memory.availableMB‚ 1747.87109375 disk.rbd1.out‚ 23.2255859375 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda1.out‚0% disk.nvme0n1.out‚81015.9189453125$ memory_percent‚89.33184146881104 disk.sda.in‚0 network.recv‚ 2.382044e+06$ disk./.usageGB‚549.6994895935059 disk.dm-0.in‚19344.95703125" disk.dm-0.out‚81015.9189453125 disk.sda1.in‚0iv»½ ±‚(:• ®þèҲѫø gpu.0.gpu‚100.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚319.524' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚63.904799999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 _V겂(:ù ®þèÒ‚Ñ×ù$ disk.nvme0c0n1.in‚19345.17578125 network.recv‚ 2.385179e+06$ disk./.usageGB‚549.7040176391602 disk.rbd0.out‚0" disk.dm-0.out‚81027.5224609375 disk.sda3.in‚0 disk.rbd3.out‚0 disk.dm-0.in‚19345.17578125" disk.nvme0n1.in‚19345.17578125# proc.memory.rssMB‚ 2943.48828125) proc.memory.percent‚17.96562671661377 proc.cpu.threads‚17 disk.sda2.in‚0 disk.rbd1.out‚ 23.2255859375 disk.drbd2165.in‚0% disk.nvme0n1.out‚81027.5224609375 disk.sda2.out‚0 disk.sda.in‚0 disk.sda.out‚ 463.6953125 disk.rbd0.in‚0 disk.sda1.out‚0 disk.drbd2165.out‚0 disk.rbd3.in‚0 disk.dm-1.in‚0' disk.nvme0c0n1.out‚81027.5224609375$ memory_percent‚89.33029174804688$ proc.memory.availableMB‚1748.125 cpu‚25.548152797460876* disk./.usagePercent‚36.891097856156044 disk.sda1.in‚0 disk.sda3.out‚ 463.6953125 network.sent‚ 1.0892551e+07 disk.rbd1.in‚ 0.01171875 disk.dm-1.out‚ 463.6953125§þ~ž³‚(:“ ½þèÒÕÑÚö gpu.0.gpu‚98.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚275.487' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚55.09740000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0»JÖ´‚(:û ½þèÒÖ¡ïø network.sent‚ 1.0897029e+07$ disk.nvme0c0n1.in‚19345.37890625 disk.sda2.out‚0 disk.rbd3.in‚0 network.recv‚ 2.388749e+06 disk.sda1.in‚0 disk.rbd0.out‚0) proc.memory.percent‚17.96562671661377 proc.cpu.threads‚17 disk.rbd1.in‚ 0.01171875 cpu‚25.54784551224253 disk.rbd3.out‚0) disk./.usagePercent‚36.89152257265535 disk.dm-0.in‚19345.37890625" disk.dm-0.out‚81036.7587890625' disk.nvme0c0n1.out‚81036.7587890625) proc.memory.availableMB‚ 1748.11328125 disk.sda1.out‚0 disk.drbd2165.in‚0% disk.nvme0n1.out‚81036.7587890625# memory_percent‚89.3303632736206 disk.sda3.out‚ 464.2890625 disk.drbd2165.out‚0 disk.rbd1.out‚ 23.2880859375 disk.dm-1.in‚0 disk.rbd0.in‚0# proc.memory.rssMB‚ 2943.48828125$ disk./.usageGB‚549.7103462219238 disk.sda.in‚0 disk.sda.out‚ 464.2890625 disk.dm-1.out‚ 464.2890625 disk.sda3.in‚0" disk.nvme0n1.in‚19345.37890625 disk.sda2.in‚0„m×3žµ‚(:“ ÌþèÒŒ óö gpu.0.gpu‚100.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚348.82' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚69.76400000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0h2˜ ¶‚(:ÿ ÌþèÒ—ºðø" disk.dm-0.out‚81049.4013671875) disk./.usagePercent‚36.89199823465397 disk.sda3.in‚0 disk.drbd2165.in‚0 disk.rbd3.in‚0 disk.dm-1.out‚ 464.48046875 disk.rbd0.out‚0 disk.sda1.out‚0 disk.rbd1.in‚ 0.01171875 disk.dm-0.in‚19345.80078125$ memory_percent‚89.33184146881104 cpu‚25.54763937574896% disk.nvme0n1.out‚81049.4013671875 disk.sda2.out‚0 proc.cpu.threads‚17 network.recv‚ 2.391912e+06 disk.dm-1.in‚0# proc.memory.rssMB‚ 2943.48828125) proc.memory.percent‚17.96562671661377 disk.sda.out‚ 464.48046875 network.sent‚ 1.0902636e+07$ disk./.usageGB‚549.7174339294434 disk.sda1.in‚0 disk.sda3.out‚ 464.48046875 disk.drbd2165.out‚0' disk.nvme0c0n1.out‚81049.4013671875 disk.sda2.in‚0 disk.rbd1.out‚ 23.2880859375 disk.rbd3.out‚0" disk.nvme0n1.in‚19345.80078125 disk.sda.in‚0 disk.rbd0.in‚0) proc.memory.availableMB‚ 1747.87109375$ disk.nvme0c0n1.in‚19345.80078125}AÝ”·‚(:‰ ÛþèÒÅäþö gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚288.439' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚57.6878 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0^-^¸‚(:ù ÛþèÒî–ýø disk.drbd2165.out‚0$ memory_percent‚89.33331966400146 network.sent‚ 1.090766e+07 network.recv‚ 2.395365e+06) disk./.usagePercent‚36.89257297156834 disk.sda.in‚0 disk.dm-1.out‚ 464.76953125 disk.sda1.out‚0 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.dm-1.in‚0 disk.rbd0.in‚0$ disk.nvme0c0n1.in‚19490.89453125 disk.sda2.in‚0 disk.sda2.out‚0 proc.cpu.threads‚17$ disk./.usageGB‚549.7259979248047 disk.dm-0.out‚81059.94140625) proc.memory.availableMB‚ 1747.62890625 disk.sda3.out‚ 464.76953125" disk.nvme0n1.in‚19490.89453125 disk.rbd1.in‚ 0.01171875 disk.rbd1.out‚ 23.3662109375 cpu‚25.547328884253634 disk.rbd0.out‚0% disk.nvme0c0n1.out‚81059.94140625# disk.nvme0n1.out‚81059.94140625# proc.memory.rssMB‚ 2943.48828125) proc.memory.percent‚17.96562671661377 disk.sda.out‚ 464.76953125 disk.dm-0.in‚19490.89453125 disk.sda1.in‚0 disk.sda3.in‚0 disk.drbd2165.in‚0ŒI¥2¹- _wandbruntime‚9540  _runtime‚9540‡‡'5ǺÁ trainer/global_step‚5749 % _timestamp‚1.7842993560118506e+09 _runtime‚9540.229057344 _step‚470 & lr-AdamW/pg1‚0.0005723078140892991 & lr-AdamW/pg2‚0.0005723078140892991·Yg½Ì»Æ _runtime‚9540.229057344 _step‚470 & lr-AdamW/pg1‚0.0005723078140892991 & lr-AdamW/pg2‚0.0005723078140892991 trainer/global_step‚5749 % _timestamp‚1.7842993560118506e+09Ö…âz2¼- _wandbruntime‚9540  _runtime‚9540ا_çÿ'½ùO ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1651469437972480 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚49036949 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16319415 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚14751301 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 trainer/global_step‚5749 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 _step‚471 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚256 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚49783558 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚453385097676800 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚49783558 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚31070716 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚49793531 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚98830486 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚453384695049216 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5398170627080 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5408769120768 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1651469463400448 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16319666 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚446412535066624 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚49036955 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚49036955 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1656867608599560 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1652183414300672 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚251 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5408769120768 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚49793531 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1657590838031872 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1656868978627850 _runtime‚9540.230322331 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚31070972 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚446412166975488 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5407423731200 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1657592209594880 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1652183414300672 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚49036949 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6972562610176 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚14751306 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5399515227402 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1657590838031872 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1657590838031872 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1656868978627850 _runtime‚9540.230322331 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚31070972 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚446412166975488 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5407423731200 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1657592209594880 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1652183414300672 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚49036949 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6972562610176 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚14751306 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5399515227402 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1657590838031872 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16319415 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚14751301 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 trainer/global_step‚5749 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 _step‚471 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚256 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚49783558 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚453385097676800 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚49783558 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚31070716 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚49793531 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚98830486 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚453384695049216 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5398170627080 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5408769120768 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1651469463400448×´ù"2¿- _wandbruntime‚9541  _runtime‚9541«Ro 'ÀƒN YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5409694163456 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚446490021474304 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1651747345178624 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚98847184 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚31076186 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16322095 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚49045161 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚49045155 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16322337 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1657146413843982 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚49802023 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1652461415030784 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1652461441204224 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5408348766720 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚247 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚453463375598592 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6973722215424 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚98847184 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1652461441204224 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚49045161 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚49792036 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚242 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚446489653383168 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6973756744704 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5409694163456 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚49045155 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚49792036 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1657871135367680 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚98837191 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1657871135367680 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚31075939 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚453463778219008ļÆD'ÁˆN C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚453463375598592 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6973722215424 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚98847184 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1652461441204224 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚49045161 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚49792036 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚242 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚446489653383168 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6973756744704 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5409694163456 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚49045155 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚49792036 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1657871135367680 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚98837191 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1657871135367680 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚31075939 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚453463778219008 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5409694163456 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚446490021474304 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1651747345178624 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚98847184 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚31076186 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16322095 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚49045161 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚49045155 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16322337 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1657146413843982 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚49802023 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1652461415030784 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1652461441204224 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5408348766720 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚247 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820ØÇŠ!2Â- _wandbruntime‚9541  _runtime‚9541†×ìÀ 0loss/FRt1/assignment_loss‚1.5875525332051654 /loss/FBt1/detection_loss‚0.7887905332300922 /loss/FBt2/detection_loss‚0.4212302874031484 trainer/global_step‚5749 $ _timestamp‚1.784299357487079e+09 _runtime‚ 9541.70407026 %loss/total_loss‚8.178319661744242 0loss/FRt2/assignment_loss‚0.5950228900959095 0loss/FBt1/assignment_loss‚0.2211031358407755 /loss/FRt1/detection_loss‚0.5230068719585975 /loss/FRt2/detection_loss‚1.4967375268509688 _step‚473 epoch‚14 2loss/SRqqt1/assignment_loss‚0.4734963576274793 1loss/SRqqt2/assignment_loss‚0.527346133805728 3loss/FBt2/assignment_loss‚0.0034338991698351514 1loss/SRqqt1/detection_loss‚0.9130490150588294 1loss/SRqqt2/detection_loss‚0.6275504774977129ÐÂ<Ñ Ä… 1loss/SRqqt1/detection_loss‚0.9130490150588294 1loss/SRqqt2/detection_loss‚0.6275504774977129 0loss/FRt1/assignment_loss‚1.5875525332051654 /loss/FBt1/detection_loss‚0.7887905332300922 /loss/FBt2/detection_loss‚0.4212302874031484 trainer/global_step‚5749 $ _timestamp‚1.784299357487079e+09 _runtime‚ 9541.70407026 %loss/total_loss‚8.178319661744242 0loss/FRt2/assignment_loss‚0.5950228900959095 0loss/FBt1/assignment_loss‚0.2211031358407755 /loss/FRt1/detection_loss‚0.5230068719585975 /loss/FRt2/detection_loss‚1.4967375268509688 _step‚473 epoch‚14 2loss/SRqqt1/assignment_loss‚0.4734963576274793 1loss/SRqqt2/assignment_loss‚0.527346133805728 3loss/FBt2/assignment_loss‚0.0034338991698351514Ù®Ê>“Å‚(:ˆ êþèÒºÃóö gpu.0.gpu‚99.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚335.505' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚67.101 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0g©ü¡þÆ‚(:ó êþèÒÝÈÌø disk.drbd2165.in‚0$ disk.nvme0c0n1.in‚19491.09765625) proc.memory.availableMB‚ 1747.64453125# proc.memory.rssMB‚ 2943.48828125 network.sent‚ 1.0912716e+07# disk./.usageGB‚549.732837677002 disk.dm-0.out‚ 81067.453125 disk.drbd2165.out‚0 disk.rbd3.out‚0 disk.rbd0.out‚0 disk.sda1.out‚0# disk.nvme0c0n1.out‚ 81067.453125 disk.sda2.out‚0 disk.rbd1.out‚ 23.3662109375 disk.sda2.in‚0) proc.memory.percent‚17.96562671661377 network.recv‚ 2.398595e+06 disk.rbd3.in‚0" disk.nvme0n1.in‚19491.09765625 proc.cpu.threads‚17) disk./.usagePercent‚36.89303199307723 disk.rbd1.in‚ 0.01171875 disk.sda.out‚ 464.95703125 disk.sda3.out‚ 464.95703125$ memory_percent‚89.33322429656982 cpu‚25.547126381619485 disk.sda.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 464.95703125 disk.rbd0.in‚0 disk.dm-0.in‚19491.09765625 disk.sda1.in‚0 disk.sda3.in‚0! disk.nvme0n1.out‚ 81067.453125](‚¡”Ç‚(:‰ ùþèÒ·ðÙö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚266.646' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.3292 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ĺz È‚(:‚ ùþèÒÊÑù disk.rbd3.out‚0 network.recv‚ 2.402851e+06 disk.rbd0.out‚0$ memory_percent‚89.33322429656982 proc.cpu.threads‚17 disk.dm-1.out‚ 465.20703125 disk.sda1.out‚0) proc.memory.percent‚17.96562671661377 disk.sda1.in‚0 disk.drbd2165.out‚0 disk.rbd3.in‚0( disk.nvme0c0n1.out‚81081.26611328125" disk.nvme0n1.in‚19491.51953125 network.sent‚ 1.0975883e+07 disk.rbd0.in‚0 disk.rbd1.out‚ 23.4912109375# disk.dm-0.out‚81081.26611328125& disk.nvme0n1.out‚81081.26611328125$ disk./.usageGB‚549.7396583557129 disk.sda.in‚0 disk.dm-0.in‚19491.51953125$ disk.nvme0c0n1.in‚19491.51953125 disk.sda2.out‚0 disk.rbd1.in‚ 0.01171875 cpu‚25.54683842678919 disk.dm-1.in‚0 disk.sda3.out‚ 465.20703125 disk.drbd2165.in‚0 disk.sda2.in‚0) proc.memory.availableMB‚ 1747.64453125# proc.memory.rssMB‚ 2943.48828125) disk./.usagePercent‚36.89348973454845 disk.sda.out‚ 465.20703125 disk.sda3.in‚0»zĞɂ(:“ ˆÿèÒê«äö gpu.0.gpu‚90.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚297.28' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚59.455999999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0>’*[ Ê‚(:ÿ ˆÿèÒéÁ–ù network.sent‚ 1.0980929e+07 disk.sda.out‚ 465.68359375 disk.dm-1.in‚0" disk.dm-0.out‚81089.4501953125 disk.sda1.in‚0$ disk.nvme0c0n1.in‚19491.73828125 disk.rbd3.out‚0' disk.nvme0c0n1.out‚81089.4501953125) disk./.usagePercent‚36.89380718389091 disk.sda.in‚0 disk.dm-1.out‚ 465.68359375 disk.rbd0.out‚0 disk.sda1.out‚0$ memory_percent‚89.33224678039551# proc.memory.rssMB‚ 2943.48828125$ disk./.usageGB‚549.7443885803223 disk.sda3.in‚0% disk.nvme0n1.out‚81089.4501953125 disk.rbd1.out‚ 23.4912109375 cpu‚25.546611678539573 disk.sda3.out‚ 465.68359375 disk.drbd2165.in‚0 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.dm-0.in‚19491.73828125" disk.nvme0n1.in‚19491.73828125 disk.sda2.in‚0 disk.rbd1.in‚ 0.01171875) proc.memory.percent‚17.96562671661377 disk.sda2.out‚0 network.recv‚ 2.405724e+06 disk.drbd2165.out‚0 disk.rbd3.in‚0( proc.memory.availableMB‚ 1747.80468757_a¸ŸË‚(:” —ÿèÒø gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚289.503' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚57.900600000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ì.< Ì‚(:ö —ÿèÒçÍËø disk.sda3.in‚0 disk.sda3.out‚ 466.015625 disk.rbd0.out‚0) proc.memory.availableMB‚ 1748.04296875 disk.nvme0n1.in‚ 19558.265625$ memory_percent‚89.33079242706299# proc.memory.rssMB‚ 2943.48828125) disk./.usagePercent‚36.89427234958062 disk.drbd2165.out‚0 disk.sda2.in‚0 disk.sda.out‚ 466.015625 proc.cpu.threads‚17$ disk./.usageGB‚549.7513198852539& disk.nvme0n1.out‚81099.68798828125) proc.memory.percent‚17.96562671661377 network.sent‚ 1.0986062e+07 disk.drbd2165.in‚0( disk.nvme0c0n1.out‚81099.68798828125 disk.dm-0.in‚ 19558.265625 network.recv‚ 2.409353e+06 disk.sda1.in‚0 disk.sda1.out‚0# disk.dm-0.out‚81099.68798828125 cpu‚25.54633455366406 disk.rbd3.out‚0" disk.nvme0c0n1.in‚ 19558.265625 disk.rbd1.out‚ 23.5693359375 disk.sda.in‚0 disk.rbd0.in‚0 disk.rbd3.in‚0 disk.sda2.out‚0 disk.rbd1.in‚ 0.01171875 disk.dm-1.in‚0 disk.dm-1.out‚ 466.015625ZéÆõ”Í‚(:‰ ¦ÿèÒ’¾…ø gpu.0.gpu‚91.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚272.007' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.4014 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0J/‹Î‚(:ú ¦ÿèÒ¹’¥ù network.sent‚ 1.0991092e+07 disk.rbd1.out‚ 23.5693359375$ memory_percent‚89.33229446411133) proc.me ë¦Uzmory.percent‚17.96562671661377 disk.dm-0.in‚19558.67578125& disk.nvme0n1.out‚81112.47021484375 disk.rbd0.in‚0$ disk./.usageGB‚549.7041969299316 disk.sda1.out‚0 disk.rbd3.in‚0( disk.nvme0c0n1.out‚81112.47021484375# disk.dm-0.out‚81112.47021484375 disk.sda1.in‚0$ disk.nvme0c0n1.in‚19558.67578125 disk.dm-1.in‚0' proc.memory.availableMB‚ 1747.796875 network.recv‚ 2.412572e+06 disk.sda3.out‚ 466.171875 disk.rbd3.out‚0" disk.nvme0n1.in‚19558.67578125 disk.sda2.out‚0 disk.sda.out‚ 466.171875 disk.rbd0.out‚0 proc.cpu.threads‚17) disk./.usagePercent‚36.89110988851015 disk.sda3.in‚0 disk.sda.in‚0 disk.dm-1.out‚ 466.171875 cpu‚25.54610304042156 disk.drbd2165.out‚0 disk.rbd1.in‚ 0.01171875# proc.memory.rssMB‚ 2943.48828125 disk.drbd2165.in‚0 disk.sda2.in‚0¼Þ¬2Ï- _wandbruntime‚9619  _runtime‚9619Ù6§¦ÇÐÁ & lr-AdamW/pg1‚0.0005702043485185906 & lr-AdamW/pg2‚0.0005702043485185906 trainer/global_step‚5799 % _timestamp‚1.7842994348308532e+09 _runtime‚9619.048020379 _step‚474A™õÌÑÆ trainer/global_step‚5799 % _timestamp‚1.7842994348308532e+09 _runtime‚9619.048020379 _step‚474 & lr-AdamW/pg1‚0.0005702043485185906 & lr-AdamW/pg2‚0.0005702043485185906ÚCö_á2Ò- _wandbruntime‚9619  _runtime‚9619EóÙÿ'ÓùO M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1671537190468096 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1666083450806272 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚49447249 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1666083476979712 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚450286855450624 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚254 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1665363526881280 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚50209310 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚31334444 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5444407740188 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚249 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 trainer/global_step‚5799 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚49447249 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚457319091452928 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 % _timestamp‚1.7842994348319116e+09 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚49447255 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚31334698 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1670809304649758 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚99666538 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚50219283 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚99656559 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚50209310 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚457319494080512 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5455085051392 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1671537190468096 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16455993 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0Lé•ù(ÔþO ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1666083476979712 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5453739661824 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1671537190468096 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16455993 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1671537190468096 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1666083450806272 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚49447249 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1666083476979712 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚450286855450624 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚254 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1665363526881280 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚50209310 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚31334444 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5444407740188 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚249 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 trainer/global_step‚5799 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚49447249 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚457319091452928 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 % _timestamp‚1.7842994348319116e+09 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚49447255 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚31334698 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1670809304649758 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚99666538 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚50219283 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚99656559 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚50209310 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚457319494080512 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5455085051392 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1666361451536384 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5445331206434 trainer/global_step‚5799 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚50217788 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1671817487803904 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1665641434087424 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚31339702 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚450363973767168 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 QDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚50227775 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚49455461 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚49455455 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚14880994 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚49455461 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚99683236 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚14880999 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7033832297472 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1671086739865890 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚457397771535360 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚99683236 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚99673243 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5456010094080 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚450364341858304 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1671088109894292 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1666361451536384 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚99673243 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚50227775 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚240 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚49455455nQ¢Ø ׈N \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1666361477709824 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7033797768192 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚31339947 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1666361451536384 XFDeviceStatsMonitor.UU„…$on_train_batch_end/requested_bytes.small_pool.freed‚ 5445331206434 trainer/global_step‚5799 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚50217788 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1671817487803904 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1665641434087424 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚31339702 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚450363973767168 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 QDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚50227775 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚49455461 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚49455455 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚14880994 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚49455461 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚99683236 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚14880999 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7033832297472 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1671086739865890 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚457397771535360 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚99683236 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚99673243 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5456010094080 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚450364341858304 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1671088109894292 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1666361451536384 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚99673243 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚50227775 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚240 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚49455455 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1666361477709824 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5454664697344 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987ÜJG*T2Ø- _wandbruntime‚9620  _runtime‚9620/O Ù 0loss/FRt2/assignment_loss‚0.7546866963549358 /loss/FBt1/detection_loss‚0.8154473681291886 .loss/FBt2/detection_loss‚0.453523757814513 %loss/total_loss‚8.131504413900664 epoch‚14 trainer/global_step‚5799 2loss/SRqqt1/assignment_loss‚0.4855034948383569 2loss/SRqqt2/assignment_loss‚0.5444878917154528 /loss/FRt1/detection_loss‚0.4873238974351721 _runtime‚9620.520460164 1loss/FBt1/assignment_loss‚0.01347211131339384 2loss/FBt2/assignment_loss‚0.031558533571237286 % _timestamp‚1.7842994363034937e+09 _step‚477 0loss/FRt1/assignment_loss‚1.5568392395752855 /loss/FRt2/detection_loss‚1.5283582977836998 0loss/SRqqt1/detection_loss‚0.840455658489056 1loss/SRqqt2/detection_loss‚0.6198474668803727ßÖÑ9 Ú† 1loss/FBt1/assignment_loss‚0.01347211131339384 2loss/FBt2/assignment_loss‚0.031558533571237286 % _timestamp‚1.7842994363034937e+09 _step‚477 0loss/FRt1/assignment_loss‚1.5568392395752855 /loss/FRt2/detection_loss‚1.5283582977836998 0loss/SRqqt1/detection_loss‚0.840455658489056 1loss/SRqqt2/detection_loss‚0.6198474668803727 0loss/FRt2/assignment_loss‚0.7546866963549358 /loss/FBt1/detection_loss‚0.8154473681291886 .loss/FBt2/detection_loss‚0.453523757814513 %loss/total_loss‚8.131504413900664 epoch‚14 trainer/global_step‚5799 2loss/SRqqt1/assignment_loss‚0.4855034948383569 2loss/SRqqt2/assignment_loss‚0.5444878917154528 /loss/FRt1/detection_loss‚0.4873238974351721 _runtime‚9620.520460164Ýüß¿’Û‚(:‡ µÿèÒáØúö gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚287.91' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚57.582 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0í ê Ü‚(:‚ µÿèÒ›ªù# memory_percent‚89.3326997756958 proc.cpu.threads‚17* disk./.usagePercent‚36.891568398003976 disk.rbd3.in‚0$ disk.nvme0c0n1.in‚19558.89453125 disk.sda.out‚ 466.50390625 disk.rbd0.in‚0# proc.memory.rssMB‚ 2943.48828125 cpu‚25.545849701893747" disk.nvme0n1.in‚19558.89453125 disk.sda2.out‚0 disk.rbd1.in‚ 0.01171875 disk.dm-1.in‚0 network.sent‚ 1.099624e+07 disk.sda1.out‚0 disk.drbd2165.in‚0& disk.nvme0n1.out‚81124.99462890625 disk.rbd1.out‚ 23.6318359375 disk.dm-1.out‚ 466.50390625) proc.memory.percent‚17.96562671661377 network.recv‚ 2.416067e+06 disk.rbd3.out‚0 disk.rbd0.out‚0 disk.sda1.in‚0 disk.sda2.in‚0) proc.memory.availableMB‚ 1747.73046875 disk.sda3.in‚0 disk.drbd2165.out‚0 disk.dm-0.in‚19558.89453125( disk.nvme0c0n1.out‚81124.99462890625$ disk./.usageGB‚549.7110290527344 disk.sda3.out‚ 466.50390625 disk.sda.in‚0# disk.dm-0.out‚81124.99462890625—òʦ”Ý‚(:‰ ÄÿèÒ¶óö gpu.0.gpu‚90.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚250.749' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚50.1498 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0“`ÃAýÞ‚(:ò ÄÿèÒ–†€ù network.recv‚ 2.42009e+06 disk.sda1.out‚0$ disk.nvme0c0n1.in‚19559.31640625 disk.rbd1.in‚ 0.01171875 disk.sda3.out‚466.6875 disk.sda2.out‚0( proc.memory.availableMB‚ 1747.1484375) disk./.usagePercent‚36.89203458772383 disk.dm-0.in‚19559.31640625 disk.sda3.in‚0 disk.sda.in‚0 network.sent‚ 1.1059211e+07 disk.dm-1.in‚0 disk.rbd0.in‚0 proc.cpu.threads‚17 disk.drbd2165.out‚0 disk.sda.out‚466.6875# proc.memory.rssMB‚ 2943.48828125$ disk./.usageGB‚549.7179756164551" disk.dm-0.out‚81137.5947265625 disk.sda2.in‚0$ memory_percent‚89.33625221252441 disk.dm-1.out‚466.6875 disk.rbd3.out‚0' disk.nvme0c0n1.out‚81137.5947265625% disk.nvme0n1.out‚81137.5947265625 disk.rbd1.out‚ 23.6318359375 disk.sda1.in‚0 disk.drbd2165.in‚0 disk.rbd3.in‚0" disk.nvme0n1.in‚19559.31640625 disk.rbd0.out‚0) proc.memory.percent‚17.96562671661377 cpu‚25.545599403970485,²|W’ß‚(:‡ ÓÿèÒðÈéö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚287.91' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚57.582 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0×r½k à‚(:ÿ ÓÿèÒ´ûìû disk.rbd0.out‚0 disk.rbd3.out‚0 disk.sda2.out‚0 disk.dm-1.in‚0$ memory_percent‚89.33758735656738$ disk./.usageGB‚549.7224426269531 disk.sda3.out‚ 467.15234375 disk.sda2.in‚0 disk.sda.in‚0( proc.memory.availableMB‚ 1746.9296875# proc.memory.rssMB‚ 2943.48828125 network.sent‚ 1.1064273e+07 disk.rbd1.out‚ 23.7099609375 disk.sda.out‚ 467.15234375 disk.sda1.in‚0 disk.sda3.in‚0& disk.nvme0n1.out‚81146.11083984375# disk.dm-0.out‚81146.11083984375 disk.sda1.out‚0 disk.rbd3.in‚0! disk.nvme0n1.in‚ 19660.1484375 disk.dm-0.in‚ 19660.1484375( disk.nvme0c0n1.out‚81146.11083984375 proc.cpu.threads‚17) disk./.usagePercent‚36.89233437254643 disk.drbd2165.in‚0 disk.drbd2165.out‚0# disk.nvme0c0n1.in‚ 19660.1484375 disk.dm-1.out‚ 467.15234375 network.recv‚ 2.423505e+06 disk.rbd0.in‚0 disk.rbd1.in‚ 0.01171875) proc.memory.percent‚17.96562671661377 cpu‚25.545383897953965þµxˆá‚(:’ âÿèÒ¡§ÿö gpu.0.gpu‚98.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚283.03' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚56.60599999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0}Š%Ÿâ‚(:û âÿèÒˆ÷¯ù disk.rbd3.in‚0 disk.rbd0.in‚0# proc.memory.rssMB‚ 2943.48828125) proc.memory.percent‚17.96562671661377$ disk./.usageGB‚549.7314758300781 disk.sda3.out‚ 467.453125 disk.sda2.in‚0 network.sent‚ 1.1069273e+07* disk./.usagePercent‚36.892940598387504 disk.sda1.out‚0 disk.drbd2165.out‚0 disk.sda1.in‚0 disk.sda3.in‚0 disk.rbd3.out‚0( disk.nvme0c0n1.out‚81156.63720703125 disk.rbd1.out‚ 23.7099609375 disk.sda.in‚0 disk.dm-1.out‚ 467.453125 network.recv‚ 2.426652e+06 disk.drbd2165.in‚0! disk.nvme0n1.in‚ 19660.5703125 disk.dm-1.in‚0$ memory_percent‚89.34228420257568 cpu‚25.545121148068674 disk.rbd0.out‚0 disk.dm-0.in‚ 19660.5703125# disk.nvme0c0n1.in‚ 19660.5703125 disk.sda.out‚ 467.453125# disk.dm-0.out‚81156.63720703125& disk.nvme0n1.out‚81156.63720703125 disk.sda2.out‚0 disk.rbd1.in‚ 0.01171875) proc.memory.availableMB‚ 1746.16015625 proc.cpu.threads‚17¦\ D”ã‚(:‰ ñÿèÒàÔÖö gpu.0.gpu‚61.0 gpu.0.memory‚19, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚39.0 gpu.0.powerWatts‚110.656' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.1312 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0»â ä‚(:€ ñÿèÒø§™ú disk.dm-1.out‚ 467.75390625# disk.dm-0.out‚81169.14990234375! disk.nvme0n1.in‚ 19660.7890625$ disk./.usageGB‚549.7368659973145 disk.sda3.in‚0 disk.rbd3.out‚0 disk.rbd1.out‚ 23.7724609375 disk.sda.in‚0 disk.rbd3.in‚0& disk.nvme0n1.out‚81169.14990234375 disk.rbd0.out‚0# disk.nvme0c0n1.in‚ 19660.7890625( disk.nvme0c0n1.out‚81169.14990234375 disk.rbd1.in‚ 0.01171875 disk.sda.out‚ 467.75390625$ memory_percent‚89.33937549591064) proc.memory.percent‚17.96562671661377 disk.drbd2165.out‚0) proc.memory.availableMB‚ 1746.63671875 disk.dm-0.in‚ 19660.7890625 disk.sda1.out‚0 network.recv‚ 2.429214e+06) disk./.usagePercent‚36.89330233703338 network.sent‚ 1.1073422e+07 disk.drbd2165.in‚0 disk.rbd0.in‚0 disk.sda3.out‚ 467.75390625 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda1.in‚0 disk.dm-1.in‚0# proc.memory.rssMB‚ 2943.48828125 cpu‚25.544866887801113 proc.cpu.threads‚17`°ïš2å- _wandbruntime‚9697  _runtime‚9697òmÇæÁ _step‚478 & lr-AdamW/pg1‚0.0005680864778547103 & lr-AdamW/pg2‚0.0005680864778547103 trainer/global_step‚5849 % _timestamp‚1.7842995136703506e+09 _runtime‚9697.887492427ëOÛÌçÆ & lr-AdamW/pg2‚0.0005680864778547103 trainer/global_step‚5849 % _timestamp‚1.7842995136703506e+09 _runtime‚9697.887492427 _step‚478 & lr-AdamW/pg1‚0.0005680864778547103ÞâD£ä2è- _wandbruntime‚9697  _runtime‚9697mc\(éýO L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚49857549 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1684748260643376 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚49857555 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1679983487311872 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1685484914467328 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 trainer/global_step‚5849 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚461253308127232 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5500055592448 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚259 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚264 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1684749630671666 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚50635062 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1679983487311872 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7092500383744 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5490644853296 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚49857555 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1679257641218048 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1685483542904320 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 _runtime‚9697.888905685 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 100492611 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 100502590 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 % _timestamp‚1.7842995136714373e+09 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1684748260643376 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚49857555 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1679983487311872 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1685484914467328 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 trainer/global_step‚5849 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚461253308127232 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5500055592448 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.F €¤äpeak‚ 1680116736 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚259 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚264 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1684749630671666 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚50635062 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1679983487311872 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7092500383744 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5490644853296 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚49857555 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1679257641218048 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1685483542904320 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 _runtime‚9697.888905685 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 100492611 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 100502590 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 % _timestamp‚1.7842995136714373e+09 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 J8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5502326024704 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 100509295 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 100509295 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚50653527 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1680261514215424 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 100519288 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚49865755 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1685763840240128 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1679535497568256 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚50653527 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚31603932 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1685028435916200 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1680261488041984 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1685763840240128 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚50643540 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚5849 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚49865761 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1679535522996224 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 _runtime‚ 9699.27547132 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚49865761 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚461332394703872 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚15008144 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚461331992083456 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7093697932288 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚31603681 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1685762468669952 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚15008149 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7093732461568 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5502326024704 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5500980627968 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚246 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5500980627968 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 _step‚480 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1685027065887798 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚50643540 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16595783 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5491568319542 % _timestamp‚1.7842995150579903e+09 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0§öû+'í‹N XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1680261488041984 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1685763840240128 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚50643540 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚5849 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚49865761 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1679535522996224 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 _runtime‚ 9699.27547132 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚49865761 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚461332394703872 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚15008144 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚461331992083456 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7093697932288 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚31603681 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1685762468669952 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚15008149 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7093732461568 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5502326024704 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5500980627968 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚246 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5500980627968 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 _step‚480 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1685027065887798 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚50643540 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16595783 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5491568319542 % _timestamp‚1.7842995150579903e+09 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5502326024704 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 100509295 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 100509295 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚50653527 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1680261514215424 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 100519288 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚49865755 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1685763840240128 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1679535497568256 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚50653527 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚31603932 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1685028435916200àjÌÑ2î- _wandbruntime‚9699  _runtime‚9699¶q')ï‚ epoch‚14 0loss/FRt1/assignment_loss‚1.5631445086919344 0loss/FRt2/assignment_loss‚0.8083232947297998 2loss/SRqqt1/assignment_loss‚0.5491119599938082 1loss/FBt2/assignment_loss‚0.03133446663157554 1loss/SRqqt1/detection_loss‚0.8166352164204991 % _timestamp‚1.7842995151437001e+09 2loss/SRqqt2/assignment_loss‚0.5229192875775771 1loss/SRqqt2/detection_loss‚0.6377638632439748 /loss/FBt1/detection_loss‚0.9104363789048159 _step‚481 /loss/FRt2/detection_loss‚1.5183241235421792 /loss/FBt2/detection_loss‚0.4760463981134817 %loss/total_loss‚8.394863991488108 trainer/global_step‚5849 _runtime‚9699.360642279 1loss/FBt1/assignment_loss‚0.01541163250795197 /loss/FRt1/detection_loss‚0.5454128611305106"/Ý3 ð‡ epoch‚14 0loss/FRt1/assignment_loss‚1.5631445086919344 0loss/FRt2/assignment_loss‚0.8083232947297998 2loss/SRqqt1/assignment_loss‚0.5491119599938082 1loss/FBt2/assignment_loss‚0.03133446663157554 1loss/SRqqt1/detection_loss‚0.8166352164204991 % _timestamp‚1.7842995151437001e+09 2loss/SRqqt2/assignment_loss‚0.5229192875775771 1loss/SRqqt2/detection_loss‚0.6377638632439748 /loss/FBt1/detection_loss‚0.9104363789048159 _step‚481 /loss/FRt2/detection_loss‚1.5183241235421792 /loss/FBt2/detection_loss‚0.4760463981134817 %loss/total_loss‚8.394863991488108 trainer/global_step‚5849 _runtime‚9699.360642279 1loss/FBt1/assignment_loss‚0.01541163250795197 /loss/FRt1/detection_loss‚0.5454128611305106áo䯟ñ‚(:” €€éÒúק÷ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚297.863' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚59.572599999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0=îÞùò‚(:ý €€éÒ»«œù' disk.nvme0c0n1.out‚81178.7841796875! disk.nvme0n1.in‚ 19660.9921875 disk.sda2.out‚0 disk.sda.in‚0 disk.dm-1.out‚ 468.20703125 disk.rbd0.out‚0 network.sent‚ 1.1078436e+07% disk.nvme0n1.out‚81178.7841796875 disk.sda.out‚ 468.20703125" disk.dm-0.out‚81178.7841796875# proc.memory.rssMB‚ 2943.48828125 proc.cpu.threads‚17) disk./.usagePercent‚36.89376110253475 disk.drbd2165.out‚0 disk.rbd1.out‚ 23.7724609375$ memory_percent‚89.34042453765869 disk.sda3.out‚ 468.20703125 disk.sda2.in‚0 disk.rbd1.in‚ 0.01171875) proc.memory.percent‚17.96562671661377 disk.sda3.in‚0 disk.drbd2165.in‚0 disk.rbd3.out‚0 disk.dm-1.in‚0 disk.sda1.out‚0 disk.rbd3.in‚0# disk.nvme0c0n1.in‚ 19660.9921875 disk.rbd0.in‚0 disk.dm-0.in‚ 19660.9921875 cpu‚25.544623074694943 network.recv‚ 2.432278e+06) proc.memory.availableMB‚ 1746.46484375$ disk./.usageGB‚549.7437019348145 disk.sda1.in‚03- ²”ó‚(:‰ €éÒÞÞœ÷ gpu.0.gpu‚49.0 gpu.0.memory‚15, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚40.0 gpu.0.powerWatts‚110.527' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.1054 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‡ÃÜ:ô‚(:ý €éÒ—•Ûø disk.rbd3.out‚0 disk.sda.in‚0 disk.dm-1.out‚ 468.5078125 disk.sda1.out‚0 disk.sda3.in‚0 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.sda1.in‚0" disk.nvme0n1.in‚19749.28515625& disk.nvme0n1.out‚81188.25927734375 disk.rbd3.in‚0$ memory_percent‚89.34149742126465( proc.memory.availableMB‚ 1746.2890625# proc.memory.rssMB‚ 2943.48828125) disk./.usagePercent‚36.89423958061624 disk.sda.out‚ 468.5078125 disk.drbd2165.in‚0 disk.drbd2165.out‚0 network.recv‚ 2.43683e+06( disk.nvme0c0n1.out‚81188.25927734375 disk.sda2.in‚0 disk.rbd1.in‚ 0.01171875# disk.dm-0.out‚81188.25927734375 disk.sda3.out‚ 468.5078125 network.sent‚ 1.1141546e+07$ disk./.usageGB‚549.7508316040039 disk.dm-1.in‚0 disk.dm-0.in‚19749.28515625) proc.memory.percent‚17.96562671661377 cpu‚25.54440321318497$ disk.nvme0c0n1.in‚19749.28515625 disk.sda2.out‚0 disk.rbd1.out‚ 23.8505859375 disk.rbd0.out‚0áä”õ‚(:‰ ž€éÒýÏ—÷ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚266.358' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.2716 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÝóÑö‚(:ø ž€éÒç–ìø network.recv‚ 2.44012e+06 disk.sda3.out‚ 468.65625 disk.rbd3.out‚0& disk.nvme0n1.out‚81204.69384765625) proc.memory.percent‚17.96562671661377 network.sent‚ 1.1146601e+07) disk./.usagePercent‚36.89454909372514 disk.drbd2165.out‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-1.out‚ 468.65625( disk.nvme0c0n1.out‚81204.69384765625" disk.nvme0n1.in‚19749.50390625 disk.rbd1.out‚ 23.8505859375 disk.sda.out‚ 468.65625# proc.memory.rssMB‚ 2943.48828125 disk.dm-0.in‚19749.50390625) proc.memory.availableMB‚ 1746.49609375# disk./.usageGB‚549.755443572998 disk.sda1.in‚0 disk.rbd3.in‚0 disk.rbd0.out‚0 disk.dm-1.in‚0 disk.rbd0.in‚0 proc.cpu.threads‚17 disk.sda1.out‚0$ disk.nvme0c0n1.in‚19749.50390625# disk.dm-0.out‚81204.69384765625$ memory_percent‚89.34023380279541 cpu‚25.544129274302932 disk.sda3.in‚0 disk.drbd2165.in‚0 disk.rbd1.in‚ 0.01171875 disk.sda.in‚0Ð>âJ÷‚(:‰ ­€éÒ•»ˆ÷ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚46.0 gpu.0.powerWatts‚341.826' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚68.3652 gpu.0.smClock‚1410 gpu.0.memoryClock‚15‡qÞIJ93! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0„ñ-/ø‚(:ü ­€éÒÊäØø" disk.nvme0n1.in‚19749.70703125 disk.sda2.out‚0 network.recv‚ 2.443664e+06) disk./.usagePercent‚36.89446000310322 disk.rbd1.in‚ 0.01171875 disk.sda.in‚0 disk.sda1.out‚0 disk.sda3.in‚0$ memory_percent‚89.34195041656494 cpu‚25.54390888756564 disk.sda.out‚ 469.140625 disk.dm-1.in‚0 disk.drbd2165.in‚0) proc.memory.availableMB‚ 1746.21484375# proc.memory.rssMB‚ 2943.48828125) proc.memory.percent‚17.96562671661377$ disk.nvme0c0n1.in‚19749.70703125( disk.nvme0c0n1.out‚81214.09619140625 disk.sda2.in‚0 disk.dm-1.out‚ 469.140625 disk.rbd0.in‚0 network.sent‚ 1.1151614e+07# disk.dm-0.out‚81214.09619140625 disk.rbd3.out‚0& disk.nvme0n1.out‚81214.09619140625 disk.rbd0.out‚0 disk.rbd3.in‚0 disk.rbd1.out‚ 23.9130859375 disk.dm-0.in‚19749.70703125 disk.sda1.in‚0 disk.sda3.out‚ 469.140625 proc.cpu.threads‚17 disk.drbd2165.out‚0$ disk./.usageGB‚549.7541160583496”Ÿù‚(:” ¼€éÒ¶„€÷ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚263.406' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚52.681200000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0úïq ú‚(:þ ¼€éÒ£Ÿ›ù% disk.nvme0n1.out‚81231.0224609375 disk.sda2.out‚0 disk.sda1.in‚0$ disk./.usageGB‚549.7619209289551' disk.nvme0c0n1.out‚81231.0224609375 disk.rbd1.in‚ 0.01171875) proc.memory.percent‚17.96562671661377 cpu‚25.543634429852514 proc.cpu.threads‚17" disk.nvme0n1.in‚19750.12890625 disk.sda.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 469.4921875 disk.rbd0.in‚0 disk.rbd0.out‚0 network.sent‚ 1.1156664e+07* disk./.usagePercent‚36.894983794518275" disk.dm-0.out‚81231.0224609375 disk.drbd2165.out‚0# proc.memory.rssMB‚ 2943.48828125 disk.sda1.out‚0$ memory_percent‚89.34214115142822$ disk.nvme0c0n1.in‚19750.12890625 disk.dm-0.in‚19750.12890625 disk.drbd2165.in‚0 disk.rbd3.out‚0 network.recv‚ 2.446766e+06 disk.sda.out‚ 469.4921875 disk.sda3.in‚0 disk.sda3.out‚ 469.4921875 disk.rbd3.in‚0 disk.sda2.in‚0 disk.rbd1.out‚ 23.9130859375) proc.memory.availableMB‚ 1746.18359375òËC2û- _wandbruntime‚9776  _runtime‚9776ŒÚ#8ÇüÁ & lr-AdamW/pg2‚0.0005659543381245339 trainer/global_step‚5899 % _timestamp‚1.7842995922853742e+09 _runtime‚9776.502518795 _step‚482 & lr-AdamW/pg1‚0.0005659543381245339l{”‡ÌýÆ & lr-AdamW/pg2‚0.0005659543381245339 trainer/global_step‚5899 % _timestamp‚1.7842995922853742e+09 _runtime‚9776.502518795 _step‚482 & lr-AdamW/pg1‚0.0005659543381245339ânRè 2þ- _wandbruntime‚9776  _runtime‚9776@7ÈL(ÿüO MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 101328663 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚50267855 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5547716912640 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚51060814 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚31862323 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚458035128127488 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _step‚483 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16729572 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1699429895340544 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚31862584 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚15132756 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚50267849 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1693883523817472 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5546371523072 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5546371523072 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚465187424561664 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1693883523817472 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1699429895340544 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1699431266903552 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚51070787 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚51060814 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚256 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7152296434176 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _step‚483 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16729572 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1699429895340544 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚31862584 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚15132756 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚50267849 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1693883523817472 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5546371523072 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5546371523072 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚465187424561664 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1693883523817472 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1699429895340544 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1699431266903552 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154ãÎÙ 22- _wandbruntime‚9777  _runtime‚9777ðG@°#‚‡N NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚51079279 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚50276061 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚51079279 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7153527463424 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1698967391909706 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5539150033084 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16732291 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚31867835 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1698968761938108 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚15135299 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1699710192676352 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5548641955328 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16732536 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 QDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 101345347 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚250 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1694161550721024 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚31867585 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚245 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1693429586477056 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 101345347 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 101355340 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚15135294 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5548641955328 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 QDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 101345347 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚250 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1694161550721024 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚31867585 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚245 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1693429586477056 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 101345347 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 101355340 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚15135294 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5548641955328 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚51079279 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7153527463424 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402äR@¡í2„- _wandbruntime‚9777  _runtime‚9777¯&#…‚ /loss/FBt2/detection_loss‚0.4011331010466315 epoch‚14 _step‚485 0loss/FRt1/assignment_loss‚1.6694309263716582 2loss/FBt1/assignment_loss‚0.013792832717973108 $loss/total_loss‚7.76849589369569 trainer/global_step‚5899 0loss/FRt2/assignment_loss‚0.6130189962685109 2loss/SRqqt1/assignment_loss‚0.6100459765471752 2loss/SRqqt2/assignment_loss‚0.4504352913229996 2loss/FBt2/assignment_loss‚0.025380239573579627 /loss/FRt1/detection_loss‚0.5073998873257953 % _timestamp‚1.7842995937585828e+09 _runtime‚ 9777.97553388 /loss/FRt2/detection_loss‚1.3841133691510186 1loss/SRqqt1/detection_loss‚0.9374642699231507 1loss/SRqqt2/detection_loss‚0.6209804358280325 /loss/FBt1/detection_loss‚0.5353005676191639`ŒÂ †‡ 0loss/FRt2/assignment_loss‚0.6130189962685109 2loss/SRqqt1/assignment_loss‚0.6100459765471752 2loss/SRqqt2/assignment_loss‚0.4504352913229996 2loss/FBt2/assignment_loss‚0.025380239573579627 /loss/FRt1/detection_loss‚0.5073998873257953 % _timestamp‚1.7842995937585828e+09 _runtime‚ 9777.97553388 /loss/FRt2/detection_loss‚1.3841133691510186 1loss/SRqqt1/detection_loss‚0.9374642699231507 1loss/SRqqt2/detection_loss‚0.6209804358280325 /loss/FBt1/detection_loss‚0.5353005676191639 /loss/FBt2/detection_loss‚0.4011331010466315 epoch‚14 _step‚485 0loss/FRt1/assignment_loss‚1.6694309263716582 2loss/FBt1/assignment_loss‚0.013792832717973108 $loss/total_loss‚7.76849589369569 trainer/global_step‚5899å,CS”‡‚(:‰ Ë€éÒ†Úâø gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚332.701' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚66.5402 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0xþ*cþˆ‚(:ó Ë€éÒƒ¸úù disk.rbd0.out‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd1.in‚ 0.01171875! disk.dm-0.out‚81239.666015625 disk.drbd2165.in‚0 disk.nvme0n1.in‚ 19901.765625 cpu‚25.543410682942177 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.dm-0.in‚ 19901.765625 disk.sda3.out‚ 469.9453125 disk.drbd2165.out‚0 disk.sda3.in‚0& disk.nvme0c0n1.out‚81239.666015625' proc.memory.availableMB‚ 1746.203125 disk.sda1.in‚0$ disk./.usageGB‚549.7768325805664 disk.sda.out‚ 469.9453125) proc.memory.percent‚17.96562671661377 network.recv‚ 2.450227e+06 disk.rbd3.in‚0$ memory_percent‚89.34202194213867# proc.memory.rssMB‚ 2943.48828125 disk.sda1.out‚0 disk.sda.in‚0 disk.rbd3.out‚0 disk.rbd1.out‚ 23.9912109375 proc.cpu.threads‚17* disk./.usagePercent‚36.895984527969624 disk.dm-1.out‚ 469.9453125" disk.nvme0c0n1.in‚ 19901.765625$ disk.nvme0n1.out‚81239.666015625 network.sent‚ 1.1161761e+07K"óÍ”‰‚(:‰ Ú€éÒë×Ýö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚266.062' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.2124 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ØmDí슂(:á Ú€éÒ€´ôø disk.sda3.in‚0 disk.rbd1.in‚ 0.01171875 disk.sda.in‚0 disk.dm-1.out‚ 470.1640625 disk.rbd0.in‚0( proc.memory.availableMB‚ 1745.9140625 disk.drbd2165.in‚0& disk.nvme0n1.out‚81251.91455078125 disk.sda.out‚ 470.1640625 disk.dm-1.in‚0# disk.dm-0.out‚81251.91455078125 disk.sda2.in‚0 disk.drbd2165.out‚0 disk.rbd3.in‚0 disk.nvme0c0n1.in‚19902# disk./.usageGB‚549.793140411377 disk.rbd3.out‚0 disk.sda1.in‚0$ memory_percent‚89.34378623962402# proc.memory.rssMB‚ 2943.48828125 network.sent‚ 1.1224268e+07 disk.nvme0n1.in‚19902 disk.rbd1.out‚ 23.9912109375 disk.rbd0.out‚0 disk.dm-0.in‚19902 network.recv‚ 2.454295e+06( disk./.usagePercent‚36.8970789601785 disk.sda2.out‚0 disk.sda3.out‚ 470.1640625) proc.memory.percent‚17.96562671661377 cpu‚25.54314630784542 proc.cpu.threads‚17( disk.nvme0c0n1.out‚81251.91455078125 disk.sda1.out‚0 %]eŸ‹‚(:” é€éÒà›÷ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚271.258' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚54.251599999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ûÐ!Œ‚(:ö é€éÒ½±±ù network.sent‚ 1.1229906e+07) disk./.usagePercent‚37.01539028205231 disk.dm-1.in‚0 disk.sda3.in‚0) proc.memory.availableMB‚ 1746.37890625) proc.memory.percent‚17.96562671661377 disk.drbd2165.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd0.in‚0 disk.rbd0.out‚0! disk.dm-0.out‚81278.685546875 disk.rbd3.in‚0 disk.sda1.out‚0 disk.sda3.out‚ 473.13671875 disk.drbd2165.out‚0 network.recv‚ 2.457953e+06 disk.rbd1.in‚ 0.01171875" disk.nvme0c0n1.in‚ 19902.390625 disk.rbd1.out‚ 24.0537109375 disk.sda1.in‚0 disk.rbd3.out‚0& disk.nvme0c0n1.out‚81278.685546875 disk.nvme0n1.in‚ 19902.390625 disk.sda.in‚0 disk.dm-0.in‚ 19902.390625 proc.cpu.threads‚17$ disk./.usageGB‚551.5560646057129$ disk.nvme0n1.out‚81278.685546875 disk.sda.out‚ 473.13671875 disk.dm-1.out‚ 473.13671875$ memory_percent‚89.34094905853271# proc.memory.rssMB‚ 2943.48828125 cpu‚25.54289991100484ùàu3”‚(:‰ ø€éÒ•ýˆ÷ gpu.0.gpu‚89.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚327.729' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚65.5458 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0œàê#ùŽ‚(:î ø€éÒ–ÜÂù disk.rbd1.in‚ 0.01171875 disk.sda3.in‚0 disk.drbd2165.in‚0 disk.sda.in‚0 disk.rbd0.out‚0 disk.sda1.in‚0 disk.drbd2165.out‚0$ memory_percent‚89.34390544891357 disk.dm-0.in‚ 19902.609375 network.sent‚ 1.1235045e+07 disk.sda2.in‚0 disk.sda.out‚ 473.79296875 cpu‚25.542732290158433 network.recv‚ 2.461617e+06) proc.memory.availableMB‚ 1745.89453125) proc.memory.percent‚17.96562671661377 disk.nvme0n1.in‚ 19902.609375 disk.dm-1.out‚ 473.79296875 disk.dm-0.out‚ 83005.703125 disk.sda3.out‚ 473.79296875# proc.memory.rssMB‚ 2943.48828125$ disk./.usageGB‚551.4803466796875! disk.nvme0n1.out‚ 83005.703125 proc.cpu.threads‚17 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.sda1.out‚0) disk./.usagePercent‚37.01030878850517 disk.rbd3.in‚0# disk.nvme0c0n1.out‚ 83005.703125 disk.sda2.out‚0 disk.rbd1.out‚ 24.0537109375 disk.rbd3.out‚0" disk.nvme0c0n1.in‚ 19902.609375'lå^Ÿ‚(:” ‡éÒ…ëö gpu.0.gpu‚89.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚242.417' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚48.483399999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¡!…1‚(:ü ‡éÒ¨ñßø disk.sda1.in‚0" disk.nvme0c0n1.in‚ 20067.234375& disk.nvme0n1.out‚83019.88232421875 disk.dm-0.in‚ 20067.234375) proc.memory.availableMB‚ 1745.87890625 disk.sda.in‚0 network.sent‚ 1.1240055e+07# disk./.usageGB‚551.479679107666 disk.sda1.out‚0 disk.nvme0n1.in‚ 20067.234375 disk.rbd1.in‚ 0.01171875 disk.rbd0.in‚0 disk.rbd0.out‚0# proc.memory.rssMB‚ 2943.48828125 disk.drbd2165.in‚0 disk.rbd3.in‚0# disk.dm-0.out‚83019.88232421875 proc.cpu.threads‚17 disk.sda3.in‚0) proc.memory.percent‚17.96562671661377 cpu‚25.542519409001624 network.recv‚ 2.465079e+06( disk.nvme0c0n1.out‚83019.88232421875 disk.sda2.in‚0 disk.rbd1.out‚ 24.1318359375 disk.dm-1.out‚ 474.51171875) disk./.usagePercent‚37.01026398718668 disk.sda3.out‚ 474.51171875 disk.rbd3.out‚0 disk.sda2.out‚0$ memory_percent‚89.34400081634521 disk.drbd2165.out‚0 disk.sda.out‚ 474.51171875 disk.dm-1.in‚0 ‡ÐÆ”‘‚(:‰ –éÒÁ¯ñö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚266.358' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.2716 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0W(Bx’‚(:û –éÒÁôŽù disk.dm-0.in‚ 20115.7890625# disk.dm-0.out‚84750.91455078125! disk.nvme0n1.in‚ 20115.7890625 disk.rbd1.out‚ 24.1318359375) proc.memory.percent‚17.96562671661377$ disk./.usageGB‚551.4822845458984 disk.sda.in‚0 disk.dm-1.out‚ 476.8984375# disk.nvme0c0n1.in‚ 20115.7890625$ memory_percent‚89.34407234191895 network.sent‚ 1.124511e+07 network.recv‚ 2.468304e+06 disk.rbd3.out‚0( disk.nvme0c0n1.out‚84750.91455078125 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd0.in‚0 disk.rbd3.in‚0( proc.memory.availableMB‚ 1745.8671875 disk.sda.out‚ 476.8984375 disk.rbd0.out‚0 disk.sda1.in‚0 disk.sda1.out‚0 cpu‚25.542301047203072 disk.dm-1.in‚0 disk.drbd2165.out‚0& disk.nvme0n1.out‚84750.91455078125 disk.rbd1.in‚ 0.01171875 proc.cpu.threads‚17 disk.sda3.in‚0 disk.sda3.out‚ 476.8984375) disk./.usagePercent‚37.01043884033257 disk.drbd2165.in‚0# proc.memory.rssMB‚ 2943.48828125Va@n2“- _wandbruntime‚9857  _runtime‚9857w<ȬÇ”Á & lr-AdamW/pg1‚0.0005638080662714119 & lr-AdamW/pg2‚0.0005638080662714119 trainer/global_step‚5949 % _timestamp‚1.7842996730401983e+09 _runtime‚9857.257411467 _step‚486Ÿ±’ŸÌ•Æ & lr-AdamW/pg1‚0.0005638080662714119 & lr-AdamW/pg2‚0.0005638080662714119 trainer/global_step‚5949 % _timestamp‚1.7842996730401983e+09 _runtime‚9857.257411467 _step‚486æÊøë±2–- _wandbruntime‚9857  _runtime‚9857Ž|WY(—üO JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚251 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1713376247776768 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1707045793607680 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚469121630047744 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5592687453696 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 _step‚487 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚50678155 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚461909816602624 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚246 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5584463679834 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5592687453696 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚51496539 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚50678155 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1707783586496512 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5594032843264 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚51486566 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 trainer/global_step‚5949 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5594032843264 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚32126113 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1712630282715482 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1713377619339776 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1713377619339776 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 102174694 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1712628912687192 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7212216072704 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚51486566 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚15259906 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚15259901 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚461909448511488 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1707783560323072 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1713376247776768 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1707783586496512 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚469122032675328 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1707783560323072 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚50678149 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16866207 $ _timestamp‚1.784299673041439e+09 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5583119079512 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 102164715 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚50678149 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440wƒDq˜P ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1707783586496512 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5594032843264 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚51486566 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 trainer/global_step‚5949 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5594032843264 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.ÆR%all.allocated‚32126113 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1712630282715482 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1713377619339776 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1713377619339776 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 102174694 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1712628912687192 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7212216072704 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚51486566 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚15259906 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚15259901 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚461909448511488 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1707783560323072 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1713376247776768 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1707783586496512 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚469122032675328 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1707783560323072 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚50678149 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16866207 $ _timestamp‚1.784299673041439e+09 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5583119079512 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 102164715 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚50678149 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚251 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1713376247776768 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1707045793607680 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚469121630047744 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5592687453696 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 _step‚487 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚50678155 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚461909816602624 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚246 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5584463679834 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5592687453696 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚51496539 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚50678155 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973çŸ9Fæ2™- _wandbruntime‚9858  _runtime‚9858’O­Ô 'š‡N WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16868866 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7213381477888 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1713656545112576 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1713656545112576 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 102191392 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚32131072 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7213416007168 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚9858.672567969 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚469200719017472 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 % _timestamp‚1.7842996744550333e+09 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1708061587226624 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚50686355 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1708061561053184 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1713655173542400 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5594957885952 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚461987303010304 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 _step‚488 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚51505031 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 102191392 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820m˜/'›ŒN aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7213416007168 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚9858.672567969 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚469200719017472 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 % _timestamp‚1.7842996744550333e+09 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1708061587226624 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚50686355 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1708061561053184 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1713655173542400 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5594957885952 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚461987303010304 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 _step‚488 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚51505031 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 102191392 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16868866 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7213381477888 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1713656545112576 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1713656545112576 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 102191392 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚32131072è í92œ- _wandbruntime‚9858  _runtime‚9858^?÷`‚ %loss/total_loss‚8.371728718014637 2loss/SRqqt2/assignment_loss‚0.4981145457534396 1loss/FBt1/assignment_loss‚0.00776197475313498 1loss/SRqqt1/detection_loss‚0.8938359141699038 /loss/FBt1/detection_loss‚0.9978485629942856 % _timestamp‚1.7842996745271063e+09 _runtime‚9858.744348862 0loss/FRt2/assignment_loss‚0.6308100772209656 2loss/SRqqt1/assignment_loss‚0.4467081824320303 _step‚489 1loss/FBt2/assignment_loss‚0.06543414499236563 /loss/FRt1/detection_loss‚0.5274628246079721 /loss/FRt2/detection_loss‚1.5167605871883914 /loss/FBt2/detection_loss‚0.5532274621720881 0loss/FRt1/assignment_loss‚1.6158427446795696 1loss/SRqqt2/detection_loss‚0.6179216970504902 epoch‚14 trainer/global_step‚5949#Štƒ ž‡ trainer/global_step‚5949 %loss/total_loss‚8.371728718014637 2loss/SRqqt2/assignment_loss‚0.4981145457534396 1loss/FBt1/assignment_loss‚0.00776197475313498 1loss/SRqqt1/detection_loss‚0.8938359141699038 /loss/FBt1/detection_loss‚0.9978485629942856 % _timestamp‚1.7842996745271063e+09 _runtime‚9858.744348862 0loss/FRt2/assignment_loss‚0.6308100772209656 2loss/SRqqt1/assignment_loss‚0.4467081824320303 _step‚489 1loss/FBt2/assignment_loss‚0.06543414499236563 /loss/FRt1/detection_loss‚0.5274628246079721 /loss/FRt2/detection_loss‚1.5167605871883914 /loss/FBt2/detection_loss‚0.5532274621720881 0loss/FRt1/assignment_loss‚1.6158427446795696 1loss/SRqqt2/detection_loss‚0.6179216970504902 epoch‚14éןŸ‚(:” ¥éÒ·¬îö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚279.015' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚55.803000000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ŠêZ ‚(:ø ¥éÒ’¨½ù disk.dm-0.in‚ 20116.0078125 disk.sda3.out‚ 477.578125 disk.drbd2165.in‚0( disk.nvme0c0n1.out‚84758.97314453125) proc.memory.availableMB‚ 1745.88671875 proc.cpu.threads‚17 disk.rbd1.out‚ 24.1943359375 disk.sda.in‚0 disk.sda1.out‚0 disk.rbd3.in‚0# disk.nvme0c0n1.in‚ 20116.0078125 cpu‚25.54205803311009 disk.sda.out‚ 477.578125 disk.drbd2165.out‚0) disk./.usagePercent‚37.01118945442296 disk.dm-1.in‚0 disk.rbd0.out‚0# memory_percent‚89.3439531326294) proc.memory.percent‚17.96562671661377 network.recv‚ 2.471828e+06# disk.dm-0.out‚84758.97314453125 disk.sda3.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd0.in‚0 disk.sda1.in‚0# proc.memory.rssMB‚ 2943.48828125 network.sent‚ 1.1250119e+07$ disk./.usageGB‚551.4934692382812 disk.rbd1.in‚ 0.01171875! disk.nvme0n1.in‚ 20116.0078125 disk.dm-1.out‚ 477.578125 disk.rbd3.out‚0& disk.nvme0n1.out‚84758.97314453125< Â{¡‚(:” ´éÒæãßö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.m­BŸÖ$emoryAllocatedBytes‚ 49616584704 gpu.0.temp‚40.0 gpu.0.powerWatts‚156.396' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚31.279199999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‘²UØ¢‚(:ý ´éҲݔù disk.rbd1.out‚ 24.1943359375 disk.dm-1.out‚ 479.7109375# disk.dm-0.out‚84787.16455078125& disk.nvme0n1.out‚84787.16455078125$ memory_percent‚89.34407234191895 cpu‚25.541895134402775 network.recv‚ 2.476251e+06* disk./.usagePercent‚37.011963365198966 disk.sda2.out‚0 disk.rbd1.in‚ 0.01171875 disk.sda.out‚ 479.7109375 disk.dm-0.in‚ 20116.4296875 disk.sda1.in‚0 disk.sda3.out‚ 479.7109375# proc.memory.rssMB‚ 2943.48828125 disk.drbd2165.in‚0# disk.nvme0c0n1.in‚ 20116.4296875) proc.memory.percent‚17.96562671661377 network.sent‚ 1.1313123e+07( disk.nvme0c0n1.out‚84787.16455078125$ disk./.usageGB‚551.5050010681152 disk.drbd2165.out‚0( proc.memory.availableMB‚ 1745.8671875 proc.cpu.threads‚17 disk.dm-1.in‚0 disk.sda3.in‚0 disk.rbd3.out‚0 disk.sda2.in‚0 disk.sda.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda1.out‚0 disk.rbd3.in‚0! disk.nvme0n1.in‚ 20116.4296875‘O^Úž£‚(:“ ÃéÒÇ“çö gpu.0.gpu‚79.0 gpu.0.memory‚25, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚336.465' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚67.29299999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¾FŸB¤‚(:ø ÃéÒî“ù disk.dm-0.in‚ 20116.6328125 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda3.in‚0 disk.sda3.out‚ 482.1171875 disk.drbd2165.out‚0 disk.rbd3.in‚0# disk.nvme0c0n1.in‚ 20116.6328125 disk.dm-1.in‚0 cpu‚25.5416807143039$ disk./.usageGB‚551.5119438171387 disk.rbd3.out‚0! disk.nvme0n1.in‚ 20116.6328125 disk.rbd1.in‚ 0.01171875 disk.sda.out‚ 482.1171875 disk.drbd2165.in‚0$ memory_percent‚89.34447765350342' disk.nvme0c0n1.out‚84799.3955078125 disk.rbd1.out‚ 24.2724609375 disk.dm-1.out‚ 482.1171875) disk./.usagePercent‚37.01242929891128 disk.sda.in‚0" disk.dm-0.out‚84799.3955078125) proc.memory.availableMB‚ 1745.80078125 network.sent‚ 1.1318183e+07 network.recv‚ 2.479716e+06 disk.rbd0.in‚0# proc.memory.rssMB‚ 2943.48828125) proc.memory.percent‚17.96562671661377 disk.rbd0.out‚0 proc.cpu.threads‚17% disk.nvme0n1.out‚84799.3955078125 disk.sda2.in‚0 disk.sda2.out‚0ê×c*”¥‚(:‰ ÒéÒÜðñ÷ gpu.0.gpu‚88.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚290.327' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚58.0654 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0bé¹n ¦‚(:‚ ÒéÒ†îŽù network.recv‚ 2.483025e+06 disk.dm-1.in‚0 disk.rbd0.in‚0' proc.memory.availableMB‚ 1746.203125# proc.memory.rssMB‚ 2943.48828125 disk.sda.in‚0 disk.dm-1.out‚ 483.48046875 disk.sda1.out‚0 disk.sda3.in‚0 disk.drbd2165.in‚0# disk.dm-0.out‚84816.66259765625 network.sent‚ 1.1323209e+07 cpu‚25.541467055621812 disk.sda3.out‚ 483.48046875$ disk.nvme0c0n1.in‚20452.43359375$ memory_percent‚89.34202194213867 disk.rbd1.in‚ 0.01171875 disk.sda1.in‚0* disk./.usagePercent‚36.900270862115406$ disk./.usageGB‚549.8407020568848 disk.rbd3.in‚0 disk.rbd3.out‚0) proc.memory.percent‚17.96562671661377 disk.dm-0.in‚20452.43359375 disk.drbd2165.out‚0" disk.nvme0n1.in‚20452.43359375& disk.nvme0n1.out‚84816.66259765625 disk.sda.out‚ 483.48046875 disk.rbd0.out‚0( disk.nvme0c0n1.out‚84816.66259765625 disk.sda2.in‚0 disk.sda2.out‚0 proc.cpu.threads‚17 disk.rbd1.out‚ 24.2724609375dËJ”§‚(:‰ áéÒûþðö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚271.478' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.2956 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0×i¸ ¨‚(:þ áéÒàÑ”ù disk.rbd3.in‚0 disk.rbd3.out‚0 disk.dm-1.out‚ 485.44140625 disk.sda1.out‚0 disk.sda3.in‚0# memory_percent‚89.3435001373291) proc.memory.percent‚17.96562671661377 proc.cpu.threads‚17 disk.drbd2165.in‚0$ disk.nvme0c0n1.in‚20452.65234375( proc.memory.availableMB‚ 1745.9609375 network.recv‚ 2.486369e+06) disk./.usagePercent‚36.90054811827499$ disk./.usageGB‚549.8448333740234 disk.sda2.in‚0 disk.rbd0.in‚0 disk.sda2.out‚0' disk.nvme0c0n1.out‚84830.7451171875" disk.nvme0n1.in‚20452.65234375 disk.rbd0.out‚0 disk.sda.in‚0 cpu‚25.541278759389897 network.sent‚ 1.1327731e+07" disk.dm-0.out‚84830.7451171875# proc.memory.rssMB‚ 2943.48828125 disk.sda.out‚ 485.44140625 disk.sda3.out‚ 485.44140625% disk.nvme0n1.out‚84830.7451171875 disk.rbd1.in‚ 0.01171875 disk.rbd1.out‚ 24.3349609375 disk.dm-1.in‚0 disk.dm-0.in‚20452.65234375 disk.sda1.in‚0 disk.drbd2165.out‚0î÷z"2©- _wandbruntime‚9938  _runtime‚9938ùã´$ǪÁ & lr-AdamW/pg1‚0.0005616478001463747 & lr-AdamW/pg2‚0.0005616478001463747 trainer/global_step‚5999 % _timestamp‚1.7842997539626594e+09 _runtime‚9938.179937369 _step‚490ØÚwíÌ«Æ & lr-AdamW/pg1‚0.0005616478001463747 & lr-AdamW/pg2‚0.0005616478001463747 trainer/global_step‚5999 % _timestamp‚1.7842997539626594e+09 _runtime‚9938.179937369 _step‚490êrnÑý2¬- _wandbruntime‚9938  _runtime‚9938x—4(­üO PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚51922291 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚51922291 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚231 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17003099 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚473056772013568 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1727322600212992 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 400530432 trainer/global_step‚5999 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚236 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚51088449 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚51912318 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚465784136986624 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚51912318 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1727322600212992 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚51088449 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17003330 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1720939907944448 _step‚491 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1721683596828672 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 _runtime‚ 9938.18133127 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5639003384320 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1721683596828672 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5640348773888 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7272602587648 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5639003384320 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚51088455 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1721683623002112 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5629356192620 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚32390386 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚465783768895488 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1720939882516480 % _timestamp‚1.7842997539638984e+09 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1726570608737390 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 103000767 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1721683623002112 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5630700792942 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1727323971776000 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚473056371483136 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1726569238709100 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚51088455 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5640348773888 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚32390150 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7272635026944 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820åšë(®P ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5639003384320 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1721683596828672 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5640348773888 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7272602587648 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5639003384320 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚51088455 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1721683623002112 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5629356192620 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚32390386 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚465783768895488 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1720939882516480 % _timestamp‚1.7842997539638984e+09 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1726570608737390 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 103000767 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1721683623002112 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5630700792942 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1727323971776000 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚473056371483136 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1726569238709100 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚51088455 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5640348773888 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚32390150 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7272635026944 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚51922291 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚51922291 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚231 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17003099 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚473056772013568 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1727322600212992 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 400530432 trainer/global_step‚5999 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚236 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚51088449 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚51912318 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚465784136986624 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚51912318 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1727322600212992 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚51088449 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17003330 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1720939907944448 _step‚491 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1721683596828672 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 _runtime‚ 9938.18133127 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979ëÖ(¥2¯- _wandbruntime‚9939  _runtime‚9939‹:l»S°‡N B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5630279658866 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1727601525978624 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚51096661 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚228 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚51930783 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 103027444 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5631624259300 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚51920796 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚51096655 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚223 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚32395440 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 _runtime‚9939.567313871 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1727602897548800 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚17005846 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1726848043953522 % _timestamp‚1.7842997553498232e+09 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚15389594 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5639928419840 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7273842661888 ?6DeviceStatsMonitor.on_train_ba59Eºtch_end/num_device_alloc‚1995 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 103027444 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚32432128 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 103017451 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚473135065532928 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚51930783 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1721961623732224 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1721961597558784 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚32395668 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7273810229760 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚51096655 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚473135466056192 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚465861623394304 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚465861255303168 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1726849413981924 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5630279658866 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1727601525978624 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚51096661 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚228 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚51930783 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 103027444 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5631624259300 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚51920796 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚51096655 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚223 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚32395440 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 _runtime‚9939.567313871 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1727602897548800 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚17005846 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1726848043953522 % _timestamp‚1.7842997553498232e+09 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚15389594 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5639928419840 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7273842661888 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 103027444 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚32432128 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 103017451 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚473135065532928 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚51930783 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1721961623732224 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1721961597558784 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚32395668 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7273810229760 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚51096655 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚473135466056192 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚465861623394304 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚465861255303168 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1726849413981924 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 Q&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 6 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.988095223903656 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 -"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4375 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9934239983558655 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 )jet/accuracy_2_of_3‚0.550629734992981 particle/accuracy_2_of_2‚1 +particle/specificity‚0.9134278893470764 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.8571428656578064 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.3028293550014496 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9911240339279175 _runtime‚9965.975373528 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 particle/accuracy_5_of_5‚0 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6568855047225952 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.25287196040153503 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6780985593795776 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.14000001549720764 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9972379803657532 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.038716286420822144 jet/accuracy_2_of_5‚1 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9927973747253418 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6793566942214966 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7035165429115295 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9925198554992676 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3776022493839264 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6333082914352417 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7379695177078247 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7399634122848511 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.3046186864376068 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.11288429796695709 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.025439782068133354 jet/accuracy_1_of_5‚1 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6326789855957031 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6380208730697632 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.17525602877140045 9"Purity/*FRt2SRqqt0FBt/event_purity‚0.1785334050655365 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.23118725419044495 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7963207960128784 *jet/accuracy_1_of_3‚0.9259123206138611 .particle/accuracy_4_of_4‚0.956376314163208 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.05000000074505806 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9964818358421326 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.2848171293735504 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9876681566238403 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 8"Purity/1FRt0SRqqt1FBt/event_purity‚0.621157169342041 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.21453459560871124 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3072916865348816 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6454894542694092 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 , Purity/0FRt2SRqqt*FBt/FBt_purity‚0.96875 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.49768516421318054 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9968253374099731 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.06650759279727936 +jet/accuracy_3_of_3‚0.06962672621011734 particle/accuracy_2_of_5‚1 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8566516041755676 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8888888955116272 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.03431258350610733 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 _step‚494 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6092938780784607 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.03752133250236511 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 *jet/accuracy_1_of_4‚0.9721348285675049 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9813995361328125 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6363009214401245 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.27824217081069946 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.13055306673049927 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.2946861684322357 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7468002438545227 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.625 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9814814329147339 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9921150207519531 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9822907447814941 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9179728627204895 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.2413475662469864 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.3158555030822754 6 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.630757749080658 8"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.731779158115387 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8504794239997864 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8607921600341797 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.4814279079437256 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7731481194496155 *jet/accuracy_1_of_2‚0.7878799438476562 6validation_average_jet_accuracy‚0.3872174322605133 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4394698739051819 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.2222222238779068 particle/accuracy_1_of_1‚1 /particle/accuracy_3_of_4‚0.9965986609458923 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7142857313156128 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 +jet/accuracy_2_of_2‚0.14775100350379944 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8411186933517456 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6275759935379028 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2319837063550949 (particle/accuracy‚0.8546742796897888 # _timestamp‚1.78429978175759e+09 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 particle/accuracy_2_of_3‚1 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7385085821151733 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7404965758323669 ,jet/accuracy_4_of_4‚0.013899499550461769 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.1666666716337204 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6710779070854187 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9952788352966309 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.42824074625968933 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6323472857475281 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 7 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.0339202843606472 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7485626935958862 jet/accuracy_5_of_5‚0 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt1SRqqt*FBt/event_purity‚0.0494614839553833 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.2870141565799713 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4842709004878998 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.03846520930528641 particle/accuracy_3_of_5‚1 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6346028447151184 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6121417284011841 , Purity/0FRt2SRqqt1FBt/FBt_purity‚0.96875 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7050763368606567 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7746409177780151 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.25870582461357117 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8504794239997864 8"Purity/*FRt*SRqqt1FBt/event_purity‚0.741935133934021 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5657650828361511 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.14813770353794098 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.30493178963661194 9"Purity/2FRt0SRqqt*FBt/event_purity‚0.0814887210726738 particle/accuracy_3_of_3‚1 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.23076923191547394 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2524831295013428 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9877882599830627 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9814814329147339 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 +jet/accuracy_1_of_1‚0.35178983211517334 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5533345341682434 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6858525276184082 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.1666666716337204 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.2946861684322357 epoch‚14 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9894997477531433 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 particle/accuracy_1_of_5‚1 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9836763739585876 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 particle/accuracy_1_of_3‚1 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6343581080436707 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5675206780433655 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 particle/accuracy_4_of_5‚1 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.25015783309936523 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.49712032079696655 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6340384483337402 +particle/sensitivity‚0.6398419141769409 jet/accuracy_3_of_5‚1 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9947829246520996 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.3990807831287384 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 6"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7940673828125 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7731481194496155 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.8125 particle/accuracy_2_of_4‚1 trainer/global_step‚5999 )loss/val_total_loss‚8.101523890243705 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8962738513946533 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.38797950744628906 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 *jet/accuracy_2_of_4‚0.7947174906730652 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.29186829924583435 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.042338185012340546 particle/accuracy_1_of_2‚1 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 +jet/accuracy_3_of_4‚0.09532209485769272 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8457293510437012 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.23429515957832336 'particle/f_score‚0.6540877819061279 8"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.683516263961792 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9349221587181091 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3752133548259735 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 7 Purity/0FRt0SRqqt*FBt/FBt_puri—(ty‚0.9938336610794067 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9755765199661255 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6333082914352417 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8525887727737427 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7397673726081848 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5674999952316284 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 jet/accuracy_4_of_5‚0 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3499075174331665 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8446522951126099 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.13002702593803406 7"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.73566734790802 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.05000000074505806 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.025439782068133354 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.2220529317855835 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.23076923191547394 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9962468147277832 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.8846153616905212 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9949080348014832 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7089226841926575 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.034555286169052124 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9166666865348816êa_Í=»Ç{ <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 particle/accuracy_5_of_5‚0 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6568855047225952 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.25287196040153503 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6780985593795776 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.14000001549720764 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9972379803657532 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.038716286420822144 jet/accuracy_2_of_5‚1 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9927973747253418 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6793566942214966 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7035165429115295 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9925198554992676 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3776022493839264 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6333082914352417 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7379695177078247 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7399634122848511 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.3046186864376068 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.11288429796695709 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.025439782068133354 jet/accuracy_1_of_5‚1 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6326789855957031 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6380208730697632 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.17525602877140045 9"Purity/*FRt2SRqqt0FBt/event_purity‚0.1785334050655365 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.23118725419044495 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7963207960128784 *jet/accuracy_1_of_3‚0.9259123206138611 .particle/accuracy_4_of_4‚0.956376314163208 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.05000000074505806 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9964818358421326 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.2848171293735504 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9876681566238403 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 8"Purity/1FRt0SRqqt1FBt/event_purity‚0.621157169342041 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.21453459560871124 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3072916865348816 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6454894542694092 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 , Purity/0FRt2SRqqt*FBt/FBt_purity‚0.96875 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.49768516421318054 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9968253374099731 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.06650759279727936 +jet/accuracy_3_of_3‚0.06962672621011734 particle/accuracy_2_of_5‚1 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8566516041755676 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8888888955116272 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.03431258350610733 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 _step‚494 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6092938780784607 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.03752133250236511 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 *jet/accuracy_1_of_4‚0.9721348285675049 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9813995361328125 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6363009214401245 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.27824217081069946 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.13055306673049927 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.2946861684322357 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7468002438545227 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.625 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9814814329147339 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9921150207519531 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9822907447814941 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9179728627204895 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.2413475662469864 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.3158555030822754 6 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.630757749080658 8"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.731779158115387 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8504794239997864 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8607921600341797 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.4814279079437256 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7731481194496155 *jet/accuracy_1_of_2‚0.7878799438476562 6validation_average_jet_accuracy‚0.3872174322605133 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4394698739051819 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.2222222238779068 particle/accuracy_1_of_1‚1 /particle/accuracy_3_of_4‚0.9965986609458923 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7142857313156128 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 +jet/accuracy_2_of_2‚0.14775100350379944 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8411186933517456 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6275759935379028 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2319837063550949 (particle/accuracy‚0.8546742796897888 # _timestamp‚1.78429978175759e+09 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 particle/accuracy_2_of_3‚1 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7385085821151733 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7404965758323669 ,jet/accuracy_4_of_4‚0.013899499550461769 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.1666666716337204 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6710779070854187 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9952788352966309 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.42824074625968933 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6323472857475281 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 7 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.0339202843606472 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7485626935958862 jet/accuracy_5_of_5‚0 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt1SRqqt*FBt/event_purity‚0.0494614839553833 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.2870141565799713 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4842709004878998 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.03846520930528641 particle/accuracy_3_of_5‚1 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6346028447151184 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6121417284011841 , Purity/0FRt2SRqqt1FBt/FBt_purity‚0.96875 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7050763368606567 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7746409177780151 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.25870582461357117 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8504794239997864 8"Purity/*FRt*SRqqt1FBt/event_purity‚0.741935133934021 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5657650828361511 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.14813770353794098 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.30493178963661194 9"Purity/2FRt0SRqqt*FBt/event_purity‚0.0814887210726738 particle/accuracy_3_of_3‚1 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.23076923191547394 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2524831295013428 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9877882599830627 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9814814329147339 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 +jet/accuracy_1_of_1‚0.35178983211517334 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5533345341682434 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6858525276184082 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.1666666716337204 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.2946861684322357 epoch‚14 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9894997477531433 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 particle/accuracy_1_of_5‚1 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9836763739585876 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 particle/accuracy_1_of_3‚1 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6343581080436707 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5675206780433655 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 particle/accuracy_4_of_5‚1 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.25015783309936523 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.49712032079696655 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6340384483337402 +particle/sensitivity‚0.6398419141769409 jet/accuracy_3_of_5‚1 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9947829246520996 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.3990807831287384 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 6"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7940673828125 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7731481194496155 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.8125 particle/accuracy_2_of_4‚1 trainer/global_step‚5999 )loss/val_total_loss‚8.101523890243705 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8962738513946533 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.38797950744628906 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 *jet/accuracy_2_of_4‚0.7947174906730652 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.29186829924583435 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.042338185012340546 particle/accuracy_1_of_2‚1 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 +jet/accuracy_3_of_4‚0.09532209485769272 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8457293510437012 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.23429515957832336 'particle/f_score‚0.6540877819061279 8"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.683516263961792 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9349221587181091 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3752133548259735 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9938336610794067 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9755765199661255 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6333082914352417 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8525887727737427 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7397673726081848 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5674999952316284 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 jet/accuracy_4_of_5‚0 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3499075174331665 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8446522951126099 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.13002702593803406 7"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.73566734790802 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.05000000074505806 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.025439782068133354 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.2220529317855835 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.23076923191547394 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9962468147277832 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.8846153616905212 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9949080348014832 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7089226841926575 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.034555286169052124 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9166666865348816 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9964818358421326 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3477487862110138 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.2222222238779068 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 particle/accuracy_1_of_4‚1 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.23836953938007355 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8741931915283203 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6401523351669312 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 6 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.988095223903656 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 -"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4375 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9934239983558655 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 )jet/accuracy_2_of_3‚0.550629734992981 particle/accuracy_2_of_2‚1 +particle/specificity‚0.9134278893470764 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.8571428656578064 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.3028293550014496 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9911240339279175 _runtime‚9965.975373528îEk»ËŸ¼‚(:” Ž‚éÒðœäø gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚35.0 gpu.0.powerWatts‚273.071' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚54.614200000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0lšYô½‚(:ü Ž‚éÒ¢¦æú disk.rbd3.in‚0 disk.rbd3.out‚0$ disk.nvme0n1.out‚85611.056640625! disk.dm-0.out‚85611.056640625( proc.memory.availableMB‚ 1947.5703125 network.recv‚ 2.496283e+06 disk.sda3.in‚0 disk.sda3.out‚ 492.33984375$ disk.nvme0c0n1.in‚20576.04296875$ disk./.usageGB‚549.9201927185059" disk.nvme0n1.in‚20576.04296875 disk.rbd1.out‚ 24.4130859375 disk.sda.in‚0 disk.dm-1.out‚ 492.33984375 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd1.in‚ 0.01171875 disk.sda.out‚ 492.33984375 disk.dm-1.in‚0 disk.rbd0.out‚0$ memory_percent‚88.11297416687012 network.sent‚ 1.1399992e+07& disk.nvme0c0n1.out‚85611.056640625 proc.cpu.threads‚17 disk.sda1.in‚0 disk.sda1.out‚0# proc.memory.rssMB‚ 2943.48828125) proc.memory.percent‚17.96562671661377 cpu‚25.547360298232405) disk./.usagePercent‚36.90560554711391 disk.drbd2165.out‚0 disk.rbd0.in‚0 disk.dm-0.in‚20576.04296875 disk.drbd2165.in‚0“©d”¾‚(:‰ ‚éÒ·Òçö gpu.0.gpu‚90.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚288.968' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚57.7936 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0I÷Åå ¿‚(: ‚éÒØÔ€ù disk.sda1.out‚0$ disk.nvme0c0n1.in‚20576.46484375& proc.memory.availableMB‚ 1947.53125 disk.rbd0.out‚0 disk.sda3.in‚0) proc.memory.percent‚17.96562671661377 network.recv‚ 2.500471e+06* disk./.usagePercent‚36.906246077964575 disk.drbd2165.out‚0 disk.rbd3.out‚0$ memory_percent‚88.11321258544922 cpu‚25.547124150217883 disk.rbd1.out‚ 24.4755859375 disk.drbd2165.in‚0 disk.rbd3.in‚0" disk.nvme0n1.in‚20576.46484375 disk.sda.out‚ 496.01171875 disk.dm-1.in‚0 network.sent‚ 1.1457836e+07 disk.rbd1.in‚ 0.01171875 disk.dm-0.in‚20576.46484375 disk.sda1.in‚0 disk.sda3.out‚ 496.01171875$ disk./.usageGB‚549.9297370910645 disk.dm-1.out‚ 496.01171875& disk.nvme0n1.out‚86916.81884765625 proc.cpu.threads‚17 disk.sda2.out‚0 disk.sda.in‚0 disk.rbd0.in‚0# disk.dm-0.out‚86916.81884765625( disk.nvme0c0n1.out‚86916.81884765625# proc.memory.rssMB‚ 2943.48828125 disk.sda2.in‚0!àžÀ‚(:“ ¬‚éÒ¾•—ø gpu.0.gpu‚55.0 gpu.0.memory‚17, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚41.0 gpu.0.powerWatts‚334.292' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚66.85839999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0%r'«Á‚(:ú ¬‚éÒè®êø disk.sda3.in‚0 disk.dm-1.out‚ 498.703125 disk.rbd0.in‚0 disk.dm-0.in‚20576.66796875 disk.sda1.out‚0 disk.sda3.out‚ 498.703125* disk./.usagePercent‚36.906297279471424( proc.memory.availableMB‚ 1947.8046875 network.sent‚ 1.1463389e+07 disk.dm-1.in‚0" disk.dm-0.out‚86938.8662109375' disk.nvme0c0n1.out‚86938.8662109375$ memory_percent‚88.11154365539551# proc.memory.rssMB‚ 2943.48828125 disk.rbd1.in‚ 0.01171875 disk.sda1.in‚0 disk.drbd2165.out‚0 disk.rbd3.in‚0 disk.rbd3.out‚0 cpu‚25.546929947187493 proc.cpu.threads‚17 disk.sda.in‚0 disk.rbd0.out‚0) proc.memory.percent‚17.96562671661377 network.recv‚ 2.503609e+06$ disk./.usageGB‚549.9305000305176" disk.nvme0n1.in‚20576.66796875 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda.out‚ 498.703125 disk.drbd2165.in‚0$ disk.nvme0c0n1.in‚20576.66796875% disk.nvme0n1.out‚86938.8662109375 disk.rbd1.out‚ 24.47558593758ÿ~&ŸÂ‚(:” »‚éÒÛ³šø gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚263.424' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚52.684799999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0›Ã‚(:ú »‚éÒùàƒù network.sent‚ 1.1467875e+07 network.recv‚ 2.506945e+06 disk.sda2.out‚0 disk.dm-1.in‚0 proc.cpu.threads‚17 disk.drbd2165.out‚0 disk.sda2.in‚0 disk.rbd1.out‚ 24.5537109375 disk.sda.in‚0$ disk./.usageGB‚549.9412536621094 disk.sda1.out‚0 disk.sda3.out‚ 501.3984375 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.rbd1.in‚ 0.01171875 disk.sda.out‚ 501.3984375$ memory_percent‚88.11311721801758 disk.sda3.in‚0$ disk.nvme0c0n1.in‚20576.88671875% disk.nvme0n1.out‚86964.3134765625 disk.dm-1.out‚ 501.3984375# proc.memory.rssMB‚ 2943.48828125 cpu‚25.5466347781622" disk.nvme0n1.in‚20576.88671875' proc.memory.availableMB‚ 1947.546875* disk./.usagePercent‚36.907018964710446 disk.drbd2165.in‚0' disk.nvme0c0n1.out‚86964.3134765625 disk.rbd0.out‚0 disk.dm-0.in‚20576.88671875" disk.dm-0.out‚86964.3134765625 disk.sda1.in‚0 disk.rbd0.in‚0) proc.memory.percent‚17.96562671661377Æ[ŸÄ‚(:” Ê‚éÒõ¬÷ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚166.668' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚33.333600000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ìïìÅ‚(:÷ Ê‚éÒÜÒêø disk.rbd0.in‚0 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.rbd1.in‚ 0.01171875 disk.sda1.out‚0& disk.nvme0c0n1.out‚86994.587890625$ disk.nvme0n1.out‚86994.587890625 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-1.in‚0 cpu‚25.546441991080847! disk.dm-0.out‚86994.587890625 disk.sda3.in‚0 disk.dm-1.out‚ 504.2265625) proc.memory.availableMB‚ 1947.51171875# proc.memory.rssMB‚ 2943.48828125) proc.memory.percent‚17.96562671661377 network.sent‚ 1.1472939e+07 disk.rbd0.out‚0 disk.sda3.out‚ 504.2265625# disk.nvme0c0n1.in‚ 20836.2734375 disk.sda.in‚0$ disk./.usageGB‚549.9465560913086 disk.sda1.in‚0 disk.drbd2165.out‚0! disk.nvme0n1.in‚ 20836.2734375 proc.cpu.threads‚17 disk.drbd2165.in‚0) disk./.usagePercent‚36.90737481518304 disk.dm-0.in‚ 20836.2734375 disk.rbd1.out‚ 24.5537109375 disk.sda.out‚ 504.2265625$ memory_percent‚88.11333179473877 network.recv‚ 2.509922e+06kRŽT4Æ/ _wandbruntime‚10049  _runtime‚10049Øê€ÇÇÁ _runtime‚10049.15524339 _step‚495 & lr-AdamW/pg1‚0.0005594736784992787 & lr-AdamW/pg2‚0.0005594736784992787 trainer/global_step‚6049 % _timestamp‚1.7842998649380586e+09¢{-³ÌÈÆ & lr-AdamW/pg1‚0.0005594736784992787 & lr-AdamW/pg2‚0.0005594736784992787 trainer/global_step‚6049 % _timestamp‚1.7842998649380586e+09 _runtime‚10049.15524339 _step‚495ïŒç¨4É/ _wandbruntime‚10049  _runtime‚10049Ãð~`öÊýO PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚51630761 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1745102776698368 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚15523119 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1739411614769152 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7339908982784 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚252 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚32439296 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _step‚496 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚51630761 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1744342672823348 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5692507318784 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5691161929216 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1745104148261376 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1745102776698368 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 _runtime‚10049.156608188 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚477046078069760 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚477046478600192 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17178263 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚469706569617408 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.å´§Õ small_pool.freed‚ 5681422437170 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1739411640942592 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5692507318784 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚52397057 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 400530432 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1745102776698368 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚15523119 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1739411614769152 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7339908982784 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚252 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚32439296 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _step‚496 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚51630761 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1744342672823348 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5692507318784 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5691161929216 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1745104148261376 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1745102776698368 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 _runtime‚10049.156608188 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚477046078069760 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚477046478600192 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17178263 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚469706569617408 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5681422437170 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1739411640942592 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5692507318784 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚52397057 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 400530432 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1739689641672704 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1739689615499264 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1744621478067882 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5683690503850 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚15525662 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1739689615499264 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 104044496 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚52415522 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚477124765079552 _step‚497 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 104054489 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5692086964736 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚15525667 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚247 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚51638961 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1739689641672704 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚32706676 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 _runtime‚10050.54856979 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚51638967 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 trainer/global_step‚6049 % _timestamp‚1.7842998663310504e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 104054489 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17181261 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚32706928 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7341077145600 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175m26ßÐŒN =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1745381702464000 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚52405535 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚477125167699968 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1744620108039480 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 104044496 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚51638961 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7341111674880 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1739689641672704 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1739689615499264 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1744621478067882 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5683690503850 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚15525662 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1739689615499264 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 104044496 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚52415522 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚477124765079552 _step‚497 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 104054489 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5692086964736 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚15525667 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚247 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚51638961 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1739689641672704 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚32706676 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 _runtime‚10050.54856979 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚51638967 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 trainer/global_step‚6049 % _timestamp‚1.7842998663310504e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 104054489 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17181261 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚32706928 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7341077145600 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚51638967 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5692086964736 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5693432361472 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚17181014 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚469784056025088ñgä–—4Ñ/ _wandbruntime‚10050  _runtime‚10050ß)tÒ‚ %loss/total_loss‚7.605165038153344 2loss/SRqqt1/assignment_loss‚0.5150986583799143 1loss/FBt2/assignment_loss‚0.02620888508277825 /loss/FRt1/detection_loss‚0.5102105325032493 /loss/FRt2/detection_loss‚1.4644229711207872 1loss/SRqqt2/detection_loss‚0.5926566742846354 $ _timestamp‚1.784299866414371e+09 0loss/FRt2/assignment_loss‚0.5603790998773399 1loss/SRqqt1/detection_loss‚0.7906658897874876 _step‚498 /loss/FBt1/detection_loss‚0.5455248399251481 0loss/FRt1/assignment_loss‚1.6050629035742194 2loss/SRqqt2/assignment_loss‚0.4650709783993184 1loss/FBt1/assignment_loss‚0.10622429315496427 0loss/FBt2/detection_loss‚0.42363931206350125 epoch‚15 trainer/global_step‚6049 _runtime‚10050.63136209¯Zñà Ó‡ /loss/FRt2/detection_loss‚1.4644229711207872 1loss/SRqqt2/detection_loss‚0.5926566742846354 $ _timestamp‚1.784299866414371e+09 0loss/FRt2/assignment_loss‚0.5603790998773399 1loss/SRqqt1/detection_loss‚0.7906658897874876 _step‚498 /loss/FBt1/detection_loss‚0.5455248399251481 0loss/FRt1/assignment_loss‚1.6050629035742194 2loss/SRqqt2/assignment_loss‚0.4650709783993184 1loss/FBt1/assignment_loss‚0.10622429315496427 0loss/FBt2/detection_loss‚0.42363931206350125 epoch‚15 trainer/global_step‚6049 _runtime‚10050.63136209 %loss/total_loss‚7.605165038153344 2loss/SRqqt1/assignment_loss‚0.5150986583799143 1loss/FBt2/assignment_loss‚0.02620888508277825 /loss/FRt1/detection_loss‚0.5102105325032493ò豟Ô‚(:” è‚éÒÖ˜ãö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚274.423' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚54.884600000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¶N˜õÕ‚(:ê è‚éÒºÙ¡ù disk.dm-0.out‚87823.71484375 disk.nvme0n1.in‚ 21061.734375 disk.sda2.out‚0 disk.rbd1.in‚ 0.01171875 cpu‚25.545979468577865) disk./.usagePercent‚37.04404085323174 disk.rbd3.out‚0% disk.nvme0c0n1.out‚87823.71484375 disk.sda2.in‚0( proc.memory.availableMB‚ 975.62890625 network.sent‚ 1.1530944e+07 network.recv‚ 2.517914e+06 disk.sda1.out‚0# proc.memory.rssMB‚ 2943.48828125 proc.cpu.threads‚17 disk.sda.in‚0 disk.dm-1.in‚0 disk.sda3.out‚ 511.53125" disk.nvme0c0n1.in‚ 21061.734375 disk.sda.out‚ 511.53125$ disk./.usageGB‚551.9829788208008 disk.rbd0.in‚0 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.rbd3.in‚0# disk.nvme0n1.out‚87823.71484375 disk.rbd1.out‚ 24.6787109375) proc.memory.percent‚17.96562671661377 disk.dm-0.in‚ 21061.734375 disk.sda1.in‚0 disk.sda3.in‚0$ memory_percent‚94.04523372650146 disk.dm-1.out‚ 511.53125 disk.rbd0.out‚0ËùØì”Ö‚(:‰ ÷‚éÒÀ›üö gpu.0.gpu‚84.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚338.878' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚67.7756 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ël]¤×‚(:õ ÷‚é҈Ɯù( disk.nvme0c0n1.out‚93210.91064453125) proc.memory.percent‚17.96562671661377$ memory_percent‚94.04387474060059 proc.cpu.threads‚17 network.recv‚ 2.522315e+06 disk.sda2.in‚0 disk.sda.in‚0 disk.sda.out‚ 512.33984375 disk.rbd0.in‚0 disk.dm-0.in‚ 21062.15625 disk.drbd2165.out‚0 disk.rbd3.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 512.33984375 disk.rbd3.out‚0 disk.dm-1.in‚0 disk.rbd0.out‚0# disk.dm-0.out‚93210.91064453125( disk./.usagePercent‚37.3563813093364 disk.drbd2165.in‚0' proc.memory.availableMB‚ 975.8515625 cpu‚25.545812421502482 disk.nvme0n1.in‚ 21062.15625 disk.sda1.out‚0! disk.nvme0c0n1.in‚ 21062.15625 disk.rbd1.out‚ 24.7412109375 disk.sda3.in‚0# proc.memory.rssMB‚ 2943.48828125 network.sent‚ 1.158345e+07# disk./.usageGB‚556.637077331543& disk.nvme0n1.out‚93210.91064453125 disk.rbd1.in‚ 0.01171875 disk.dm-1.out‚ 512.33984375 disk.sda1.in‚0‡` Ø‚(:’ †ƒéÒ¹€©ø gpu.0.gpu‚98.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚283.03' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚56.60599999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ü÷Y>üÙ‚(:ñ †ƒéÒ—å¯ù network.recv‚ 2.525618e+06 disk.sda.out‚ 512.8828125 disk.dm-0.out‚93942.49609375 disk.sda1.out‚0 disk.sda3.out‚ 512.8828125 disk.sda2.in‚0) disk./.usagePercent‚37.35668135016653 disk.dm-1.out‚ 512.8828125 disk.dm-0.in‚ 21185.8203125 disk.sda3.in‚0 disk.rbd3.out‚0 network.sent‚ 1.1588448e+07 disk.sda.in‚0 disk.drbd2165.out‚0% disk.nvme0c0n1.out‚93942.49609375 disk.sda2.out‚0 disk.sda1.in‚0! disk.nvme0n1.in‚ 21185.8203125 disk.rbd1.out‚ 24.7412109375 disk.rbd0.out‚0" memory_percent‚94.046950340271( proc.memory.availableMB‚ 975.34765625# proc.memory.rssMB‚ 2943.48828125 cpu‚25.545595455324687 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.drbd2165.in‚0 disk.rbd3.in‚0 disk.rbd1.in‚ 0.01171875 disk.dm-1.in‚0# disk.nvme0c0n1.in‚ 21185.8203125$ disk./.usageGB‚556.6415481567383# disk.nvme0n1.out‚93942.49609375) proc.memory.percent‚17.96562671661377xwò&žÚ‚(:“ •ƒéÒ‡Ùãö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚294.864' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚58.97279999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¢úìÛ‚(:ü •ƒéÒ˜ê‚ù disk.rbd1.out‚ 24.8037109375 disk.sda1.in‚0 disk.sda2.in‚0' proc.memory.availableMB‚ 975.3515625 cpu‚25.54535711225747# disk.nvme0c0n1.in‚ 21359.9296875& disk.nvme0n1.out‚96639.87939453125 proc.cpu.threads‚17 network.sent‚ 1.1592926e+07) disk./.usagePercent‚37.35704052873707 disk.rbd1.in‚ 0.01171875 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda3.in‚0 disk.drbd2165.in‚0 disk.rbd3.in‚0 network.recv‚ 2.528826e+06# disk./.usageGB‚556.646900177002 disk.sda.in‚0 disk.sda.out‚ 513.95703125( disk.nvme0c0n1.out‚96639.87939453125$ memory_percent‚94.04692649841309# proc.memory.rssMB‚ 2943.48828125 disk.dm-1.out‚ 513.95703125 disk.dm-0.in‚ 21359.9296875 disk.sda1.out‚0 disk.sda3.out‚ 513.95703125 disk.drbd2165.out‚0 disk.dm-1.in‚0! disk.nvme0n1.in‚ 21359.9296875 disk.sda2.out‚0# disk.dm-0.out‚96639.87939453125 disk.rbd3.out‚0) proc.memory.percent‚17.96562671661377|óžÜ‚(:“ ¤ƒéÒãê’÷ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚302.343' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚60.46860000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0æx³Ý‚(:û ¤ƒéÒà™»ù disk.sda3.out‚ 514.35546875# disk.nvme0c0n1.in‚ 21360.3515625 disk.rbd1.in‚ 0.01171875 disk.sda.out‚ 514.35546875 disk.sda1.in‚0 disk.drbd2165.in‚0' disk.nvme0c0n1.out‚96654.6337890625' proc.memory.availableMB‚ 975.6015625 network.sent‚ 1.1598521e+07 network.recv‚ 2.531811e+06 disk.drbd2165.out‚0" disk.dm-0.out‚96654.6337890625# proc.memory.rssMB‚ 2943.48828125) proc.memory.percent‚17.96562671661377 cpu‚25.545164590760738 disk.rbd3.in‚0! disk.nvme0n1.in‚ 21360.3515625 disk.sda2.out‚0 disk.rbd1.out‚ 24.8037109375 disk.sda.in‚0 disk.dm-1.out‚ 514.35546875 disk.sda2.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 proc.cpu.threads‚17 disk.sda1.out‚0 disk.sda3.in‚0 disk.rbd3.out‚0% disk.nvme0n1.out‚96654.6337890625 disk.dm-1.in‚0 disk.dm-0.in‚ 21360.3515625$ memory_percent‚94.04540061950684) disk./.usagePercent‚37.35750441438911$ disk./.usageGB‚556.6538124084473eÊåÕ4Þ/ _wandbruntime‚10130  _runtime‚10130;BfÇßÁ & lr-AdamW/pg1‚0.0005572858409698947 & lr-AdamW/pg2‚0.0005572858409698947 trainer/global_step‚6099 $ _timestamp‚1.784299945876832e+09 _runtime‚10130.094098413 _step‚499_–`ÌàÆ _runtime‚10130.094098413 _step‚499 & lr-AdamW/pg1‚0.0005572858409698947 & lr-AdamW/pg2‚0.0005572858409698947 trainer/global_step‚6099 $ _timestamp‚1.784299945876832e+09ó›4á/ _wandbruntime‚10130  _runtime‚10130òz”(âþO VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _runtime‚10130.095549622 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 _step‚500 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1758281628816966 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚6099 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1752553994694656 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5727659550278 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1759049129134592 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 104873843 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚15650269 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚480980470663168 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚480980873290752 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚52041061 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1759050500697600 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚473580890001408 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚52822809 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7399983289344 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5738823249408 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚15650274 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5729004150600 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚52041055 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚52041055 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚269 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚52041061 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1759050500697600 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚473580521910272 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1758282998845256 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4735159500813 (ãƒP KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚52041061 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1759050500697600 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚473580890001408 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚52822809 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7399983289344 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5738823249408 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚15650274 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5729004150600 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚52041055 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚52041055 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚269 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚52041061 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1759050500697600 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚473580521910272 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1758282998845256 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _runtime‚10130.095549622 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 _step‚500 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1758281628816966 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚6099 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1752553994694656 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5727659550278 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1759049129134592 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 104873843 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚15650269 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚480980470663168 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚480980873290752 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119ôÚj«4ä/ _wandbruntime‚10131  _runtime‚10131½ìç¤åˆN [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1753589652004864 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5739748292096 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚473658008317952 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5728583016524 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚52049261 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚17320292 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚481059148860416 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.smÈÿôj#all_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17320549 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1759329426470400 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5729927616958 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1759328054900224 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚52049261 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚481059551480832 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1758561804089790 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚52841274 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚32973104 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7401140542464 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 104880548 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚15652812 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚52841274 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1758560434061388 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1752831876472832 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚52049267 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1753589652004864 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 104890541 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚52049267 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚10131.506935306 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 QDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚52831287 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5738402895360 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚262 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚15652817 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚473658376409088 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5738402895360 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1752831851044864 trainer/global_step‚6099 % _timestamp‚1.7842999472894032e+09 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7401175071744 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008³Òn8'æN I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚52049261 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚481059551480832 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1758561804089790 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚52841274 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚32973104 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7401140542464 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 104880548 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚15652812 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚52841274 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1758560434061388 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1752831876472832 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚52049267 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1753589652004864 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 104890541 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚52049267 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚10131.506935306 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 QDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚52831287 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5738402895360 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚262 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚15652817 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚473658376409088 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5738402895360 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1752831851044864 trainer/global_step‚6099 % _timestamp‚1.7842999472894032e+09 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7401175071744 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1753589652004864 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5739748292096 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚473658008317952 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5728583016524 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚52049261 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚17320292 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚481059148860416 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17320549 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1759329426470400 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5729927616958 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1759328054900224 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995õ. X.4ç/ _wandbruntime‚10131  _runtime‚10131F« è† 2loss/FBt1/assignment_loss‚0.011631413126130497 /loss/FRt1/detection_loss‚0.5313979777107806 % _timestamp‚1.7842999473648934e+09 trainer/global_step‚6099 %loss/total_loss‚7.785957103866865 /loss/FRt2/detection_loss‚1.6269440992727544 1loss/SRqqt1/detection_loss‚0.9336705613347542 /loss/FBt1/detection_loss‚0.5414967159430186 3loss/SRqqt2/assignment_loss‚0.47329298222039473 3loss/FBt2/assignment_loss‚0.0003075634846180151 1loss/SRqqt2/detection_loss‚0.5946763810314557 .loss/FBt2/detection_loss‚0.434758663516026 epoch‚15 _runtime‚10131.582074502 _step‚502 0loss/FRt1/assignment_loss‚1.5037832795910135 0loss/FRt2/assignment_loss‚0.6296650936676599 2loss/SRqqt1/assignment_loss‚0.5043323729682592ç"æVé‹ 3loss/SRqqt2/assignment_loss‚0.47329298222039473 3loss/FBt2/assignment_loss‚0.0003075634846180151 1loss/SRqqt2/detection_loss‚0.5946763810314557 .loss/FBt2/detection_loss‚0.434758663516026 epoch‚15 _runtime‚10131.582074502 _step‚502 0loss/FRt1/assignment_loss‚1.5037832795910135 0loss/FRt2/assignment_loss‚0.6296650936676599 2loss/SRqqt1/assignment_loss‚0.5043323729682592 2loss/FBt1/assignment_loss‚0.011631413126130497 /loss/FRt1/detection_loss‚0.5313979777107806 % _timestamp‚1.7842999473648934e+09 trainer/global_step‚6099 %loss/total_loss‚7.785957103866865 /loss/FRt2/detection_loss‚1.6269440992727544 1loss/SRqqt1/detection_loss‚0.9336705613347542 /loss/FBt1/detection_loss‚0.5414967159430186öå”ê‚(:‰ ³ƒéÒÇÎæö gpu.0.gpu‚80.0 gpu.0.memory‚24, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚319.073' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚63.8146 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0@6"€öë‚(:ë ³ƒéÒ¶·‚ú# disk.nvme0c0n1.in‚ 21361.9453125 disk.rbd1.in‚ 0.01171875 disk.sda.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚514.75 cpu‚25.54497209576584 disk.sda1.in‚0 disk.sda3.out‚514.75! disk.nvme0n1.in‚ 21361.9453125& disk.nvme0n1.out‚96661.78173828125$ memory_percent‚94.04687881469727 disk.rbd0.out‚0 disk.drbd2165.in‚0 disk.rbd3.in‚0 disk.rbd3.out‚0 network.sent‚ 1.1603546e+07 disk.rbd0.in‚0 disk.sda2.out‚0& proc.memory.availableMB‚ 975.359375# proc.memory.rssMB‚ 2943.48828125) proc.memory.percent‚17.96562671661377 disk.sda3.in‚0 disk.sda2.in‚0 disk.rbd1.out‚ 24.8779296875 disk.sda1.out‚0 network.recv‚ 2.535248e+06 disk.dm-0.in‚ 21361.9453125( disk.nvme0c0n1.out‚96661.78173828125* disk./.usagePercent‚37.358032813939786 disk.drbd2165.out‚0 proc.cpu.threads‚17$ disk./.usageGB‚556.6616859436035 disk.sda.out‚514.75# disk.dm-0.out‚96661.78173828125S^nÕ”ì‚(:‰ ƒéÒŽÙëö gpu.0.gpu‚87.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚288.968' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚57.7936 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0xYñõüí‚(:ñ ƒéÒ×’üø# proc.memory.rssMB‚ 2943.48828125 disk.drbd2165.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.dm-0.in‚ 21362.3671875) proc.memory.percent‚17.96562671661377 disk.sda2.out‚0 disk.dm-0.out‚96672.85546875 disk.sda3.in‚0# disk.nvme0c0n1.in‚ 21362.3671875 disk.sda2.in‚0 disk.dm-1.out‚ 515.3359375 disk.sda1.in‚0! disk.nvme0n1.in‚ 21362.3671875 disk.rbd1.in‚ 0.01171875 proc.cpu.threads‚17 network.recv‚ 2.539569e+06 disk.rbd3.out‚0% disk.nvme0c0n1.out‚96672.85546875# disk.nvme0n1.out‚96672.85546875 disk.rbd1.out‚ 24.8779296875 disk.sda3.out‚ 515.3359375$ memory_percent‚94.04745101928711& proc.memory.availableMB‚ 975.265625 disk.rbd3.in‚0 disk.sda.out‚ 515.3359375 disk.sda1.out‚0 cpu‚25.544739331255478 network.sent‚ 1.1666733e+07$ disk./.usageGB‚556.6685104370117 disk.drbd2165.out‚0 disk.dm-1.in‚0) disk./.usagePercent‚37.35849081141854 disk.sda.in‚0€ÈãŸî‚(:” уéÒŠÄõö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚288.151' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚57.630199999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Naá ï‚(: уéÒìî–ù$ disk.nvme0c0n1.in‚21515.55078125$ memory_percent‚94.04909610748291( disk.nvme0c0n1.out‚99373.98681640625 disk.sda2.out‚0 disk.dm-1.out‚ 515.69140625) proc.memory.percent‚17.96562671661377 cpu‚25.54454975762055 disk.rbd3.out‚0 disk.sda2.in‚0 disk.dm-1.in‚0 disk.dm-0.in‚21515.55078125 disk.sda1.out‚0 disk.sda3.in‚0) disk./.usagePercent‚37.35845266629594# disk.dm-0.out‚99373.98681640625 disk.sda1.in‚0 network.recv‚ 2.543285e+06 disk.drbd2165.in‚0 disk.sda.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 proc.cpu.threads‚17$ disk./.usageGB‚556.6679420471191 disk.drbd2165.out‚0 disk.rbd3.in‚0" disk.nvme0n1.in‚21515.55078125 disk.rbd1.in‚ 0.01171875 disk.rbd1.out‚ 24.9404296875 disk.sda.out‚ 515.69140625( proc.memory.availableMB‚ 974.99609375& disk.nvme0n1.out‚99373.98681640625 disk.sda3.out‚ 515.69140625# proc.memory.rssMB‚ 2943.48828125 network.sent‚ 1.1671831e+07ùU o”ð‚(:‰ àƒéÒ®ùˆø gpu.0.gpu‚98.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚265.822' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.1644 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0”fÅV ñ‚(:ÿ àƒéÒ©çÀù& proc.memory.availableMB‚ 974.984375 network.sent‚ 1.1676889e+07 disk.rbd0.out‚0 disk.sda1.in‚0 disk.sda1.out‚0$ disk.nvme0c0n1.in‚21515.75390625 disk.sda.out‚ 515.88671875 disk.dm-1.out‚ 515.88671875 disk.drbd2165.in‚0 disk.rbd3.out‚0 disk.sda.in‚0 disk.rbd0.in‚0 disk.rbd3.in‚0# proc.memory.rssMB‚ 2943.48828125 disk.sda2.in‚0 disk.drbd2165.out‚0 network.recv‚ 2.546431e+06( disk.nvme0c0n1.out‚99383.93798828125 disk.sda2.out‚0$ memory_percent‚94.04916763305664# disk.dm-0.out‚99383.93798828125 cpu‚25.54431119754778) disk./.usagePercent‚37.35889658336031$ disk./.usageGB‚556.6745567321777& disk.nvme0n1.out‚99383.93798828125 disk.rbd1.in‚ 0.01171875 disk.dm-1.in‚0 disk.dm-0.in‚21515.75390625) proc.memory.percent‚17.96562671661377 proc.cpu.threads‚17" disk.nvme0n1.in‚21515.75390625 disk.rbd1.out‚ 24.9404296875 disk.sda3.in‚0 disk.sda3.out‚ 515.88671875[‡ ìžò‚(:“ ïƒéÒ®„üö gpu.0.gpu‚82.0 gpu.0.memory‚25, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚327.145' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚65.42899999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ȉÀ¿ ó‚(:‚ ïƒéÒ馠ù network.sent‚ 1.1681908e+07 disk.sda.in‚0 disk.rbd0.in‚0) proc.memory.percent‚17.96562671661377 disk.sda.out‚ 516.22265625 disk.rbd0.out‚0$ disk./.usageGB‚556.6851959228516 disk.sda2.out‚0' proc.memory.availableMB‚ 974.6953125 proc.cpu.threads‚17 network.recv‚ 2.549978e+06 disk.rbd3.in‚0( disk.nvme0c0n1.out‚99396.39599609375 disk.rbd1.in‚ 0.01171875 cpu‚25.544149665197768 disk.drbd2165.in‚0" disk.nvme0n1.in‚21516.17578125 disk.sda2.in‚0 disk.dm-0.in‚21516.17578125& disk.nvme0n1.out‚99396.39599609375 disk.dm-1.out‚ 516.22265625 disk.sda1.out‚0# proc.memory.rssMB‚ 2943.48828125 disk.rbd3.out‚0$ disk.nvme0c0n1.in‚21516.17578125# disk.dm-0.out‚99396.39599609375$ memory_percent‚94.05093193054199 disk.drbd2165.out‚0) disk./.usagePercent‚37.35961058837331 disk.rbd1.out‚25.06201171875 disk.dm-1.in‚0 disk.sda1.in‚0 disk.sda3.in‚0 disk.sda3.out‚ 516.22265625G4[Ö4ô/ _wandbruntime‚10211  _runtime‚10211׳IÈõ & lr-AdamW/pg2‚0.0005550844280789389 trainer/global_step‚6149 % _timestamp‚1.7843000267994642e+09 _runtime‚10211.016669977 _step‚503 & lr-AdamW/pg1‚0.0005550844280789389FLÍöÇ % _timestamp‚1.7843000267994642e+09 _runtime‚10211.016669977 _step‚503 & lr-AdamW/pg1‚0.0005550844280789389 & lr-AdamW/pg2‚0.0005550844280789389 trainer/global_step‚6149÷³&iS4÷/ _wandbruntime‚10211  _runtime‚10211f4k›øþO S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1772995481570816 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚52451361 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚52451355 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 105709895 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1767211713953792 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1772995481570816 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7459492229120 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚15777419 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7459526765568 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1766448083603456 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5785139180032 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚484914737150976 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5773896663386 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚53258534 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5783793790464 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚33233180 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 E?DeviceStats0 ãiMonitor.on_train_batch_start/oversize_segments.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚52451355 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚53248561 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17455756 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚260 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1772996853133824 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1772996853133824 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1767211713953792 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1772223324867164 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚484914334523392 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚53248561 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚33232920 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1772221954838874 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚53258534 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1772996853133824 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1772996853133824 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1767211713953792 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1772223324867164 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚484914334523392 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚53248561 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚33232920 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1772221954838874 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚53258534 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1772995481570816 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚52451361 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚52451355 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 105709895 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1767211713953792 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1772995481570816 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7459492229120 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚15777419 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7459526765568 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1766448083603456 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5785139180032 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚484914737150976 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5773896663386 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚53258534 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5783793790464 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚33233180 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚52451355 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚53248561 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17455756 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚260 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 JDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚52459567 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 105726593 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚33238206 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚53257039 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1767489714683904 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚53267026 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1767489688510464 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚15779962 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚484993410426880 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7460679104512 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 trainer/global_step‚6149 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 $ _timestamp‚1.784300028211501e+09 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5784718825984 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚53257039 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚477532696793088 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1773275778906624 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚477532328701952 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0ã¦`*üŒN A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5784718825984 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1766725939953664 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚484993007806464 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 105716600 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚246 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1773274407336448 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚477532696793088 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1773275778906624 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚477532328701952 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5774820129632 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚52459567 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 105726593 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚33238206 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚53257039 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1767489714683904 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚53267026 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1767489688510464 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚15779962 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚484993410426880 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7460679104512 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 trainer/global_step‚6149 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 $ _timestamp‚1.784300028211501e+09 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5784718825984 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚53257039 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 EDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7518971090944 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1786162280860782 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚52861661 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1786941834007040 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚52861661 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17592839 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7519005627392 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1780342172512256 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 % _timestamp‚1.7843001079419658e+09 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1786943205570048 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚15904574 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚481329530769408 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1786941834007040 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚33497671 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648Ïž­°^‘ƒP =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚52861655 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚53674313 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1781111750459392 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5831455110656 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚33497408 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1786943205570048 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 106545947 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚488848133769216 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5820133776494 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1781111750459392 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 _runtime‚10292.159330139 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5830109721088 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚53684286 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚488848536396800 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7518971090944 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1786162280860782 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚52861661 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1786941834007040 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚52861661 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 OIDeviceStatsMonitor.on_train_batch_start/inactiÜ(-«ve_split.large_pool.current‚5 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17592839 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7519005627392 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1780342172512256 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 % _timestamp‚1.7843001079419658e+09 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1786943205570048 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚15904574 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚481329530769408 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1786941834007040 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚33497671 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚53684286 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17593097 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448ü\Æ„4’/ _wandbruntime‚10293  _runtime‚10293Ç '“‡N ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 trainer/global_step‚6199 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚481407017177088 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚52869861 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 106552652 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚481406649085952 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1781389751189504 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚52869861 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5832380153344 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚488927209536512 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5832380153344 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 106552652 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚53692778 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚33502912 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚33502666 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚488926806916096 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚241 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚17595554 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1781389725016064 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 106562645 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 106562645 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17595795 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7520192359424 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1787222131342848 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚246 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 $ _timestamp‚1.784300109349376e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚53682791 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5831034756608 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1786441086105204 _runtime‚10293.566770046 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1781389751189504 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚33502666 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚488926806916096 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚241 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚17595554 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1781389725016064 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 106562645 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 106562645 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17595795 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7520192359424 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1787222131342848 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚246 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 $ _timestamp‚1.784300109349376e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚53682791 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5831034756608 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1786441086105204 _runtime‚10293.566770046 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1781389751189504 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 B”ž‚(:‰ ……éÒâ¹€÷ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚277.374' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.4748 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÊFgÆýŸ‚(:ò ……éÒ®ˆÒù disk.rbd0.out‚0 disk.dm-0.in‚ 21737.671875 disk.rbd1.out‚25.40576171875 disk.dm-1.out‚ 519.70703125 disk.rbd0.in‚0 proc.cpu.threads‚17 network.sent‚ 1.1847692e+07 disk.sda1.out‚0 disk.sda3.out‚ 519.70703125 disk.drbd2165.out‚0 disk.sda2.out‚0 cpu‚25.54213549274157 network.recv‚ 2.58455e+06$ disk./.usageGB‚554.8588066101074% disk.nvme0c0n1.out‚104884.1484375# disk.nvme0n1.out‚104884.1484375 disk.sda.in‚0( proc.memory.availableMB‚ 976.17578125 disk.sda3.in‚0 disk.rbd3.out‚0 disk.rbd1.in‚ 0.01171875) disk./.usagePercent‚37.23704006915234 disk.dm-0.out‚104884.1484375 disk.drbd2165.in‚0 disk.rbd3.in‚0 disk.sda2.in‚0 disk.dm-1.in‚0$ memory_percent‚94.04189586639404) proc.memory.percent‚17.96562671661377" disk.nvme0c0n1.in‚ 21737.671875# proc.memory.rssMB‚ 2943.48828125 disk.sda1.in‚0 disk.nvme0n1.in‚ 21737.671875 disk.sda.out‚ 519.70703125‰›F¤” ‚(:‰ ”…éÒËíëö gpu.0.gpu‚95.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚235.414' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚47.0828 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0D\ûû¡‚(:ð ”…éÒ‰ÓÃù# disk.nvme0n1.out‚104906.5703125 disk.sda3.out‚ 520.90234375# proc.memory.rssMB‚ 2943.48828125 disk.dm-1.out‚ 520.90234375 disk.rbd0.in‚0 disk.sda2.out‚0 cpu‚25.541930811066788 disk.sda.in‚0 disk.sda1.in‚0 disk.rbd3.in‚0 disk.rbd3.out‚0" disk.nvme0c0n1.in‚ 21737.890625$ proc.memory.availableMB‚975.6875 proc.cpu.threads‚17 network.sent‚ 1.1852755e+07 network.recv‚ 2.588053e+06$ disk./.usageGB‚552.2342987060547 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.nvme0n1.in‚ 21737.890625 disk.rbd1.out‚25.40576171875 disk.sda.out‚ 520.90234375 disk.dm-1.in‚0 disk.rbd0.out‚0 disk.sda3.in‚0 disk.sda2.in‚0) proc.memory.percent‚17.96562671661377) disk./.usagePercent‚37.06090714160256% disk.nvme0c0n1.out‚104906.5703125$ memory_percent‚94.04487609863281 disk.rbd1.in‚ 0.01171875 disk.dm-0.in‚ 21737.890625 disk.dm-0.out‚104906.5703125´¦V4¢/ _wandbruntime‚10372  _runtime‚103726Kfȣ trainer/global_step‚6249 % _timestamp‚1.7843001880423787e+09 _runtime‚10372.259546484 _step‚511 & lr-AdamW/pg1‚0.0005506414426456957 & lr-AdamW/pg2‚0.0005506414426456957W•u›Í¤Ç & lr-AdamW/pg1‚0.0005506414426456957 & lr-AdamW/pg2‚0.0005506414426456957 trainer/global_step‚6249 % _timestamp‚1.7843001880423787e+09 _runtime‚10372.259546484 _step‚511ÿŽv}†4¥/ _wandbruntime‚10372  _runtime‚10372K>÷ƒ ¦þO _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7578728117248 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚54110038 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 107381999 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1800103976910980 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1800888186443264 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚53271961 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 JBDeviceSta¬¡?ä#tsMonitor.on_train_batch_start/segment.small_pool.current‚820 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 107372020 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1800889558006272 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17730623 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚485203483062272 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚265 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5867715489924 trainer/global_step‚6249 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚54100065 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚492782211179520 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 _runtime‚10372.260751859 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1795011786964992 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1795011760791552 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚33762082 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚54100065 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1800102606882690 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7578762653696 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚53271955 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17730363 _step‚512 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚33762347 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚16031719…rãz (§ƒP TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚54100065 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1800102606882690 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7578762653696 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚53271955 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17730363 _step‚512 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚33762347 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚16031719 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7578728117248 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚54110038 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 107381999 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1800103976910980 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1800888186443264 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚53271961 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 107372020 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1800889558006272 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17730623 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚485203483062272 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚265 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5867715489924 trainer/global_step‚6249 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚54100065 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚492782211179520 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 _runtime‚10372.260751859 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1795011786964992 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1795011760791552 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚33762082 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648€wÅÝ4¨/ _wandbruntime‚10373  _runtime‚10373]|•;'©ˆN GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5877350687232 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5878696083968 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚485280969469952 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚54108543 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚53280167 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚254 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _runtime‚10373.645309351 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5868638956282 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7579952430080 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 % _timestamp‚1.7843001894279468e+09 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1801167112208896 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1800382782155514 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚259 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚33767367 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚53280161 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 107398697 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5867294355848 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926‹µ¶t® ªN T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1801167112208896 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1800382782155514 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚259 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚33767367 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚53280161 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 107398697 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5867294355848 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5877350687232 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5878696083968 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚485280969469952 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚54108543 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚53280167 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚254 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _runtime‚10373.645309351 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5868638956282 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7579952430080 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 % _timestamp‚1.7843001894279468e+09òe÷X4«/ _wandbruntime‚10373  _runtime‚10373›ûÉÅ ¬‡ &loss/total_loss‚7.8835057932511905 % _timestamp‚1.7843001895148823e+09 .loss/FRt1/detection_loss‚0.531348626849729 /loss/FRt2/detection_loss‚1.5095176317077625 epoch‚15 3loss/FBt1/assignment_loss‚0.0024862529505172274 /loss/FBt1/detection_loss‚0.9344796436613031 _runtime‚10373.731883131 0loss/FRt2/assignment_loss‚0.5997033280895111 2loss/FBt2/assignment_loss‚0.011017391765542986 0loss/SRqqt1/detection_loss‚0.795685819687386 trainer/global_step‚6249 _step‚514 0loss/FRt1/assignment_loss‚1.5141119277648378 2loss/SRqqt1/assignment_loss‚0.5646662547132287 3loss/SRqqt2/assignment_loss‚0.46759170351865453 1loss/SRqqt2/detection_loss‚0.6124407981156266 0loss/FBt2/detection_loss‚0.34045641442709085¹–²V­Œ .loss/FRt1/detection_loss‚0.531348626849729 /loss/FRt2/detection_loss‚1.5095176317077625 epoch‚15 3loss/FBt1/assignment_loss‚0.0024862529505172274 /loss/FBt1/detection_loss‚0.9344796436613031 _runtime‚10373.731883131 0loss/FRt2/assignment_loss‚0.5997033280895111 2loss/FBt2/assignment_loss‚0.011017391765542986 0loss/SRqqt1/detection_loss‚0.795685819687386 trainer/global_step‚6249 _step‚514 0loss/FRt1/assignment_loss‚1.5141119277648378 2loss/SRqqt1/assignment_loss‚0.5646662547132287 3loss/SRqqt2/assignment_loss‚0.46759170351865453 1loss/SRqqt2/detection_loss‚0.6124407981156266 0loss/FBt2/detection_loss‚0.34045641442709085 &loss/total_loss‚7.8835057932511905 % _timestamp‚1.7843001895148823e+09‚ð4 LŸ®‚(:” £…éÒÝ•óö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚298.392' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚59.678399999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¡óY¯‚(:ú £…éÒì÷¡ø disk.sda.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 524.78515625$ disk.dm-0.out‚104919.82275390625 disk.rbd1.in‚ 0.01171875 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda3.in‚0) proc.memory.percent‚17.96562671661377 disk.sda2.in‚0$ disk./.usageGB‚550.1690711975098 disk.rbd1.out‚25.46826171875 disk.sda3.out‚ 524.78515625 disk.rbd3.out‚0 disk.sda.out‚ 524.78515625! disk.nvme0c0n1.in‚ 21738.09375 disk.drbd2165.in‚0 disk.drbd2165.out‚0) disk.nvme0c0n1.out‚104919.82275390625# memory_percent‚94.0424919128418 network.recv‚ 2.591618e+06 disk.dm-0.in‚ 21738.09375 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd3.in‚0 disk.nvme0n1.in‚ 21738.09375 disk.sda2.out‚0) disk./.usagePercent‚36.92230799066281' disk.nvme0n1.out‚104919.82275390625& proc.memory.availableMB‚ 976.078125 network.sent‚ 1.1857862e+07# proc.memory.rssMB‚ 2943.48828125 cpu‚25.541684607248303 proc.cpu.threads‚17ðèƒ ”°‚(:‰ ²…éÒäýÛö gpu.0.gpu‚90.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚286.263' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚57.2526 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0éÝ˼±‚(:õ ²…éÒ¬´íø$ disk./.usageGB‚550.1673736572266 disk.sda.out‚ 526.83984375 disk.sda1.in‚0 disk.rbd3.out‚0& disk.nvme0c0n1.out‚104937.17578125) proc.memory.percent‚17.96562671661377 disk.sda.in‚0 disk.sda3.out‚ 526.83984375 disk.drbd2165.in‚0 disk.rbd1.in‚ 0.01171875$ memory_percent‚94.04878616333008 network.sent‚ 1.1920886e+07 disk.dm-1.out‚ 526.83984375 disk.drbd2165.out‚0 disk.rbd1.out‚25.46826171875 disk.sda3.in‚0 disk.rbd3.in‚0& proc.memory.availableMB‚ 975.046875 disk.rbd0.out‚0 disk.nvme0n1.in‚ 21738.515625 disk.sda2.in‚0# proc.memory.rssMB‚ 2943.48828125 network.recv‚ 2.595705e+06 disk.rbd0.in‚0 disk.dm-0.in‚ 21738.515625 cpu‚25.541477589940026) disk./.usagePercent‚36.92219406731007" disk.nvme0c0n1.in‚ 21738.515625 disk.sda2.out‚0 proc.cpu.threads‚17 disk.dm-1.in‚0! disk.dm-0.out‚104937.17578125 disk.sda1.out‚0$ disk.nvme0n1.out‚104937.17578125!6H”²‚(:‰ Á…éÒÒžðö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚125.932' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚25.1864 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0óãƒ0 ³‚(:‚ Á…éÒÀáãø disk.sda3.in‚0 disk.rbd1.in‚ 0.01171875 disk.dm-1.out‚ 528.65234375) disk./.usagePercent‚36.92253506934568$ disk./.usageGB‚550.1724548339844 disk.sda1.out‚0( disk.nvme0c0n1.out‚104952.5498046875# proc.memory.rssMB‚ 2943.48828125 disk.sda.out‚ 528.65234375$ disk.nvme0c0n1.in‚21849.79296875' proc.memory.availableMB‚ 974.7890625 disk.rbd0.in‚0 disk.drbd2165.out‚0& disk.nvme0n1.out‚104952.5498046875 disk.rbd1.out‚25.54638671875$ memory_percent‚94.05035972595215 disk.drbd2165.in‚0 disk.sda2.out‚0) proc.memory.percent‚17.96562671661377 cpu‚25.541250633151158 network.sent‚ 1.1925939e+07 disk.sda.in‚0 disk.rbd0.out‚0 proc.cpu.threads‚17 disk.sda2.in‚0# disk.dm-0.out‚104952.5498046875 disk.sda1.in‚0 disk.sda3.out‚ 528.65234375 disk.rbd3.in‚0 disk.rbd3.out‚0" disk.nvme0n1.in‚21849.79296875 network.recv‚ 2.599244e+06 disk.dm-1.in‚0 disk.dm-0.in‚21849.79296875kN̤Ÿ´‚(:” Ð…éÒŒäÛö gpu.0.gpu‚90.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚300.279' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚60.055800000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ên¹Ê µ‚(: Ð…éÒƦù$ disk.nvme0c0n1.in‚21849.99609375" disk.nvme0n1.in‚21849.99609375 disk.rbd1.out‚25.54638671875 disk.sda3.in‚0 disk.drbd2165.out‚0& disk.nvme0n1.out‚104966.9443359375 disk.sda2.in‚0# disk.dm-0.out‚104966.9443359375* disk./.usagePercent‚36.922790564864854$ disk./.usageGB‚550.1762619018555 disk.sda.out‚ 530.66015625$ memory_percent‚94.04549598693848 cpu‚25.5410466425962 disk.dm-1.in‚0 disk.dm-1.out‚ 530.66015625 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.rbd3.in‚0 disk.rbd3.out‚0 network.recv‚ 2.602478e+06( disk.nvme0c0n1.out‚104966.9443359375 disk.sda2.out‚0 disk.dm-0.in‚21849.99609375) proc.memory.percent‚17.96562671661377 proc.cpu.threads‚17 network.sent‚ 1.1930946e+07 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda1.in‚0' proc.memory.availableMB‚ 975.5859375 disk.sda.in‚0 disk.sda3.out‚ 530.66015625 disk.rbd1.in‚ 0.01171875# proc.memory.rssMB‚ 2943.48828125¦t›Ìž¶‚(:“ ß…éÒ®€÷ gpu.0.gpu‚17.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚41.0 gpu.0.powerWatts‚158.288' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚31.657600000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¿sã ·‚(:€ ß…éÒ´ˆ£ù network.recv‚ 2.605804e+06 disk.rbd0.in‚0 disk.sda1.out‚0 disk.sda3.in‚0 disk.drbd2165.in‚0 disk.rbd3.in‚0 disk.rbd1.in‚ 0.01171875) proc.memory.percent‚17.96562671661377 disk.sda2.out‚0# proc.memory.rssMB‚ 2943.48828125 disk.rbd0.out‚0 disk.dm-0.in‚21850.41796875# disk.dm-0.out‚104982.1318359375& disk.nvme0n1.out‚104982.1318359375 disk.sda2.in‚0 cpu‚25.54084309014334 disk.dm-1.in‚0 disk.dm-1.out‚ 531.41015625" disk.nvme0n1.in‚21850.41796875 disk.sda.out‚ 531.41015625 disk.sda1.in‚0 disk.drbd2165.out‚0# memory_percent‚94.0471887588501 network.sent‚ 1.1935981e+07 disk.sda3.out‚ 531.41015625 disk.rbd3.out‚0# disk./.usageGB‚550.189136505127$ disk.nvme0c0n1.in‚21850.41796875( disk.nvme0c0n1.out‚104982.1318359375 proc.cpu.threads‚17) disk./.usagePercent‚36.92365459029291 disk.sda.in‚0 disk.rbd1.out‚25.60888671875( proc.memory.availableMB‚ 975.30859375€¶E4¸/ _wandbruntime‚10450  _runtime‚10450Lº’ȹ trainer/global_step‚6299 % _timestamp‚1.7843002667163768e+09 _runtime‚10450.933530959 _step‚515 & lr-AdamW/pg1‚0.0005484001554680605 & lr-AdamW/pg2‚0.0005484001554680605ÿu•˜ÍºÇ trainer/global_step‚6299 % _timestamp‚1.7843002667163768e+09 _runtime‚10450.933530959 _step‚515 & lr-AdamW/pg1‚0.0005484001554680605 & lr-AdamW/pg2‚0.0005484001554680605ƒy’„À4»/ _wandbruntime‚10450  _runtime‚10450_3®(¼þO TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚54525817 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1814834538879488 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚54535790 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 _runtime‚10450.934863665 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5922741582336 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚489078171537408 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚53682255 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1808911797297152 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 400530432 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚16158869 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1814835910442496 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚53682255 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚16158874 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 108208072 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚54525817 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1814834538879488 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5922741582336 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 % _timestamp‚1.7843002667174633e+09 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1808911823470592 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚34027340 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1814044302932888 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚496716993146880 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚53682261 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚489077803446272 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚239 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7638821609472 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5924086971904 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚34027101 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 400530432 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚16158869 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1814835910442496 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚53682255 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚16158874 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 108208072 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚54525817 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1814834538879488 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5922741582336 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 % _timestamp‚1.7843002667174633e+09 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1808911823470592 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚34027340 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1814044302932888 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 ^JDeviceStatsz'½ {Monitor.on_train_batch_start/inactive_split_bytes.all.allocated‚496716993146880 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚53682261 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚489077803446272 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚239 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7638821609472 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5924086971904 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚34027101 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1814834538879488 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚54535790 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 _runtime‚10450.934863665„uÃiž4¾/ _wandbruntime‚10452  _runtime‚10452˜r™ '¿‡N FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 108234749 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1809189824200704 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1809189824200704 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚34032372 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚17870960 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚10452.31913439 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚16161412 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1814323108177422 _step‚517 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7639996009472 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 108234749 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚54544282 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚232 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚54534295 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5923666617856 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚34032604 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1809189798027264 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1815113464645120 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1814321738149020 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648óØ'ÀŒN VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚16161417 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚53690467 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 108234749 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1809189824200704 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1809189824200704 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚34032372 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚17870960 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚10452.31913439 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚16161412 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1814323108177422 _step‚517 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7639996009472 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 108234749 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚54544282 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚232 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚54534295 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5923666617856 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚34032604 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1809189798027264 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1815113464645120 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1814321738149020 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 FDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1827983258926506 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5969057512960 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚265 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚54092555 _runtime‚10529.651163672 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1828782262878720 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚54961542 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚54092561 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚18006301 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚16286024 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1828780891315712 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1822811859976192 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7698646896640 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 trainer/global_step‚6349 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚54951569 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5970402902528 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5969057512960 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚270 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5960189716140 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5958845115818 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚54961542 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7698612360192 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 109054103 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1827984628954796 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1822811833802752 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚34292590 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 $ _timestamp‚1.784300345433816e+09 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1828780891315712 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5970402902528 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚54092555 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚500651138818048 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚500650736190464 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1828780891315712 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5970402902528 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚54092555 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚500651138818048 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚500650736190464 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1827983258926506 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5969057512960 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚265 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚54092555 _runtime‚10529.651163672 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1828782262878720 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚54961542 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚54092561 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚18006301 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚16286024 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1828780891315712 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1822811859976192 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7698646896640 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 trainer/global_step‚6349 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚54951569 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5970402902528 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5969057512960 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚270 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5960189716140 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5958845115818 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚54961542 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7698612360192 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 109054103ˆ(¥<¿Ô‚(:… ¹†éÒ‰ø gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚344.93' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚68.986 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ñéÂÕ‚(:ø ¹†é҈ţù network.recv‚ 2.626921e+06 disk.sda2.in‚0 disk.rbd3.out‚0& disk.nvme0n1.out‚105051.9833984375 network.sent‚ 1.2023979e+07 disk.rbd1.out‚ 25.91796875 disk.rbd0.out‚0# disk.dm-0.out‚105051.9833984375 disk.rbd0.in‚0 disk.sda3.out‚ 533.5859375 disk.drbd2165.out‚0# disk.nvme0c0n1.in‚ 22519.6953125) proc.memory.percent‚17.96562671661377 proc.cpu.threads‚17 disk.rbd1.in‚ 0.01171875 disk.sda.in‚0 disk.dm-1.out‚ 533.5859375 disk.dm-0.in‚ 22519.6953125 disk.rbd3.in‚0( disk.nvme0c0n1.out‚105051.9833984375 cpu‚25.539499868801236 disk.dm-1.in‚0 disk.sda1.out‚0 disk.sda3.in‚0! disk.nvme0n1.in‚ 22519.6953125$ memory_percent‚94.04735565185547% proc.memory.availableMB‚ 975.28125* disk./.usagePercent‚36.926544915354484 disk.drbd2165.in‚0# proc.memory.rssMB‚ 2943.48828125$ disk./.usageGB‚550.2322044372559 disk.sda1.in‚0 disk.sda2.out‚0 disk.sda.out‚ 533.5859375êY4Ö/ _wandbruntime‚10531  _runtime‚10531u¾“'׈N J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚54970034 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5969982548480 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5971327945216 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1829059817081344 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 109060808 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18009054 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1823089834532864 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 109070801 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1822302321016832 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚34297616 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1823089860706304 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1823089834532864 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5971327945216 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7699842216960 _step‚521 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚54100767 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚264 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1822302295588864 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1829061188651520 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5961113182498 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 109070801 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 _runtime‚10531.036192172 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1823089860706304 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚493029978329088 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1828262064170928 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1829061188651520 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 trainer/global_step‚6349 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚54960047 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7699807687680 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚259 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚54100767 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚54100761 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚493029610237952 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7843003468188357e+09 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚54100761 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚16288562 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚500729820546048 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5969982548480 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚16288567 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1828263434199330 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚54960047 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7843003468188357e+09 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚54100761 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚16288562 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚500729820546048 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5969982548480 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚16288567 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1828263434199330 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚54960047 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1822302321016832 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚34297616 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1823089860706304 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1823089834532864 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5971327945216 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7699842216960 _step‚521 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚54100767 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚264 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1822302295588864 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1829061188651520 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5961113182498 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 109070801 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 _runtime‚10531.036192172 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1823089860706304 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚493029978329088 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1828262064170928 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1829061188651520 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 trainer/global_step‚6349 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚54960047 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7699807687680 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚259 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820‰é¹w¥4Ù/ _wandbruntime‚10531  _runtime‚10531´×Ú 2loss/SRqqt1/assignment_loss‚0.5968748831156406 2loss/SRqqt2/assignment_loss‚0.4842566970707571 /loss/FBt1/detection_loss‚0.5055611585808355 1loss/FBt1/assignment_loss‚0.07818267376182568 1loss/SRqqt1/detection_loss‚1.0934117528513374 trainer/global_step‚6349 0loss/FRt1/assignment_loss‚1.5431822678655216 3loss/FBt2/assignment_loss‚0.0007864850341591358 -loss/FRt1/detection_loss‚0.51437656946898 .loss/FRt2/detection_loss‚1.457789849782766 1loss/SRqqt2/detection_loss‚0.6007705485631787 %loss/total_loss‚8.030391254214097 _runtime‚10531.122609913 _step‚522 0loss/FRt2/assignment_loss‚0.6270656751603195 /loss/FBt2/detection_loss‚0.5281326929587753 epoch‚15 $ _timestamp‚1.784300346905684e+09¨EÒ Û† epoch‚15 $ _timestamp‚1.784300346905684e+09 2loss/SRqqt1/assignment_loss‚0.5968748831156406 2loss/SRqqt2/assignment_loss‚0.4842566970707571 /loss/FBt1/detection_loss‚0.5055611585808355 1loss/FBt1/assignment_loss‚0.07818267376182568 1loss/SRqqt1/detection_loss‚1.0934117528513374 trainer/global_step‚6349 0loss/FRt1/assignment_loss‚1.5431822678655216 3loss/FBt2/assignment_loss‚0.0007864850341591358 -loss/FRt1/detection_loss‚0.51437656946898 .loss/FRt2/detection_loss‚1.457789849782766 1loss/SRqqt2/detection_loss‚0.6007705485631787 %loss/total_loss‚8.030391254214097 _runtime‚10531.122609913 _step‚522 0loss/FRt2/assignment_loss‚0.6270656751603195 /loss/FBt2/detection_loss‚0.5281326929587753ŠF‘¶ŸÜ‚(:” ȆéÒýøÜö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚292.983' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚58.596599999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0BiÄÉÝ‚(:ø ȆéÒˆÊÄù disk.sda.out‚ 533.7109375' proc.memory.availableMB‚ 975.7578125) disk./.usagePercent‚36.92711606816338 disk.rbd0.out‚0 disk.rbd3.in‚0 disk.rbd1.in‚ 0.01171875 disk.sda.in‚0$ memory_percent‚94.04444694519043# proc.memory.rssMB‚ 2943.48828125) proc.memory.percent‚17.96562671661377 disk.dm-0.in‚ 22683.3515625 disk.drbd2165.in‚0 disk.rbd3.out‚0 network.sent‚ 1.2075586e+07$ disk./.usageGB‚550.2407150268555# disk.dm-0.out‚105063.4716796875 disk.drbd2165.out‚0 disk.rbd1.out‚ 25.91796875 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.sda3.in‚0# disk.nvme0c0n1.in‚ 22683.3515625 disk.sda1.in‚0 disk.sda2.out‚0 disk.sda1.out‚0! disk.nvme0n1.in‚ 22683.3515625 disk.dm-1.out‚ 533.7109375 network.recv‚ 2.630944e+06& disk.nvme0n1.out‚105063.4716796875 disk.sda2.in‚0 cpu‚25.53925323206918 proc.cpu.threads‚17 disk.sda3.out‚ 533.7109375( disk.nvme0c0n1.out‚105063.4716796875e”ɔނ(:‰ ׆éÒ…û÷ gpu.0.gpu‚32.0 gpu.0.memory‚10, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚46.0 gpu.0.powerWatts‚347.469' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚69.4938 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0nC÷ì ß‚(:€ ׆éÒ—ÿ²ù disk.sda.out‚ 534.05078125$ memory_percent‚94.04323101043701 disk.drbd2165.in‚0' disk.nvme0n1.out‚105071.81201171875( proc.memory.availableMB‚ 975.95703125 disk.sda1.out‚0 disk.sda2.in‚0 disk.dm-1.out‚ 534.05078125 network.sent‚ 1.2128565e+07 disk.sda1.in‚0 disk.rbd1.in‚ 0.01171875 disk.dm-1.in‚0 disk.rbd0.in‚0# proc.memory.rssMB‚ 2943.48828125 disk.sda3.out‚ 534.05078125 disk.drbd2165.out‚0 disk.rbd3.in‚0 disk.rbd3.out‚0# disk.nvme0c0n1.in‚ 22683.7734375 disk.sda.in‚0* disk./.usagePercent‚36.927601970463364$ disk./.usageGB‚550.2479553222656 disk.sda3.in‚0 disk.rbd1.out‚ 25.98046875 disk.sda2.out‚0 cpu‚25.53908019322748 disk.rbd0.out‚0) proc.memory.percent‚17.96562671661377 proc.cpu.threads‚17 network.recv‚ 2.635222e+06 disk.dm-0.in‚ 22683.7734375$ disk.dm-0.out‚105071.81201171875) disk.nvme0c0n1.out‚105071.81201171875! disk.nvme0n1.in‚ 22683.7734375h:/rŸà‚(:” æ†éÒ‚ëøö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚301.343' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚60.268600000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0•¨  á‚(:€ æ†éÒŽÿø disk.drbd2165.out‚0 disk.rbd3.out‚0 network.sent‚ 1.2133044e+07 network.recv‚ 2.638678e+06$ disk./.usageGB‚550.2519073486328 disk.sda2.out‚0$ disk.dm-0.out‚105086.91259765625 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.in‚0 disk.rbd1.out‚ 25.98046875 disk.sda.out‚ 534.23046875 disk.dm-1.in‚0 disk.dm-1.out‚ 534.23046875 disk.rbd0.in‚0 disk.dm-0.in‚ 22683.9765625' proc.memory.availableMB‚ 975.7109375! disk.nvme0n1.in‚ 22683.9765625 disk.sda.in‚0 disk.rbd3.in‚0) disk.nvme0c0n1.out‚105086.91259765625 cpu‚25.538885347184728 proc.cpu.threads‚17 disk.rbd1.in‚ 0.01171875* disk./.usagePercent‚36.927867194268835 disk.sda3.in‚0' disk.nvme0n1.out‚105086.91259765625 disk.sda3.out‚ 534.23046875 disk.drbd2165.in‚0$ memory_percent‚94.04473304748535# proc.memory.rssMB‚ 2943.48828125 disk.rbd0.out‚0# disk.nvme0c0n1.in‚ 22683.9765625) proc.memory.percent‚17.96562671661377Û±™Ÿâ‚(:” õ†éÒîô³ö gpu.0.gpu‚92.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚269.302' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚53.860400000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0“›Ýà ã‚(:€ õ†éÒ±äûø disk.sda1.in‚0 disk.sda3.in‚0 cpu‚25.538689215021936 disk.rbd3.out‚0 disk.rbd1.in‚ 0.01171875 disk.sda.in‚0 disk.rbd0.out‚0 disk.dm-1.in‚0 disk.sda3.out‚ 534.55859375 disk.drbd2165.out‚0$ memory_percent‚94.04323101043701 disk.sda.out‚ 534.55859375 disk.rbd3.in‚0' disk.nvme0n1.out‚105094.76611328125( proc.memory.availableMB‚ 975.95703125# proc.memory.rssMB‚ 2943.48828125) proc.memory.percent‚17.96562671661377* disk./.usagePercent‚36.928339016154446# disk.nvme0c0n1.in‚ 22684.1953125 disk.sda2.in‚0 disk.dm-1.out‚ 534.55859375 disk.sda1.out‚0 network.sent‚ 1.213865e+07$ disk.dm-0.out‚105094.76611328125 network.recv‚ 2.642262e+06) disk.nvme0c0n1.out‚105094.76611328125 disk.rbd1.out‚ 26.05859375 disk.rbd0.in‚0 disk.drbd2165.in‚0 proc.cpu.threads‚17$ disk./.usageGB‚550.2589378356934! disk.nvme0n1.in‚ 22684.1953125 disk.sda2.out‚0 disk.dm-0.in‚ 22684.19531256'4é/ _wandbruntime‚10608  _runtime‚10608$‚R(êüO HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1842728615314944 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚496826444214272 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6016718833152 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1841923584948414 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6005082228926 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 trainer/global_step‚6399 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7758519077888 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚55387294 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 $ _timestamp‚1.784300423985518e+09 _step‚524 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚54502855 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚250 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚496826812305408 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1841924954976704 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚54502855 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚255 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 109890155 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1842728615314944 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚54502861 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚54502861 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1842727243751936 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1836711896481792 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7758484541440 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6016718833152 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1841923584948414 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6005082228926 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 trainer/global_step‚6399 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 cQDeviclÃaXeStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7758519077888 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚55387294 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 $ _timestamp‚1.784300423985518e+09 _step‚524 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚54502855 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚250 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚496826812305408 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1841924954976704 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚54502855 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚255 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 109890155 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1842728615314944 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚54502861 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚54502861 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1842727243751936 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1836711896481792 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7758484541440 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6017643875840 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6007350295606 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 109906853 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18146707 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1842203760221238 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚34562663 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1842202390192836 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 109896860 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚55395786 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7759682836480 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚6399 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 109906853 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6016298479104 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 _runtime‚10609.588148868 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 109896860 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1843007541087744 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1836196384497664 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚54511067 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚16415712 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚504663613458432 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚54511061 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚504664013981696 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚55395786 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1836989897211904 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚34562419 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 400523264 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚32432128 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 % _timestamp‚1.7843004253707159e+09 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚54511061ßü„ì'îN OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚34562663 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1842202390192836 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 109896860 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚55395786 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7759682836480 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚6399 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 109906853 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6016298479104 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 _runtime‚10609.588148868 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 109896860 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1843007541087744 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1836196384497664 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚54511067 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚16415712 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚504663613458432 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚54511061 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚504664013981696 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚55395786 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1836989897211904 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚34562419 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 400523264 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚32432128 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 % _timestamp‚1.7843004253707159e+09 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚54511061 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6016298479104 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚55385799 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1836989871038464 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚54511067 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚16415717 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6017643875840 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6007350295606 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 109906853 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18146707 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1842203760221238î`¡4ï/ _wandbruntime‚10609  _runtime‚10609|_° ðƒ epoch‚15 2loss/SRqqt1/assignment_loss‚0.5354690437346888 2loss/FBt1/assignment_loss‚0.016507854997663163 trainer/global_step‚6399 $ _timestamp‚1.784300425457772e+09 _runtime‚10609.67475143 0loss/FRt1/assignment_loss‚1.5670819805672411 /loss/FRt2/assignment_loss‚0.697563546519835 2loss/SRqqt2/assignment_loss‚0.5108888350608152 2loss/FBt2/assignment_loss‚0.006364113199764715 /loss/FRt2/detection_loss‚1.5243369229770836 1loss/SRqqt1/detection_loss‚0.9018408740478361 _step‚526 1loss/SRqqt2/detection_loss‚0.5975554617637485 %loss/total_loss‚8.028665446828064 /loss/FRt1/detection_loss‚0.5388639634159719 /loss/FBt1/detection_loss‚0.7049238120234804 0loss/FBt2/detection_loss‚0.42726903851993764§Ð{+ñˆ trainer/global_step‚6399 $ _timestamp‚1.784300425457772e+09 _runtime‚10609.67475143 0loss/FRt1/assignment_loss‚1.5670819805672411 /loss/FRt2/assignment_loss‚0.697563546519835 2loss/SRqqt2/assignment_loss‚0.5108888350608152 2loss/FBt2/assignment_loss‚0.006364113199764715 /loss/FRt2/detection_loss‚1.5243369229770836 1loss/SRqqt1/detection_loss‚0.9018408740478361 _step‚526 1loss/SRqqt2/detection_loss‚0.5975554617637485 %loss/total_loss‚8.028665446828064 /loss/FRt1/detection_loss‚0.5388639634159719 /loss/FBt1/detection_loss‚0.7049238120234804 0loss/FBt2/detection_loss‚0.42726903851993764 epoch‚15 2loss/SRqqt1/assignment_loss‚0.5354690437346888 2loss/FBt1/assignment_loss‚0.016507854997663163Ž´¥©œò‚(:‘ “‡éÒÌÃéø gpu.0.gpu‚1.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚330.528' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚66.10560000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0(ÄÕÍ ó‚(: “‡éÒ¥ƒú disk.sda.out‚ 535.27734375 disk.dm-1.out‚ 535.27734375 disk.drbd2165.in‚0 disk.rbd1.out‚ 26.12109375 disk.rbd0.in‚0 cpu‚25.54075932616688 disk.sda.in‚0 disk.rbd0.out‚0) proc.memory.percent‚17.96562671661377) disk.nvme0c0n1.out‚105119.00732421875 disk.dm-1.in‚0 disk.sda1.in‚0 disk.sda3.in‚0( proc.memory.availableMB‚ 975.24609375 network.recv‚ 2.649147e+06( disk./.usagePercent‚36.9291267513373$ disk./.usageGB‚550.2706756591797 disk.rbd3.in‚0 disk.sda2.in‚0 disk.dm-0.in‚22825.21484375# proc.memory.rssMB‚ 2943.48828125$ disk.nvme0c0n1.in‚22825.21484375$ disk.dm-0.out‚105119.00732421875 disk.sda3.out‚ 535.27734375 disk.drbd2165.out‚0 network.sent‚ 1.2148855e+07 disk.rbd3.out‚0' disk.nvme0n1.out‚105119.00732421875 disk.sda2.out‚0 disk.rbd1.in‚ 0.01171875 proc.cpu.threads‚17" disk.nvme0n1.in‚22825.21484375 disk.sda1.out‚0$ memory_percent‚94.04757022857666#×TU‘ô‚(:† ¢‡éÒäÄãö gpu.0.gpu‚1.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚38.0 gpu.0.powerWatts‚110.125' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.025 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0r?1õ‚(:ü ¢‡éÒ•ÐÜø) disk./.usagePercent‚36.92958014068044 disk.sda1.out‚0 disk.rbd3.in‚0) disk.nvme0c0n1.out‚105131.56689453125' disk.nvme0n1.out‚105131.56689453125 disk.sda.in‚0# proc.memory.rssMB‚ 2943.48828125 cpu‚25.54429723161187 proc.cpu.threads‚17 disk.dm-0.in‚ 22825.453125 disk.sda1.in‚0 disk.sda3.out‚ 535.55078125 disk.nvme0n1.in‚ 22825.453125 disk.sda2.out‚0 network.sent‚ 1.2211452e+07 network.recv‚ 2.653588e+06 disk.rbd3.out‚0 disk.sda.out‚ 535.55078125 disk.dm-1.out‚ 535.55078125$ disk./.usageGB‚550.2774314880371 disk.sda2.in‚0 disk.rbd0.out‚0$ disk.dm-0.out‚105131.56689453125 disk.drbd2165.out‚0 disk.rbd1.in‚ 0.01171875 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.drbd2165.in‚0 disk.sda3.in‚0" disk.nvme0c0n1.in‚ 22825.453125 disk.rbd1.out‚ 26.12109375$ memory_percent‚94.04785633087158( proc.memory.availableMB‚ 975.19921875) proc.memory.percent‚17.96562671661377>®Á4ö/ _wandbruntime‚10635  _runtime‚10635£Ö’‘÷ßz =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.8571428656578064 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9994766712188721 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 *jet/accuracy_2_of_2‚0.1269209384918213 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 8"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.706550657749176 +jet/accuracy_3_of_4‚0.09748660027980804 particle/accuracy_5_of_5‚0 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6691478490829468 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 jet/accuracy_4_of_5‚0 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.747543275™ŠœÔ93562927 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9996031522750854 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 _step‚527 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.24707907438278198 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.07692307978868484 jet/accuracy_1_of_5‚1 jet/accuracy_5_of_5‚0 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9368979334831238 6 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.329354465007782 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.4148986339569092 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.5217329859733582 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.2741747200489044 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2747748792171478 & Purity/*FRt1SRqqt1FBt/FBt_purity‚1 8"Purity/0FRt*SRqqt0FBt/event_purity‚0.397561252117157 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5819577574729919 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9993734955787659 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9890425801277161 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2838568687438965 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7792965769767761 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.6094048023223877 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 :"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.48818162083625793 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 :"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.45126041769981384 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8199924230575562 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.6144219040870667 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8009259104728699 *jet/accuracy_2_of_4‚0.7553048133850098 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.07407407462596893 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.41203704476356506 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4140625 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 particle/accuracy_3_of_5‚1 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.4849902093410492 & Purity/0FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4619373679161072 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.1838648021221161 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 8"Purity/1FRt0SRqqt2FBt/event_purity‚0.819652259349823 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5096545219421387 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.19043691456317902 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.050279539078474045 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9780850410461426 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_2_of_3‚1 particle/accuracy_3_of_3‚1 /particle/accuracy_4_of_4‚0.9692972302436829 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.2568216621875763 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5750319361686707 particle/accuracy_1_of_4‚1 particle/accuracy_2_of_4‚1 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5757237672805786 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6033943891525269 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9966116547584534 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.26230376958847046 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.5797989368438721 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8683283925056458 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 *jet/accuracy_2_of_3‚0.5011165142059326 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9964489936828613 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.32165414094924927 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2846483290195465 /particle/accuracy_3_of_4‚0.9968254566192627 (particle/accuracy‚0.8456218242645264 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6619914174079895 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3207266926765442 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4764881730079651 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 particle/accuracy_1_of_1‚1 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.11039062589406967 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.4264788031578064 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.5311021208763123 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9966704249382019 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.27296918630599976 5 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.98738032579422 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.19741328060626984 'particle/f_score‚0.6401387453079224 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8279668092727661 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9965357184410095 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 8"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.609519898891449 *jet/accuracy_1_of_4‚0.9674538373947144 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8201436400413513 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.18238699436187744 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.24271400272846222 :"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.49851807951927185 particle/accuracy_1_of_2‚1 particle/accuracy_2_of_2‚1 $ _timestamp‚1.784300451734941e+09 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.20733647048473358 jet/accuracy_2_of_5‚1 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9992193579673767 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.48179030418395996 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.1928388774394989 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.13166667520999908 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.07407407462596893 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9951876401901245 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7509849071502686 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.2640378475189209 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8009259104728699 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.3246384561061859 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2624887228012085 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4814814329147339 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.16383172571659088 8"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.706550657749176 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.2548500895500183 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.625 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.8125 (loss/val_total_loss‚8.05444750902674 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.5834145545959473 *jet/accuracy_1_of_3‚0.9103512763977051 particle/accuracy_1_of_5‚1 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.5327484607696533 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.5255484580993652 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 6validation_average_jet_accuracy‚0.3224329948425293 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.07692307978868484 trainer/global_step‚6399 6 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.670245885848999 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.06551012396812439 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.3461988866329193 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.5140268206596375 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 +particle/sensitivity‚0.6393116116523743 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9615384340286255 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 jet/accuracy_3_of_5‚1 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.2640869617462158 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.10826073586940765 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8755301237106323 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.2744404673576355 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.6127637028694153 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9880425333976746 8"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.462698757648468 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9961053133010864 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 6 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.819652259349823 particle/accuracy_2_of_5‚1 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6256846785545349 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.42211341857910156 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9904571771621704 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9992193579673767 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5533333420753479 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6546615362167358 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.4807790517807007 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9629628658294678 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9205806255340576 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.028260767459869385 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.4148986339569092 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7449358105659485 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.3171585500240326 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.24282948672771454 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.16383172571659088 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2758815288543701 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6990988254547119 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.03389977663755417 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8936848640441895 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.28271907567977905 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.5254412293434143 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 *jet/accuracy_1_of_1‚0.2745746970176697 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7142857313156128 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.04074816033244133 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.026222066953778267 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.4353065490722656 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4669686257839203 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 *jet/accuracy_1_of_2‚0.7342504858970642 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.2916666865348816 :"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.44372880458831787 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.3550509512424469 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6302083730697632 epoch‚15 _runtime‚10635.95271367 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.3198401629924774 particle/accuracy_1_of_3‚1 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9960087537765503 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.23964795470237732 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.2640378475189209 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2786148488521576 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 +jet/accuracy_3_of_3‚0.07413200289011002 +jet/accuracy_4_of_4‚0.01670061983168125 *particle/specificity‚0.902052640914917 particle/accuracy_4_of_5‚1 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.24683184921741486 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272@Å j=øäz 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9993734955787659 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9890425801277161 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2838568687438965 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7792965769767761 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.6094048023223877 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 :"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.48818162083625793 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 :"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.45126041769981384 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8199924230575562 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.6144219040870667 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8009259104728699 *jet/accuracy_2_of_4‚0.7553048133850098 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.07407407462596893 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.41203704476356506 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4140625 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 particle/accuracy_3_of_5‚1 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.4849902093410492 & Purity/0FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4619373679161072 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.1838648021221161 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 8"Purity/1FRt0SRqqt2FBt/event_purity‚0.819652259349823 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5096545219421387 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.19043691456317902 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.050279539078474045 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9780850410461426 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_2_of_3‚1 particle/accuracy_3_of_3‚1 /particle/accuracy_4_of_4‚0.9692972302436829 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.2568216621875763 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5750319361686707 particle/accuracy_1_of_4‚1 particle/accuracy_2_of_4‚1 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5757237672805786 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6033943891525269 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9966116547584534 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.26230376958847046 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.5797989368438721 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8683283925056458 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 *jet/accuracy_2_of_3‚0.5011165142059326 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9964489936828613 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.32165414094924927 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2846483290195465 /particle/accuracy_3_of_4‚0.9968254566192627 (particle/accuracy‚0.8456218242645264 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6619914174079895 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3207266926765442 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4764881730079651 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 particle/accuracy_1_of_1‚1 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.11039062589406967 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.4264788031578064 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.5311021208763123 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9966704249382019 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.27296918630599976 5 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.98738032579422 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.19741328060626984 'particle/f_score‚0.6401387453079224 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8279668092727661 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9965357184410095 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 8"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.609519898891449 *jet/accuracy_1_of_4‚0.9674538373947144 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8201436400413513 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.18238699436187744 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.24271400272846222 :"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.49851807951927185 particle/accuracy_1_of_2‚1 particle/accuracy_2_of_2‚1 $ _timestamp‚1.784300451734941e+09 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.20733647048473358 jet/accuracy_2_of_5‚1 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9992193579673767 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.48179030418395996 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.1928388774394989 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.13166667520999908 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.07407407462596893 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9951876401901245 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7509849071502686 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.2640378475189209 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8009259104728699 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.3246384561061859 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2624887228012085 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4814814329147339 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.16383172571659088 8"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.706550657749176 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.2548500895500183 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.625 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.8125 (loss/val_total_loss‚8.05444750902674 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.5834145545959473 *jet/accuracy_1_of_3‚0.9103512763977051 particle/accuracy_1_of_5‚1 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.5327484607696533 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.5255484580993652 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 6validation_average_jet_accuracy‚0.3224329948425293 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.07692307978868484 trainer/global_step‚6399 6 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.670245885848999 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.06551012396812439 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.3461988866329193 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.5140268206596375 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 +particle/sensitivity‚0.6393116116523743 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9615384340286255 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 jet/accuracy_3_of_5‚1 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.2640869617462158 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.10826073586940765 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8755301237106323 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.2744404673576355 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.6127637028694153 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9880425333976746 8"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.462698757648468 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9961053133010864 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 6 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.819652259349823 particle/accuracy_2_of_5‚1 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6256846785545349 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.42211341857910156 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9904571771621704 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9992193579673767 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5533333420753479 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6546615362167358 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.4807790517807007 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9629628658294678 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9205806255340576 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.028260767459869385 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.4148986339569092 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7449358105659485 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.3171585500240326 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.24282948672771454 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.16383172571659088 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2758815288543701 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6990988254547119 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.03389977663755417 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8936848640441895 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.28271907567977905 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.5254412293434143 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 *jet/accuracy_1_of_1‚0.2745746970176697 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7142857313156128 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.04074816033244133 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.026222066953778267 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.4353065490722656 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4669686257839203 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 *jet/accuracy_1_of_2‚0.7342504858970642 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.2916666865348816 :"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.44372880458831787 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.3550509512424469 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6302083730697632 epoch‚15 _runtime‚10635.95271367 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.3198401629924774 particle/accuracy_1_of_3‚1 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9960087537765503 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.23964795470237732 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.2640378475189209 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2786148488521576 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 +jet/accuracy_3_of_3‚0.07413200289011002 +jet/accuracy_4_of_4‚0.01670061983168125 *particle/specificity‚0.902052640914917 particle/accuracy_4_of_5‚1 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.24683184921741486 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.8571428656578064 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9994766712188721 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 *jet/accuracy_2_of_2‚0.1269209384918213 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 8"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.706550657749176 +jet/accuracy_3_of_4‚0.09748660027980804 particle/accuracy_5_of_5‚0 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6691478490829468 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 jet/accuracy_4_of_5‚0 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7475432753562927 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9996031522750854 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 _step‚527 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.24707907438278198 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.07692307978868484 jet/accuracy_1_of_5‚1 jet/accuracy_5_of_5‚0 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9368979334831238 6 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.329354465007782 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.4148986339569092 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.5217329859733582 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.2741747200489044 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2747748792171478 & Purity/*FRt1SRqqt1FBt/FBt_purity‚1 8"Purity/0FRt*SRqqt0FBt/event_purity‚0.397561252117157 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5819577574729919¹¢EþŸù‚(:” ±‡éÒªÑäö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚285.446' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚57.089200000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0®V±ú‚(:ƒ ±‡éÒ£¬Žù disk.drbd2165.out‚0 disk.sda.out‚ 536.79296875 disk.rbd0.in‚0 disk.sda3.in‚0' disk.nvme0n1.out‚105145.81396484375' proc.memory.availableMB‚ 974.9921875 cpu‚25.544420416467343 disk.sda2.in‚0$ disk./.usageGB‚550.2832069396973 disk.dm-1.out‚ 536.79296875) proc.memory.percent‚17.96562671661377 disk.rbd1.out‚ 26.19921875" disk.nvme0n1.in‚22825.85546875 proc.cpu.threads‚17 disk.dm-0.in‚22825.85546875 network.recv‚ 2.657361e+06* disk./.usagePercent‚36.929967736087285 disk.sda.in‚0 disk.sda2.out‚0$ memory_percent‚94.04911994934082# proc.memory.rssMB‚ 2943.48828125 disk.sda1.in‚0 disk.sda3.out‚ 536.79296875 disk.drbd2165.in‚0 disk.rbd3.out‚0) disk.nvme0c0n1.out‚105145.81396484375$ disk.dm-0.out‚105145.81396484375 disk.dm-1.in‚0 disk.rbd0.out‚0 disk.rbd3.in‚0$ disk.nvme0c0n1.in‚22825.85546875 disk.rbd1.in‚ 0.01171875 disk.sda1.out‚0 network.sent‚ 1.2217015e+07k&«º û‚(:• À‡éÒ½Ýúö gpu.0.gpu‚100.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚216.544' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚43.308800000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ê¢ZKü‚(:ú À‡éÒÓ¥ø# disk.nvme0c0n1.in‚ 22998.0859375' disk.nvme0c0n1.out‚105158.865234375 disk.sda2.out‚0 disk.rbd0.out‚0 disk.dm-0.in‚ 22998.0859375 disk.sda1.out‚0$ memory_percent‚94.04919147491455 disk.sda.out‚ 537.04296875 disk.dm-1.out‚ 537.04296875# proc.memory.rssMB‚ 2943.488¥¬Âº28125 disk.sda.in‚0 disk.dm-1.in‚0( proc.memory.availableMB‚ 974.98046875 disk.rbd0.in‚0 disk.sda1.in‚0 disk.sda3.out‚ 537.04296875 network.recv‚ 2.661746e+06 disk.drbd2165.in‚0 disk.rbd3.out‚0% disk.nvme0n1.out‚105158.865234375 disk.rbd1.in‚ 0.01171875 disk.sda2.in‚0" disk.dm-0.out‚105158.865234375 disk.sda3.in‚0 disk.rbd1.out‚ 26.19921875 proc.cpu.threads‚17 disk.drbd2165.out‚0$ disk./.usageGB‚550.2899208068848 disk.rbd3.in‚0! disk.nvme0n1.in‚ 22998.0859375 cpu‚25.544221861623942 network.sent‚ 1.227458e+07* disk./.usagePercent‚36.930418309347544) proc.memory.percent‚17.96562671661377ªû'?”ý‚(:‰ χéÒƒääø gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚285.779' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚57.1558 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0à_–þ‚(:ö χéÒŽé§û disk.dm-1.in‚0 disk.sda3.out‚ 537.40234375 disk.rbd1.in‚ 0.01171875 disk.dm-1.out‚ 537.40234375 disk.sda1.in‚0 disk.sda1.out‚0 cpu‚25.54395796157182 disk.sda2.out‚0$ proc.memory.availableMB‚974.8125 proc.cpu.threads‚17* disk./.usagePercent‚36.930828689424935$ disk./.usageGB‚550.2960357666016 disk.drbd2165.in‚0& disk.nvme0c0n1.out‚105166.71484375 disk.rbd0.in‚0 network.sent‚ 1.2279665e+07" disk.nvme0n1.in‚23448.50390625$ disk.nvme0n1.out‚105166.71484375 disk.rbd1.out‚ 26.32421875 disk.sda.out‚ 537.40234375 disk.rbd3.out‚0 disk.sda2.in‚0 disk.rbd0.out‚0 disk.dm-0.in‚23448.50390625! disk.dm-0.out‚105166.71484375$ memory_percent‚94.05021667480469# proc.memory.rssMB‚ 2943.48828125$ disk.nvme0c0n1.in‚23448.50390625 network.recv‚ 2.665078e+06 disk.sda3.in‚0 disk.drbd2165.out‚0 disk.rbd3.in‚0 disk.sda.in‚0) proc.memory.percent‚17.96562671661377Fÿîì”ÿ‚(:‰ Þ‡éÒúÛÒö gpu.0.gpu‚99.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚257.521' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚51.5042 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0l1§ €‚(:ÿ Þ‡éÒûĹø cpu‚25.54377074221895 disk.dm-1.out‚ 537.5859375$ disk.dm-0.out‚105181.30322265625$ disk.nvme0c0n1.in‚23448.72265625 network.recv‚ 2.667978e+06 disk.sda2.in‚0 disk.rbd0.in‚0 disk.drbd2165.out‚0) disk.nvme0c0n1.out‚105181.30322265625' disk.nvme0n1.out‚105181.30322265625 disk.rbd1.out‚ 26.32421875 disk.sda.in‚0 disk.sda.out‚ 537.5859375 disk.dm-0.in‚23448.72265625 disk.drbd2165.in‚0 disk.rbd3.out‚0$ memory_percent‚94.05019283294678 disk.sda1.out‚0 network.sent‚ 1.2284155e+07$ disk./.usageGB‚550.3004341125488 disk.rbd1.in‚ 0.01171875 disk.sda3.in‚0( proc.memory.availableMB‚ 974.81640625 disk.rbd3.in‚0" disk.nvme0n1.in‚23448.72265625 disk.dm-1.in‚0 disk.sda1.in‚0 disk.sda3.out‚ 537.5859375 proc.cpu.threads‚17 disk.sda2.out‚0 disk.rbd0.out‚0# proc.memory.rssMB‚ 2943.48828125) proc.memory.percent‚17.96562671661377) disk./.usagePercent‚36.93112386611191ÆGéž‚(:“ í‡éÒ£Ëñ÷ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚329.225' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚65.84500000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ûœd ‚‚(:‚ í‡éÒÞˆçø" disk.nvme0n1.in‚23448.92578125# proc.memory.rssMB‚ 2943.48828125 disk.rbd0.out‚0 disk.rbd3.in‚0 network.sent‚ 1.2289845e+07$ disk.dm-0.out‚105189.00244140625 disk.sda.in‚0 cpu‚25.54356802360919 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.rbd0.in‚0* disk./.usagePercent‚36.931576999447515 disk.sda3.out‚ 538.09765625 disk.drbd2165.out‚0) disk.nvme0c0n1.out‚105189.00244140625 disk.sda2.out‚0( proc.memory.availableMB‚ 974.55859375 disk.sda3.in‚0' disk.nvme0n1.out‚105189.00244140625$ memory_percent‚94.05176639556885 network.recv‚ 2.671551e+06# disk./.usageGB‚550.307186126709 disk.sda1.in‚0$ disk.nvme0c0n1.in‚23448.92578125 disk.sda2.in‚0 proc.cpu.threads‚17 disk.dm-0.in‚23448.92578125 disk.rbd3.out‚0 disk.rbd1.in‚ 0.01171875 disk.rbd1.out‚ 26.40234375 disk.dm-1.in‚0 disk.dm-1.out‚ 538.09765625) proc.memory.percent‚17.96562671661377 disk.sda.out‚ 538.09765625ÅPÖ4ƒ/ _wandbruntime‚10713  _runtime‚10713Ç–6È„Â & lr-AdamW/pg2‚0.0005415988460133483 trainer/global_step‚6449 % _timestamp‚1.7843005288421514e+09 _runtime‚10713.059327605 _step‚528 & lr-AdamW/pg1‚0.0005415988460133483µ6ëÍ…Ç & lr-AdamW/pg1‚0.0005415988460133483 & lr-AdamW/pg2‚0.0005415988460133483 trainer/global_step‚6449 % _timestamp‚1.7843005288421514e+09 _runtime‚10713.059327605 _step‚528˜añÉ4†/ _wandbruntime‚10713  _runtime‚107135d¥(‡þO 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1854439888248832 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚508574942007808 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚55862055 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6068877375488 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1853638525988864 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 110907216 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1854439914422272 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚240 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1859697019062657 _runtime‚10713.067973941 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚55862055 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚500748876845056 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7826097602048 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚245 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _step‚529 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚55872028 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 400530432 % _timestamp‚1.7843005288432262e+09 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1854439914422272 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚16549242 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6067531985920 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚32439296 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1853638500560896 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1860508791797760 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚34866581 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚500749244936192 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1860508791797760 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18317584 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚16549237 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6068877375488 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚34866826 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1860507420234752 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1854439888248832 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1860507420234752 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 110917195 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚55045161 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6067531985920 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6057148473471 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚55045161 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736Ê;+A (ˆƒP ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1853638500560896 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1860508791797760 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚34866581 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚500749244936192 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1860508791797760 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18317584 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚16549237 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6068877375488 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚34866826 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1860507420234752 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1854439888248832 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1860507420234752 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 110917195 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚55045161 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6067531985920 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6057148473471 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚55045161 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1854439888248832 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚508574942007808 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚55862055 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6068877375488 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1853638525988864 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 110907216 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1854439914422272 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚240 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1859697019062657 _runtime‚10713.067973941 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚55862055 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚500748876845056 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7826097602048 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚245 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _step‚529 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚55872028 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 400530432 % _timestamp‚1.7843005288432262e+09 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1854439914422272 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚16549242 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6067531985920 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚32439296 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0‘„!Â4‰/ _wandbruntime‚10714  _runtime‚10714lî1ÁŠˆN YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6069802418176 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚500826363252736 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚55880520 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1860786346000384 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 110923900 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 _step‚530 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1859974454278789 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1854717915152384 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6058071939717 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚55053367 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1860787717570560 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 110933893 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚231 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 _runtime‚10714.453246551 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 400523264 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6068457021440 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚55053373 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6068457021440 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚55880520 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1854717915152384 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7843005302359035e+09 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1854717888978944 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚34871870 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6069802418176 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1854717888978944 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1859975824307191 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7827271813632 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 110933893 QDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18320090 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚500826731343872 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6059416540151 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚55870533 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚508654035589632 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7827304245760 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚16551785 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1853916382339072 trainer/global_step‚6449 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1860787717570560 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚55870533 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1853916407767040 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1£ßš'‹N M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18320090 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚500826731343872 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6059416540151 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚55870533 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚508654035589632 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7827304245760 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚16551785 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1853916382339072 trainer/global_step‚6449 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1860787717570560 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚55870533 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1853916407767040 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6069802418176 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚500826363252736 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚55880520 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1860786346000384 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 110923900 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 _step‚530 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1859974454278789 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1854717915152384 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6058071939717 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚55053367 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1860787717570560 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 110933893 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚231 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 _runtime‚10714.453246551 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 400523264 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6068457021440 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚55053373 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6068457021440 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚55880520 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1854717915152384 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7843005302359035e+09 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1854717888978944 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚34871870 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6069802418176 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1854717888978944 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1859975824307191 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7827271813632 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 110933893 QDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FWÉŽ /loss/FRt2/assignment_loss‚0.566682358919564 0loss/FRt1/detection_loss‚0.49296861230349137 trainer/global_step‚6449 _step‚531 1loss/SRqqt1/detection_loss‚0.8639440346442695 /loss/FBt2/detection_loss‚0.5611146491911908 epoch‚16 _runtime‚10714.539727386 2loss/SRqqt2/assignment_loss‚0.4473072374454622 3loss/FBt1/assignment_loss‚4.560821079370011e-05 3loss/FBt2/assignment_loss‚0.0009057872161469184 /loss/FRt2/detection_loss‚1.5675132790158703 /loss/FBt1/detection_loss‚0.6236022916755506 0loss/FRt1/assignment_loss‚1.5098319797707884 2loss/SRqqt1/assignment_loss‚0.4563261154012837 1loss/SRqqt2/detection_loss‚0.5675697176550226 &loss/total_loss‚7.6578116714494335 % _timestamp‚1.7843005303228028e+09“8=¡’‚(:‡ ü‡éÒ­Ùçö gpu.0.gpu‚96.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚276.31' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.262 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0â< ý‚(:ò ü‡éÒü÷°ù disk.rbd1.out‚ 26.40234375 disk.sda3.out‚538.25 disk.drbd2165.in‚0$ disk.nvme0c0n1.in‚23449.34765625) disk.nvme0c0n1.out‚105199.76806640625# proc.memory.rssMB‚ 2766.87109375 disk.rbd0.in‚0 disk.sda1.out‚0 disk.drbd2165.out‚0' disk.nvme0n1.out‚105199.76806640625 network.sent‚ 1.2294882e+07$ disk./.usageGB‚550.3011322021484# memory_percent‚93.0680513381958) disk./.usagePercent‚36.93117071549067 disk.sda1.in‚0 disk.sda3.in‚0" disk.nvme0n1.in‚23449.34765625 network.recv‚ 2.674906e+06 disk.dm-0.in‚23449.34765625 disk.rbd3.in‚0 disk.rbd1.in‚ 0.01171875 disk.dm-1.in‚0) proc.memory.availableMB‚ 1135.73046875 cpu‚25.543341617896306 disk.sda.out‚538.25 disk.dm-1.out‚538.25 disk.rbd3.out‚0 disk.sda2.in‚0 disk.sda2.out‚0* proc.memory.percent‚16.887640953063965 proc.cpu.threads‚17 disk.sda.in‚0 disk.rbd0.out‚0$ disk.dm-0.out‚105199.76806640625Aw©”‘‚(:‰ ‹ˆéÒÉô€÷ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚37.0 gpu.0.powerWatts‚110.656' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.1312 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¿·¦"’‚(:÷ ‹ˆéÒé°§ù disk.sda.in‚0 disk.sda.out‚ 538.5703125 disk.rbd3.out‚0% disk.nvme0n1.out‚105212.623046875 proc.cpu.threads‚17 network.recv‚ 2.67932e+06) disk./.usagePercent‚36.93203269285846 disk.rbd0.out‚0 disk.rbd3.in‚0 disk.sda2.out‚0 cpu‚25.54307245688236 disk.dm-1.out‚ 538.5703125 disk.rbd0.in‚0 disk.sda1.out‚0 disk.sda2.in‚0# proc.memory.rssMB‚ 2767.43359375 disk.dm-0.in‚ 23977.4765625& proc.memory.availableMB‚ 1134.96875 disk.dm-1.in‚0 disk.sda3.in‚0 disk.drbd2165.in‚0* proc.memory.percent‚16.891074180603027$ disk./.usageGB‚550.3139762878418' disk.nvme0c0n1.out‚105212.623046875 disk.rbd1.in‚ 0.03173828125 network.sent‚ 1.2357927e+07" disk.dm-0.out‚105212.623046875 disk.sda1.in‚0 disk.sda3.out‚ 538.5703125# disk.nvme0c0n1.in‚ 23977.4765625 disk.rbd1.out‚ 26.66015625$ memory_percent‚93.07270050048828 disk.drbd2165.out‚0! disk.nvme0n1.in‚ 23977.4765625Pøž“‚(:“ šˆéÒ˜Óàö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚258.057' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚51.61140000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0—¤Õ”‚(:ù šˆéÒèºóø disk.sda2.in‚0 disk.sda.in‚0 disk.drbd2165.out‚0) proc.memory.availableMB‚ 1134.72265625 network.recv‚ 2.682726e+06 disk.dm-0.in‚ 24088.359375 disk.sda3.out‚ 538.828125 disk.drbd2165.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 24088.359375& disk.nvme0n1.out‚105222.5888671875) disk./.usagePercent‚36.93252013120365$ disk./.usageGB‚550.3212394714355 disk.dm-1.out‚ 538.828125 disk.sda1.in‚0 disk.sda2.out‚0 disk.rbd1.out‚ 26.66015625# disk.dm-0.out‚105222.5888671875 disk.rbd3.out‚0( disk.nvme0c0n1.out‚105222.5888671875 disk.rbd1.in‚ 0.03173828125 cpu‚25.542898559552274 proc.cpu.threads‚17 disk.dm-1.in‚0# proc.memory.rssMB‚ 2767.43359375 network.sent‚ 1.2362982e+07 disk.nvme0n1.in‚ 24088.359375 disk.sda.out‚ 538.828125$ memory_percent‚93.07420253753662* proc.memory.percent‚16.891074180603027 disk.sda1.out‚0 disk.sda3.in‚0 disk.rbd3.in‚0P¸²ž•‚(:“ ©ˆéÒʹêö gpu.0.gpu‚79.0 gpu.0.memory‚24, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚41.0 gpu.0.powerWatts‚217.897' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚43.57939999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0© }+–‚(:ù ©ˆéÒ–ç€ù disk.dm-0.in‚ 24088.78125 disk.sda1.in‚0 disk.sda1.out‚0# proc.memory.rssMB‚ 2767.43359375 proc.cpu.threads‚17 network.recv‚ 2.68614e+06 disk.rbd1.in‚ 0.03173828125 disk.sda.in‚0# disk.dm-0.out‚105230.6474609375) proc.memory.availableMB‚ 1135.08203125 disk.sda2.out‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 539.1484375 disk.sda3.out‚ 539.1484375 disk.drbd2165.in‚0! disk.nvme0c0n1.in‚ 24088.78125 disk.sda.out‚ 539.1484375 disk.rbd3.in‚0 disk.nvme0n1.in‚ 24088.78125) disk./.usagePercent‚36.93299348913447 disk.rbd1.out‚ 26.8134765625 disk.drbd2165.out‚0 cpu‚25.542676280127868 network.sent‚ 1.236801e+07 disk.rbd0.in‚0$ disk./.usageGB‚550.3282928466797( disk.nvme0c0n1.out‚105230.6474609375$ memory_percent‚93.07200908660889 disk.sda3.in‚0 disk.rbd3.out‚0& disk.nvme0n1.out‚105230.6474609375* proc.memory.percent‚16.891074180603027 disk.sda2.in‚0 disk.rbd0.out‚0¡ªi;ž—‚(:“ ¸ˆéÒÎØÖö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚281.143' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚56.22859999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¸Iá¸í˜‚(:â ¸ˆéÒÿÅù disk.drbd2165.out‚0 disk.rbd3.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd1.out‚ 26.8134765625 disk.rbd3.out‚0' disk.nvme0c0n1.out‚105244.978515625) disk./.usagePercent‚36.93328431369336 disk.dm-1.in‚0 disk.rbd0.out‚0 disk.rbd1.in‚ 0.03173828125% disk.nvme0n1.out‚105244.978515625$ disk./.usageGB‚550.3326263427734" disk.dm-0.out‚105244.978515625 disk.drbd2165.in‚0 disk.nvme0c0n1.in‚24089) proc.memory.availableMB‚ 1135.07421875# proc.memory.rssMB‚ 2767.43359375 proc.cpu.threads‚17 network.sent‚ 1.2373246e+07 network.recv‚ 2.689374e+06 disk.sda.out‚ 539.328125 disk.nvme0n1.in‚24089* proc.memory.percent‚16.891074180603027 disk.dm-1.out‚ 539.328125 disk.sda1.in‚0 disk.dm-0.in‚24089 cpu‚25.542432016116084 disk.sda.in‚0 disk.sda1.out‚0 disk.sda3.in‚0 disk.sda3.out‚ 539.328125# memory_percent‚93.0720567703247 disk.rbd0.in‚0Ýÿ<[4™/ _wandbruntime‚10791  _runtime‚10791n^Ț trainer/global_step‚6499 % _timestamp‚1.7843006076358678e+09 _runtime‚10791.853044373 _step‚532 & lr-AdamW/pg1‚0.0005393064122615446 & lr-AdamW/pg2‚0.0005393064122615446´[XGÍ›Ç & lr-AdamW/pg1‚0.0005393064122615446 & lr-AdamW/pg2‚0.0005393064122615446 trainer/global_step‚6499 % _timestamp‚1.7843006076358678e+09 _runtime‚10791.853044373 _step‚532”€Î4œ/ _wandbruntime‚10791  _runtime‚10791|ÓÿÂÎ&þO A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚35130635 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚35130903 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 111753247 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚55455467 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1868339950927872 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6113847916544 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7886212738048 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚55455461 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚512509778058240 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚512509375430656 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 111743268 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6113847916544 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1873637345084565 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6103385586579 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚16676392 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚16676387 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18454511 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6104730186901 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1874453772670976 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1867532614897664 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6115193306112 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚56297780 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1873635975056275 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 % _timestamp‚1.7843006076369662e+09 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚56287807 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚55455467 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚504623565320192 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚504623197229056 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1874453772670976 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚55455461 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1868339924754432 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1874455144233984 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1868339950927872 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚263 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1868339924754432 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6115193306112 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7886178201600 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1867532589469696 trainer/global_step‚6499 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚268 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1874455144233984 KBDevyíþ?6iceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚18454248 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872^Qt (žƒP G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1874455144233984 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1868339950927872 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚263 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1868339924754432 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6115193306112 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7886178201600 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1867532589469696 trainer/global_step‚6499 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚268 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1874455144233984 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚18454248 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚35130635 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚35130903 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 111753247 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚55455467 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1868339950927872 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6113847916544 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7886212738048 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚55455461 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚512509778058240 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚512509375430656 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 111743268 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6113847916544 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1873637345084565 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6103385586579 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚16676392 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚16676387 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18454511 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6104730186901 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1874453772670976 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1867532614897664 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6115193306112 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚56297780 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1873635975056275 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 % _timestamp‚1.7843006076369662e+09 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚56287807 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚55455467 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚504623565320192 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚504623197229056 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1874453772670976 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚55455461 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1868339924754432 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448•%¼ã€4Ÿ/ _wandbruntime‚10793  _runtime‚10793^Üz® ' ‡N REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7887379306496 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 111769945 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚35135933 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚55463667 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚16678930 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 _step‚534 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6116118348800 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18457256 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 111759952 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚55463673 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚504701051727872 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1873914780300697 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1874734070006784 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚56306272 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚504700683636736 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6104309052825 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚56296285 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚56296285 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1867810496675840 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 trainer/global_step‚6499 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚258 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 111769945 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 111759952 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1868617925484544 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚56306272 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1868617925484544 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1874732698436608 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚35136191 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6114772952064 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6116118348800 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚55463667 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚512588062943232 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7887413835776 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚55463673 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚253 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18457003 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚16678935 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6105653653259 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1874734070006784 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648Ñò0'¡ŒN _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚504700683636736 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6104309052825 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚56296285 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚56296285 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1867810496675840 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 trainer/global_step‚6499 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚258 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 111769945 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 111759952 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1868617925484544 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚56306272 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1868617925484544 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1874732698436608 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚35136191 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6114772952064 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6116118348800 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚55463667 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚512588062943232 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7887413835776 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚55463673 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚253 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18457003 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚16678935 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6105653653259 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1874734070006784 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7887379306496 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 111769945 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚35135933 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚55463667 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚16678930 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 _step‚534 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6116118348800 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18457256 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 111759952 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚55463673 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚504701051727872 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1873914780300697 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1874734070006784 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚56306272–¨ š 4¢/ _wandbruntime‚10793  _runtime‚10793ò­4¥ £„ 2loss/FBt1/assignment_loss‚0.030550803078878336 2loss/FBt2/assignment_loss‚0.014442563360763038 /loss/FRt1/detection_loss‚0.5345912998642038 1loss/SRqqt1/detection_loss‚1.0110046953028335 /loss/FRt2/detection_loss‚1.5421328725750316 0loss/FBt2/detection_loss‚0.49638719157912103 trainer/global_step‚6499 $ _timestamp‚1.784300609110024e+09 _runtime‚10793.326952421 0loss/FRt1/assignment_loss‚1.5289633651495453 2loss/SRqqt1/assignment_loss‚0.5571380620193742 0loss/SRqqt2/detection_loss‚0.613873916590222 _step‚535 /loss/FRt2/assignment_loss‚0.686401067784539 /loss/FBt1/detection_loss‚0.8141253181349706 epoch‚16 %loss/total_loss‚8.304930422571758 3loss/SRqqt2/assignment_loss‚0.47531926713227496ÔâÈѤ‰ 0loss/FRt1/assignment_loss‚1.5289633651495453 2loss/SRqqt1/assignment_loss‚0.5571380620193742 0loss/SRqqt2/detection_loss‚0.613873916590222 _step‚535 /loss/FRt2/assignment_loss‚0.686401067784539 /loss/FBt1/detection_loss‚0.8141253181349706 epoch‚16 %loss/total_loss‚8.304930422571758 3loss/SRqqt2/assignment_loss‚0.47531926713227496 2loss/FBt1/assignment_loss‚0.030550803078878336 2loss/FBt2/assignment_loss‚0.014442563360763038 /loss/FRt1/detection_loss‚0.5345912998642038 1loss/SRqqt1/detection_loss‚1.0110046953028335 /loss/FRt2/detection_loss‚1.5421328725750316 0loss/FBt2/detection_loss‚0.49638719157912103 trainer/global_step‚6499 $ _timestamp‚1.784300609110024e+09 _runtime‚10793.326952421—2¾Ç1”¥‚(:‰ LjéÒþÚ“ø gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚332.266' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚66.4532 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0±ƒ5G¦‚(:÷ Ljé҆تù* proc.memory.percent‚16.8910741806030274¡D¢» disk.rbd3.in‚0 disk.sda2.in‚0 disk.rbd1.in‚ 0.03173828125 disk.rbd0.in‚0 disk.sda3.out‚ 539.5859375) proc.memory.availableMB‚ 1134.01171875 disk.drbd2165.out‚0 disk.nvme0n1.in‚ 24089.203125 disk.sda2.out‚0 disk.dm-1.out‚ 539.5859375 disk.dm-0.in‚ 24089.203125! disk.dm-0.out‚105252.33203125( disk./.usagePercent‚36.9337272067276# disk./.usageGB‚550.339225769043 disk.sda.in‚0 disk.dm-1.in‚0$ disk.nvme0n1.out‚105252.33203125 disk.rbd1.out‚27.07958984375 disk.sda1.out‚0 disk.sda3.in‚0 proc.cpu.threads‚17 disk.rbd0.out‚0 network.sent‚ 1.2382021e+07 disk.drbd2165.in‚0& disk.nvme0c0n1.out‚105252.33203125 disk.sda.out‚ 539.5859375$ memory_percent‚93.07854175567627# proc.memory.rssMB‚ 2767.43359375 cpu‚25.542233031585486 network.recv‚ 2.700404e+06 disk.rbd3.out‚0" disk.nvme0c0n1.in‚ 24089.203125 disk.sda1.in‚0§sqi“§‚(:ˆ ÖˆéÒ ûÉö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚245.915' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚49.183 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0µœLr¨‚(:‡ ÖˆéÒÚ°…ú network.recv‚ 2.705117e+06! disk.nvme0n1.in‚ 24627.7109375) disk./.usagePercent‚36.93420926891457$ disk./.usageGB‚550.3464088439941$ memory_percent‚93.07606220245361$ disk.dm-0.out‚105263.34619140625 disk.drbd2165.in‚0 disk.drbd2165.out‚0 cpu‚25.54200961832261 disk.sda1.in‚0 disk.sda3.in‚0 disk.rbd3.out‚0) disk.nvme0c0n1.out‚105263.34619140625 disk.rbd1.out‚27.07958984375 disk.rbd0.out‚0 disk.dm-0.in‚ 24627.7109375 disk.sda1.out‚0# disk.nvme0c0n1.in‚ 24627.7109375' disk.nvme0n1.out‚105263.34619140625 disk.sda2.in‚0 disk.sda2.out‚0) proc.memory.availableMB‚ 1134.41796875 disk.rbd0.in‚0 disk.sda3.out‚ 539.72265625 proc.cpu.threads‚17 network.sent‚ 1.2444489e+07 disk.dm-1.out‚ 539.72265625 disk.rbd3.in‚0 disk.rbd1.in‚ 0.03173828125# proc.memory.rssMB‚ 2767.43359375 disk.sda.in‚0 disk.sda.out‚ 539.72265625 disk.dm-1.in‚0* proc.memory.percent‚16.891074180603027@&‘šž©‚(:“ åˆéÒ½¥ãö gpu.0.gpu‚92.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚330.288' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚66.05760000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚55865761 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1882239961260032 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6150967300009 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1888401496670208 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚516444019486720 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚16803537 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18592247 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚56723532 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚516443618956288 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1882239987433472 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 400530432 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚508497885704192 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1888400125107200 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6160163847168 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1888401496670208 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚32439296 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1881426678378496 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6161509236736 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 112589299 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚16279øÛ(´P M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚35395536 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 $ _timestamp‚1.784300686320285e+09 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚55865761 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1882239961260032 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6150967300009 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1888401496670208 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚516444019486720 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚16803537 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18592247 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚56723532 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚516443618956288 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1882239987433472 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 400530432 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚508497885704192 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1888400125107200 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6160163847168 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1888401496670208 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚32439296 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1881426678378496 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6161509236736 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 112589299 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚56713559 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚55865767 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6161509236736 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7946101343232 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1882239987433472 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6™ƒpâ!4µ/ _wandbruntime‚10871  _runtime‚10871?ÂﶇN REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚16806085 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6162434279424 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 112596004 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚6549 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚35400826 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚56722037 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1881704585584640 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6161088882688 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚242 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚10871.923466029 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6161088882688 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚16806080 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1888680422443008 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 $ _timestamp‚1.784300687705972e+09 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6162434279424 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚56732024 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 400523264 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1887856476351007 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1888680422443008 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚56722037 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚35401073 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18594988 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 112605997 FDeviceStatsMonitor.on_train_batch_end/reqM.Ú uested_bytes.all.peak‚ 47499471119 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1881704560156672 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚55873973 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚55873967 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 _step‚538 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1888679050872832 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚55873973 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6151890766367 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 112605997 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1882517961990144 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 112596004 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18594746 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1882517988163584 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1882517961990144 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1887855106322605 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚55873967 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚508575004020736 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820>ÜgÞ'·ŒN B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚516522710457344 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚508575372111872 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚516522309934080 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7947305913344 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6150546165933 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚55873973 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚55873967 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 _step‚538 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1888679050872832 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚55873973 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6151890766367 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 112605997 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1882517961990144 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 112596004 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18594746 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1882517988163584 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1882517961990144 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1887855106322605 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚55873967 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚508575004020736 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚16806085 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6162434279424 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 112596004 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚6549 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚35400826 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚56722037 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1881704585584640 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6161088882688 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚242 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚10871.923466029 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6161088882688 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚16806080 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1888680422443008 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 $ _timestamp‚1.784300687705972e+09 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6162434279424 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚56732024 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 400523264 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1887856476351007 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1888680422443008 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚56722037 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚35401073 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18594988 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 112605997 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1881704560156672 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 B\¾‚(:† ¡‰éÒ Ýõú disk.sda2.out‚0 disk.rbd1.in‚ 0.47314453125 disk.rbd0.in‚0# disk.dm-0.out‚105347.1416015625 network.recv‚ 2.73025e+06 disk.dm-1.in‚0 disk.sda1.in‚0 disk.rbd3.in‚0 network.sent‚ 1.2531205e+07" disk.nvme0n1.in‚24801.26171875 disk.drbd2165.out‚0) proc.memory.availableMB‚ 1135.41015625) disk./.usagePercent‚36.93630289852959$ memory_percent‚93.07000637054443 cpu‚25.541029367275716$ disk.nvme0c0n1.in‚24801.26171875& disk.nvme0n1.out‚105347.1416015625( disk.nvme0c0n1.out‚105347.1416015625 disk.rbd1.out‚ 28.0224609375 disk.sda.in‚0 disk.sda1.out‚0* proc.memory.percent‚16.891074180603027$ disk./.usageGB‚550.3776054382324 disk.sda2.in‚0 disk.dm-1.out‚ 547.70703125 disk.dm-0.in‚24801.26171875 disk.drbd2165.in‚0 disk.rbd3.out‚0# proc.memory.rssMB‚ 2767.43359375 disk.sda.out‚ 547.70703125 disk.rbd0.out‚0 disk.sda3.in‚0 disk.sda3.out‚ 547.70703125 proc.cpu.threads‚17ãìͬ“¿‚(:ˆ °‰éÒ¡êù gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚282.495' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚56.499 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0yÌx¿À‚(:… °‰éÒéÈìû* disk./.usagePercent‚36.936924996837796( disk.nvme0c0n1.out‚105359.7705078125" disk.nvme0n1.in‚24801.64453125 disk.dm-0.in‚24801.64453125 disk.sda3.out‚ 548.08984375& proc.memory.availableMB‚ 1135.40625# proc.memory.rssMB‚ 2767.43359375* proc.memory.percent‚16.891074180603027 network.recv‚ 2.733295e+06 disk.sda.in‚0 disk.rbd0.in‚0 proc.cpu.threads‚17 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd1.in‚ 0.47314453125# disk.dm-0.out‚105359.7705078125 disk.drbd2165.out‚0$ disk.nvme0c0n1.in‚24801.64453125 disk.sda.out‚ 548.08984375 disk.rbd0.out‚0 disk.sda3.in‚0 disk.rbd3.in‚0$ disk./.usageGB‚550.3868751525879 disk.rbd3.out‚0 disk.drbd2165.in‚0& disk.nvme0n1.out‚105359.7705078125 disk.dm-1.in‚0 disk.sda1.in‚0 disk.sda1.out‚0$ memory_percent‚93.07003021240234 network.sent‚ 1.2535698e+07 disk.rbd1.out‚ 28.0224609375 disk.dm-1.out‚ 548.08984375 cpu‚25.540785491663964þoÖ¤”Á‚(:‰ ¿‰éÒ¹‚úö gpu.0.gpu‚99.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚170.686' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.1372 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ä­× ‚(:… ¿‰éҷͲù disk.sda.out‚ 549.58984375 disk.dm-0.in‚ 24801.8359375' disk.nvme0n1.out‚105378.68896484375$ memory_percent‚93.07150840759277 disk.drbd2165.in‚0 disk.rbd3.out‚0 disk.sda2.out‚0( proc.memory.availableMB‚ 1135.1640625 cpu‚25.54058166241276 network.sent‚ 1.2540767e+07 disk.rbd0.in‚0 disk.sda3.out‚ 549.58984375 disk.rbd3.in‚0 disk.rbd1.in‚ 0.47314453125* proc.memory.percent‚16.891074180603027) disk./.usagePercent‚36.93697927043505$ disk./.usageGB‚550.3876838684082 disk.sda1.out‚0) disk.nvme0c0n1.out‚105378.68896484375! disk.nvme0n1.in‚ 24801.8359375 disk.sda2.in‚0 disk.sda.in‚0$ disk.dm-0.out‚105378.68896484375# disk.nvme0c0n1.in‚ 24801.8359375 network.recv‚ 2.736715e+06 disk.dm-1.in‚0 disk.rbd1.out‚ 28.8388671875# proc.memory.rssMB‚ 2767.43359375 proc.cpu.threads‚17 disk.dm-1.out‚ 549.58984375 disk.rbd0.out‚0 disk.sda1.in‚0 disk.sda3.in‚0 disk.drbd2165.out‚0ÍòVžÃ‚(:“ ΉéÒÑÆ…÷ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚279.79' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚55.958000000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0±NÄ‚(:ý ΉéÒ³¤éù) disk.nvme0c0n1.out‚105393.58740234375 disk.rbd1.out‚ 28.8505859375( proc.memory.availableMB‚ 1135.3828125 cpu‚25.54036430448853 network.sent‚ 1.2546324e+07 network.recv‚ 2.739892e+06 disk.rbd0.out‚0 disk.dm-0.in‚24905.55078125# memory_percent‚93.0701732635498# proc.memory.rssMB‚ 2767.43359375 disk.drbd2165.out‚0 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.sda2.out‚0$ disk.dm-0.out‚105393.58740234375 disk.sda1.in‚0 disk.sda1.out‚0$ disk.nvme0c0n1.in‚24905.55078125" disk.nvme0n1.in‚24905.55078125 disk.dm-1.in‚0 disk.sda3.in‚0) disk./.usagePercent‚36.93730286395833 disk.rbd0.in‚0* proc.memory.percent‚16.891074180603027 disk.sda2.in‚0 disk.rbd1.in‚ 0.47314453125 disk.sda.out‚ 550.84375 disk.dm-1.out‚ 550.84375 disk.drbd2165.in‚0# disk./.usageGB‚550.392505645752' disk.nvme0n1.out‚105393.58740234375 disk.sda.in‚0 disk.sda3.out‚ 550.84375 proc.cpu.threads‚17FŠãë4Å/ _wandbruntime‚10949  _runtime‚10949lääÈÆ trainer/global_step‚6599 % _timestamp‚1.7843007651199048e+09 _runtime‚10949.337033866 _step‚540 & lr-AdamW/pg1‚0.0005346844310650344 & lr-AdamW/pg2‚0.0005346844310650344RåoÍÇÇ & lr-AdamW/pg1‚0.0005346844310650344 & lr-AdamW/pg2‚0.0005346844310650344 trainer/global_step‚6599 % _timestamp‚1.7843007651199048e+09 _runtime‚10949.337033866 _step‚540œÆ8†X4È/ _wandbruntime‚10949  _runtime‚10949Ug(ÙÉþO ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6207825167360 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6206479777792 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8006289586176 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1901517997128381 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1895320767287296 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6207825167360 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1896139997765632 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚512371837997056 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚56276067 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚16930692 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1902346477543424 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1902347849106432 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18729024 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 trainer/global_step‚6599 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚260 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RBDeviceStatsMonito¢³0"+r.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1896139997765632 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚57139311 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _step‚541 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚520378495674368 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚57139311 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1902346477543424 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1902347849106432 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18729024 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 trainer/global_step‚6599 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚260 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1896139997765632 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚57139311 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _step‚541 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚520378495674368 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚57139311 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1902346477543424 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1902346477543424 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JžË‚(:“ ݉éÒò¬êö gpu.0.gpu‚90.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚269.087' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚53.81739999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ñªƒÌ‚(:‰ ݉éÒ©Îëø disk.sda2.out‚0 network.sent‚ 1.2551361e+07 disk.sda3.in‚0 disk.sda3.out‚ 551.89453125 disk.drbd2165.out‚0 disk.rbd3.in‚0" disk.nvme0n1.in‚24905.97265625* proc.memory.percent‚16.891074180603027# proc.memory.rssMB‚ 2767.43359375) disk.nvme0c0n1.out‚105403.42919921875 disk.rbd1.in‚ 0.47314453125( proc.memory.availableMB‚ 1135.1328125) disk./.usagePercent‚36.93783894373504 disk.drbd2165.in‚0 disk.rbd1.out‚ 28.9287109375 disk.sda.in‚0 network.recv‚ 2.743339e+06$ disk.dm-0.out‚105403.42919921875 disk.sda1.in‚0 disk.sda.out‚ 551.89453125$ memory_percent‚93.07169914245605 proc.cpu.threads‚17$ disk./.usageGB‚550.4004936218262 disk.rbd0.in‚0 disk.dm-0.in‚24905.97265625 disk.rbd3.out‚0' disk.nvme0n1.out‚105403.42919921875 cpu‚25.540154466285863 disk.rbd0.out‚0 disk.sda1.out‚0 disk.sda2.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 551.89453125$ disk.nvme0c0n1.in‚24905.97265625VnfÁ4Í/ _wandbruntime‚10950  _runtime‚109506·4·'ΈN \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1896418024669184 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚57147789 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 113432056 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚56284267 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6196783279041 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6208750210048 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1902626774879232 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚16933230 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1896417998495744 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚56284273 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1901796802372915 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚56284267 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚35664751 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚512449692495872 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18731767 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6198127879475 % _timestamp‚1.7843007665063183e+09 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚57147789 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 113442049 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1895598649065472 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1902626774879232 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚6599 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚35665002 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚16933235 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 _runtime‚10950.723676571 _step‚542 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1902625403309056 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚520457177781248 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6207404813312 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8007485285376 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 113442049 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚57157776 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8007450756096 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚246 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚56284273 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6207404813312 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1901795432344513 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1895598674493440 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚251 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402¨ÄjÛ§"ÏN XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1901796802372915 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚56284267 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚35664751 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚512449692495872 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18731767 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6198127879475 % _timestamp‚1.7843007665063183e+09 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚57147789 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 113442049 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1895598649065472 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1902626774879232 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚6599 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚35665002 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚16933235 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 _runtime‚10950.723676571 _step‚542 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1902625403309056 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚520457177781248 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6207404813312 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8007485285376 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 113442049 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚57157776 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8007450756096 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚246 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚56284273 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6207404813312 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1901795432344513 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1895598674493440 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚251 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1896418024669184 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚57147789 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 113432056 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚56284267 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.÷õÞòlfreed‚ 6196783279041 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6208750210048 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1902626774879232 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚16933230 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1896417998495744 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚56284273 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0ž-Ý÷g4Ð/ _wandbruntime‚10950  _runtime‚10950§·ÁY Ñ… 1loss/FBt2/assignment_loss‚0.02705348973734783 0loss/FRt1/detection_loss‚0.47319631255162375 1loss/SRqqt1/detection_loss‚0.9074124211435064 trainer/global_step‚6599 _runtime‚10950.810274628 epoch‚16 0loss/FRt2/assignment_loss‚0.7606214603820405 3loss/SRqqt2/assignment_loss‚0.46810497380410515 3loss/FBt1/assignment_loss‚0.0027443913419347155 1loss/SRqqt2/detection_loss‚0.6493318946615341 /loss/FBt1/detection_loss‚0.5718383676100832 %loss/total_loss‚7.860144079575263 2loss/SRqqt1/assignment_loss‚0.5001543625254562 /loss/FBt2/detection_loss‚0.4822491842675053 $ _timestamp‚1.784300766593359e+09 _step‚543 0loss/FRt1/assignment_loss‚1.5323684018722084 .loss/FRt2/detection_loss‚1.485068819677917!JÜ#ÒŠ 1loss/FBt2/assignment_loss‚0.02705348973734783 0loss/FRt1/detection_loss‚0.47319631255162375 1loss/SRqqt1/detection_loss‚0.9074124211435064 trainer/global_step‚6599 _runtime‚10950.810274628 epoch‚16 0loss/FRt2/assignment_loss‚0.7606214603820405 3loss/SRqqt2/assignment_loss‚0.46810497380410515 3loss/FBt1/assignment_loss‚0.0027443913419347155 1loss/SRqqt2/detection_loss‚0.6493318946615341 /loss/FBt1/detection_loss‚0.5718383676100832 %loss/total_loss‚7.860144079575263 2loss/SRqqt1/assignment_loss‚0.5001543625254562 /loss/FBt2/detection_loss‚0.4822491842675053 $ _timestamp‚1.784300766593359e+09 _step‚543 0loss/FRt1/assignment_loss‚1.5323684018722084 .loss/FRt2/detection_loss‚1.485068819677917Ÿ '‘’Ó‚(:‡ ì‰éÒü°ïö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚260.99' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.198 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0îc—Ô‚(:û ì‰éÒœô¾ù$ disk./.usageGB‚550.4094276428223 disk.sda.in‚0$ disk.dm-0.out‚105430.65966796875 disk.sda1.in‚0$ disk.nvme0c0n1.in‚24906.19140625" disk.nvme0n1.in‚24906.19140625# proc.memory.rssMB‚ 2767.43359375 disk.rbd0.out‚0 disk.sda1.out‚0) disk.nvme0c0n1.out‚105430.65966796875' disk.nvme0n1.out‚105430.65966796875 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd1.out‚ 28.9287109375 disk.dm-1.in‚0 disk.sda3.out‚554.1875 disk.drbd2165.out‚0 disk.rbd3.in‚0 disk.rbd3.out‚0 cpu‚25.53993718633826 disk.rbd0.in‚0 disk.sda3.in‚0$ memory_percent‚93.07107925415039' proc.memory.availableMB‚ 1135.234375) disk./.usagePercent‚36.93843851338022 disk.rbd1.in‚ 0.47314453125 disk.dm-0.in‚24906.19140625* proc.memory.percent‚16.891074180603027 disk.sda.out‚554.1875 disk.dm-1.out‚554.1875 proc.cpu.threads‚17 network.sent‚ 1.2603648e+07 network.recv‚ 2.747696e+06 disk.drbd2165.in‚0<6žÕ‚(:“ û‰éÒøæñö gpu.0.gpu‚83.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚332.025' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚66.40499999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ŽFËÖ‚(:ü û‰éÒü¡ù disk.sda3.out‚ 554.90625 disk.sda.in‚0 disk.rbd0.in‚0 disk.sda1.in‚0* proc.memory.percent‚16.891074180603027 disk.drbd2165.out‚0 network.recv‚ 2.75182e+06$ disk.nvme0c0n1.in‚24906.39453125 disk.rbd1.in‚ 0.47314453125 disk.dm-1.in‚0 disk.sda1.out‚0) proc.memory.availableMB‚ 1134.79296875 cpu‚25.53979299172251 network.sent‚ 1.2655955e+07$ disk./.usageGB‚552.5163650512695& disk.nvme0n1.out‚107590.5166015625# disk.dm-0.out‚107590.5166015625# proc.memory.rssMB‚ 2767.43359375 disk.sda3.in‚0 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.dm-0.in‚24906.39453125" disk.nvme0n1.in‚24906.39453125 disk.rbd1.out‚ 29.0224609375 disk.dm-1.out‚ 554.90625 disk.rbd0.out‚0) disk./.usagePercent‚37.07983685069935 disk.drbd2165.in‚0$ memory_percent‚93.07377338409424 proc.cpu.threads‚17( disk.nvme0c0n1.out‚107590.5166015625 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda.out‚ 554.90625çÁ9”ׂ(:‰ ŠŠéÒçÖÿö gpu.0.gpu‚99.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚266.886' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.3772 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0° 9« Ø‚(:‚ ŠŠéÒ¢ÈÄù disk.rbd0.in‚0( proc.memory.availableMB‚ 1134.7578125 proc.cpu.threads‚17 disk.rbd1.in‚ 0.47314453125 disk.sda.out‚ 555.140625( disk./.usagePercent‚37.0802917760877 disk.sda1.out‚0 disk.sda3.out‚ 555.140625 disk.rbd3.out‚0) disk.nvme0c0n1.out‚107602.35986328125' disk.nvme0n1.out‚107602.35986328125# proc.memory.rssMB‚ 2767.43359375 cpu‚25.539601342213576 disk.sda1.in‚0$ disk.nvme0c0n1.in‚24912.59765625 disk.sda.in‚0 disk.dm-1.out‚ 555.140625 network.recv‚ 2.755325e+06 disk.dm-0.in‚24912.59765625 disk.drbd2165.in‚0 disk.rbd3.in‚0 disk.rbd1.out‚ 29.0224609375 network.sent‚ 1.2661023e+07 disk.sda2.out‚0 disk.rbd0.out‚0$ memory_percent‚93.07398796081543* proc.memory.percent‚16.891074180603027" disk.nvme0n1.in‚24912.59765625$ disk.dm-0.out‚107602.35986328125 disk.drbd2165.out‚0 disk.dm-1.in‚0$ disk./.usageGB‚552.5231437683105 disk.sda3.in‚0 disk.sda2.in‚0îÈŸÙ‚(:” ™ŠéÒ­—Ú÷ gpu.0.gpu‚98.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚274.743' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚54.948600000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0"ܽºÚ‚(:… ™ŠéÒðÒ­ø) disk./.usagePercent‚37.08090107401919 disk.dm-0.in‚ 25122.0859375 disk.drbd2165.out‚0 disk.rbd3.out‚0 disk.sda.in‚0 disk.rbd0.out‚0 proc.cpu.threads‚17! disk.nvme0n1.in‚ 25122.0859375 disk.sda2.in‚0 disk.rbd1.in‚ 0.47314453125$ disk.dm-0.out‚107631.93017578125 disk.sda1.out‚0$ disk./.usageGB‚552.5322227478027' disk.nvme0n1.out‚107631.93017578125 disk.sda3.in‚0 disk.sda2.out‚0 disk.sda.out‚ 557.37109375# memory_percent‚93.0753231048584( proc.memory.availableMB‚ 1134.5390625* proc.memory.percent‚16.891074180603027 network.sent‚ 1.2666142e+07 network.recv‚ 2.758839e+06 disk.dm-1.in‚0# proc.memory.rssMB‚ 2767.43359375 disk.sda3.out‚ 557.37109375 disk.drbd2165.in‚0 disk.rbd3.in‚0# disk.nvme0c0n1.in‚ 25122.0859375) disk.nvme0c0n1.out‚107631.93017578125 disk.dm-1.out‚ 557.37109375 disk.rbd0.in‚0 disk.sda1.in‚0 disk.rbd1.out‚ 29.1162109375 cpu‚25.539394627063235é„ú§žÛ‚(:“ ¨ŠéÒ¾»Ž÷ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚282.255' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚56.45099999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0œ“œÜ‚(:„ ¨ŠéÒ¹ž™ù disk.drbd2165.in‚0( disk.nvme0c0n1.out‚107643.8212890625! disk.nvme0n1.in‚ 25122.2890625 disk.dm-1.out‚ 557.76171875 proc.cpu.threads‚17 disk.sda3.in‚0 disk.rbd1.out‚ 29.1162109375 disk.rbd0.in‚0 disk.sda3.out‚ 557.76171875 disk.drbd2165.out‚0 disk.rbd3.in‚0 disk.sda2.in‚0 disk.sda.in‚0) proc.memory.availableMB‚ 1134.60546875 disk.dm-0.in‚ 25122.2890625 disk.sda1.out‚0 disk.sda2.out‚0$ disk./.usageGB‚550.4384384155273 disk.rbd3.out‚0 disk.sda.out‚ 557.76171875$ memory_percent‚93.07491779327393 cpu‚25.539246542577608 network.recv‚ 2.762339e+06 disk.rbd0.out‚0# disk.dm-0.out‚107643.8212890625# proc.memory.rssMB‚ 2767.43359375) disk./.usagePercent‚36.94038545067812# disk.nvme0c0n1.in‚ 25122.2890625& disk.nvme0n1.out‚107643.8212890625 disk.rbd1.in‚ 0.47314453125 disk.dm-1.in‚0 disk.sda1.in‚0* proc.memory.percent‚16.891074180603027 network.sent‚ 1.2671177e+07Ye|4Ý/ _wandbruntime‚11029  _runtime‚11029¹..²ÆÞÀ _runtime‚11029.785191306 _step‚544 % lr-AdamW/pg1‚0.000532355180481543 % lr-AdamW/pg2‚0.000532355180481543 trainer/global_step‚6649 % _timestamp‚1.7843008455680604e+09e.ñËßÅ % lr-AdamW/pg1‚0.000532355180481543 % lr-AdamW/pg2‚0.000532355180481543 trainer/global_step‚6649 % _timestamp‚1.7843008455680604e+09 _runtime‚11029.785191306 _step‚544 ÔÕ4à/ _wandbruntime‚11029  _runtime‚11029bd%(áþO TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚35922760 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8066144214016 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1909214856196096 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1910040060444672 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 % _timestamp‚1.7843008455691009e+09 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1916294201542656 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚18864923 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 _runtime‚11029.786424143 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚56686361 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚17057837 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6242096925903 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1910040034271232 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚252 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚545 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚516246158381056 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6254141097984 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6252795708416 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚257 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1916294201542656 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1910040034271232 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚57565063 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18865175 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚17057842 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚57565063 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1910040060444672 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6252795708416 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1909214881624064 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚57575036 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1916292829979648 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8066178750464 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1916292829979648 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚35923017 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 trainer/global_step‚6649 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1915458323150289 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 114261403 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6243441526225 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0çBó (âƒP D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18865175 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚17057842 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚57565063 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1910040060444672 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6252795708416 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1909214881624064 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚57575036 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1916292829979648 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8066178750464 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1916292829979648 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚35923017 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 trainer/global_step‚6649 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1915458323150289 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 114261403 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6243441526225 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚35922760 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8066144214016 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1909214856196096 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1910040060444672 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 % _timestamp‚1.7843008455691009e+09 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1916294201542656 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚18864923 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 _runtime‚11029.786424143 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚56686361 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚17057837 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6242096925903 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1910040034271232 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚252 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚545 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚516246158381056 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6254141097984 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6252795708416 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚257 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1916294201542656 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1910040034271232 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚57565063 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚524390985348096 ;4DeviceStatsMonp*V0× itor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6253720743936 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 114268108 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8067375088640 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚57583528 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1910318035001344 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1916573127315456 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _runtime‚11031.172114168 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1909492737974272 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚56694567 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚17060385 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚56694573 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6243020392149 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚35927936 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 114278101 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚524391387968512 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚245 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1910318035001344 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1916571755745280 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚35928186 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 114268108 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8067340559360 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 % _timestamp‚1.7843008469547372e+09 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚250 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚56694567 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ´G‡'åN TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚56694573 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6243020392149 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚35927936 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 114278101 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚524391387968512 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚245 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1910318035001344 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1916571755745280 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚35928186 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 114268108 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8067340559360 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 % _timestamp‚1.7843008469547372e+09 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚250 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚56694567 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 QDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚524390985348096 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6253720743936 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 114268108 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8067375088640 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚57583528 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1910318035001344 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1916573127315456 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _runtime‚11031.172114168 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1909492737974272 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚56694567 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚17060385¢¸" é4æ/ _wandbruntime‚11031  _runtime‚11031íoÅ ç† 3loss/FBt1/assignment_loss‚0.0012603500480807054 1loss/FBt2/assignment_loss‚0.03145437140359288 /loss/FRt1/detection_loss‚0.5177953553373204 1loss/SRqqt2/detection_loss‚0.6464900262182743 %loss/total_loss‚7.293866924573683 $ _timestamp‚1.784300847041601e+09 0loss/FRt2/assignment_loss‚0.5925953621449678 3loss/SRqqt1/assignment_loss‚0.46671107796114575 3loss/SRqqt2/assignment_loss‚0.43204389072818145 0loss/SRqqt1/detection_loss‚0.800681772815376 0loss/FRt1/assignment_loss‚1.5313333105543308 /loss/FRt2/detection_loss‚1.3736003767865737 0loss/FBt2/detection_loss‚0.44658028827263757 _step‚547 /loss/FBt1/detection_loss‚0.4533207423032023 epoch‚16 trainer/global_step‚6649 _runtime‚11031.258524077Û…& è‹ 3loss/FBt1/assignment_loss‚0.0012603500480807054 1loss/FBt2/assignment_loss‚0.03145437140359288 /loss/FRt1/detection_loss‚0.5177953553373204 1loss/SRqqt2/detection_loss‚0.6464900262182743 %loss/total_loss‚7.293866924573683 $ _timestamp‚1.784300847041601e+09 0loss/FRt2/assignment_loss‚0.5925953621449678 3loss/SRqqt1/assignment_loss‚0.46671107796114575 3loss/SRqqt2/assignment_loss‚0.43204389072818145 0loss/SRqqt1/detection_loss‚0.800681772815376 0loss/FRt1/assignment_loss‚1.5313333105543308 /loss/FRt2/detection_loss‚1.3736003767865737 0loss/FBt2/detection_loss‚0.44658028827263757 _step‚547 /loss/FBt1/detection_loss‚0.4533207423032023 epoch‚16 trainer/global_step‚6649 _runtime‚11031.258524077£wsÈžé‚(:“ ·ŠéÒÝ¥èö gpu.0.gpu‚90.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚273.684' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚54.73680000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ßvD¨ ê‚(:ÿ ·ŠéÒï¿Úø disk.dm-1.out‚ 560.0546875 disk.sda1.out‚0# proc.memory.rssMB‚ 2767.43359375 disk.sda2.out‚0 disk.sda1.in‚0$ disk./.usageGB‚550.4425048828125 disk.sda.out‚ 560.0546875 disk.dm-1.in‚0 disk.sda3.out‚ 560.0546875 disk.drbd2165.in‚0 disk.sda3.in‚0) disk.nvme0c0n1.out‚107661.61376953125 disk.rbd1.out‚ 29.1787109375$ memory_percent‚93.07670593261719 disk.rbd0.in‚0 disk.rbd3.out‚0 disk.rbd1.in‚ 0.47314453125 disk.rbd0.out‚0 disk.dm-0.in‚ 25122.7109375 disk.rbd3.in‚0! disk.nvme0n1.in‚ 25122.7109375) disk./.usagePercent‚36.94065835470962$ disk.dm-0.out‚107661.61376953125 cpu‚25.53899233640479 network.recv‚ 2.765965e+06% proc.memory.availableMB‚ 1134.3125* proc.memory.percent‚16.891074180603027 proc.cpu.threads‚17 network.sent‚ 1.2676252e+07 disk.sda.in‚0 disk.drbd2165.out‚0# disk.nvme0c0n1.in‚ 25122.7109375' disk.nvme0n1.out‚107661.61376953125 disk.sda2.in‚0 {ëžë‚(:“ ÆŠéÒ­ÄÙö gpu.0.gpu‚96.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚305.112' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚61.02240000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0(ò'ì‚(:ƒ ÆŠéÒù¿Ôù disk.dm-1.in‚0$ disk.dm-0.out‚107672.57080078125 disk.drbd2165.out‚0$ memory_percent‚93.07675361633301 disk.rbd0.out‚0 disk.sda1.in‚0 disk.sda1.out‚0$ disk./.usageGB‚550.4391899108887 disk.rbd0.in‚0 disk.rbd3.out‚0* proc.memory.percent‚16.891074180603027 disk.sda2.out‚0 disk.sda.out‚ 560.63671875 disk.dm-1.out‚ 560.63671875 disk.sda3.in‚0 disk.sda.in‚0 disk.dm-0.in‚ 25122.9296875) disk./.usagePercent‚36.94043588416236# disk.nvme0c0n1.in‚ 25122.9296875 disk.sda2.in‚0 disk.sda3.out‚ 560.63671875# proc.memory.rssMB‚ 2767.43359375 cpu‚25.5387996068768 network.sent‚ 1.2739356e+07) disk.nvme0c0n1.out‚107672.57080078125! disk.nvme0n1.in‚ 25122.9296875' disk.nvme0n1.out‚107672.57080078125 disk.rbd1.out‚ 29.1787109375 disk.drbd2165.in‚0 disk.rbd3.in‚0 network.recv‚ 2.77075e+06 disk.rbd1.in‚ 0.47314453125( proc.memory.availableMB‚ 1134.3046875 proc.cpu.threads‚17 ¨lØ”í‚(:‰ ÕŠéÒ§¥óö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚271.718' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.3436 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÿÁñ˜ î‚(: ÕŠéÒÒ¦£ø disk.sda.out‚ 561.3671875 disk.dm-1.out‚ 561.3671875 disk.sda1.in‚0 disk.sda2.out‚0$ memory_percent‚93.07675361633301 disk.sda1.out‚0 disk.drbd2165.in‚0# proc.memory.rssMB‚ 2767.43359375* proc.memory.percent‚16.891074180603027 disk.sda3.in‚0 disk.sda3.out‚ 561.3671875 disk.drbd2165.out‚0! disk.nvme0n1.in‚ 25212.5703125 proc.cpu.threads‚17# disk./.usageGB‚550.444709777832 disk.sda.in‚0$ disk.dm-0.out‚107683.92724609375) disk.nvme0c0n1.out‚107683.92724609375 disk.rbd0.in‚0 disk.rbd3.in‚0' disk.nvme0n1.out‚107683.92724609375 network.sent‚ 1.2744427e+07 network.recv‚ 2.77415e+06 disk.dm-0.in‚ 25212.5703125 disk.rbd3.out‚0 disk.rbd1.in‚ 0.47314453125 disk.rbd1.out‚ 29.2568359375( proc.memory.availableMB‚ 1134.3046875 cpu‚25.53859597612703* disk./.usagePercent‚36.940806327064415 disk.dm-1.in‚0 disk.rbd0.out‚0# disk.nvme0c0n1.in‚ 25212.5703125 disk.sda2.in‚0}o$‡Ÿï‚(:” äŠéÒê´‹÷ gpu.0.gpu‚54.0 gpu.0.memory‚16, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚38.0 gpu.0.powerWatts‚110.415' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.083000000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0B“ïð‚(:‰ äŠéÒŒ„¹ù disk.dm-1.out‚ 563.31640625 disk.rbd3.in‚0 proc.cpu.threads‚17 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda3.in‚0 disk.sda3.out‚ 563.31640625 disk.drbd2165.out‚0 disk.sda2.in‚0 disk.rbd0.in‚0* proc.memory.percent‚16.891074180603027 network.sent‚ 1.2749451e+07" disk.nvme0n1.in‚25212.98046875$ memory_percent‚93.07684898376465 cpu‚25.538380693214446 disk.rbd0.out‚0) disk./.usagePercent‚36.94777920427452 disk.dm-0.in‚25212.98046875$ disk.nvme0c0n1.in‚25212.98046875 disk.rbd1.in‚ 0.47314453125 disk.rbd3.out‚0) disk.nvme0c0n1.out‚107708.90087890625' disk.nvme0n1.out‚107708.90087890625 disk.drbd2165.in‚0 network.recv‚ 2.777322e+06$ disk./.usageGB‚550.5486106872559 disk.sda2.out‚0 disk.rbd1.out‚ 29.2568359375 disk.sda.in‚0$ disk.dm-0.out‚107708.90087890625 disk.sda.out‚ 563.31640625( proc.memory.availableMB‚ 1134.2890625# proc.memory.rssMB‚ 2767.43359375 disk.dm-1.in‚0“<º”ñ‚(:‰ óŠéÒÃáòö gpu.0.gpu‚91.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚343.177' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚68.6354 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Xš ¯ ò‚(:€ óŠéÒöüŠù$ memory_percent‚93.07687282562256 disk.sda.in‚0 disk.dm-1.in‚0 disk.dm-0.in‚25213.19921875 disk.sda3.out‚ 564.171875 disk.drbd2165.out‚0$ disk.nvme0c0n1.in‚25213.19921875 disk.sda2.out‚0# proc.memory.rssMB‚ 2767.43359375) disk./.usagePercent‚36.94863682951422 disk.dm-1.out‚ 564.171875 disk.rbd3.in‚0 disk.rbd3.out‚0* proc.memory.percent‚16.891074180603027 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.drbd2165.in‚0) proc.memory.availableMB‚ 1134.28515625 disk.sda2.in‚0 network.recv‚ 2.780689e+06$ disk.dm-0.out‚107717.03955078125 disk.sda1.in‚0 disk.sda3.in‚0) disk.nvme0c0n1.out‚107717.03955078125" disk.nvme0n1.in‚25213.19921875 disk.sda.out‚ 564.171875 disk.rbd1.in‚ 0.47314453125 cpu‚25.53817333609904 disk.sda1.out‚0' disk.nvme0n1.out‚107717.03955078125 disk.rbd1.out‚ 29.41796875 network.sent‚ 1.275448e+07$ disk./.usageGB‚550.5613899230957:í 4ó/ _wandbruntime‚11109  _runtime‚11109¹ÅÈô & lr-AdamW/pg2‚0.0005300139557862164 trainer/global_step‚6699 % _timestamp‚1.7843009248169918e+09 _runtime‚11109.034192036 _step‚548 & lr-AdamW/pg1‚0.0005300139557862164Nq›òÍõÇ trainer/global_step‚6699 % _timestamp‚1.7843009248169918e+09 _runtime‚11109.034192036 _step‚548 & lr-AdamW/pg1‚0.0005300139557862164 & lr-AdamW/pg2‚0.0005300139557862164¤gÜ3_4ö/ _wandbruntime‚11109  _runtime‚11109@÷FÕö÷þO \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 115087476 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚57096667 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1930239182415872 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚520120478765056 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚17184987 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚36186997 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 % _timestamp‚1.7843009248181503e+09 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1923108945104896 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1930240553978880 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚58000788 [IDeviceStatsMonitor.on8~_train_batch_start/active_bytes.small_pool.allocated‚ 6300457028608 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8126107492352 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1923940096950272 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚520120846856192 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6300457028608 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6299111639040 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1930239182415872 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8126142028800 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6289678639333 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚57096661 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚57096667 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1923940070776832 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1923940070776832 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1923108945104896 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1930240553978880 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚58000788 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6300457028608 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8126107492352 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1923940096950272 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚520120846856192 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6300457028608 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6299111639040 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1930239182415872 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8126142028800 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6289678639333 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚57096661 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚57096667 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872¥Mèn4ù/ _wandbruntime‚11110  _runtime‚11110N†ž.'úˆN _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚520197965172736 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1923386852311040 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6300036674560 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚246 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚57999293 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1929676084388329 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 trainer/global_step‚6699 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6301382071296 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚17187530 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 % _timestamp‚1.7843009262175531e+09 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚58009280 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19004749 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1930519479751680 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚36192525 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 _step‚550 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚57104873 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 115114153 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1924218097680384 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚241 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1924218097680384 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 115114153 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚36192279 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8127308785664 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 _runtime‚11110.435116905 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚58009280 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚520198333263872 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 115104160 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0UXq‡„ûN WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚528325273958400 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1924218071506944 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8127343314944 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 115114153 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1924218097680384 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚241 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1924218097680384 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 115114153 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚36192279 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8127308785664 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 _runtime‚11110.435116905 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚58009280 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚520198333263872 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 115104160 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚520197965172736 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1923386852311040 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6300036674560 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚246 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚57999293 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1929676084388329 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 trainer/global_step‚6699 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6301382071296 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚17187530 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 % _timestamp‚1.7843009262175531e+09 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚58009280 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19004749 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1930519479751680 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚36192525 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 _step‚550 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚57104873 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 FDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1937003059441664 _step‚553 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1944185534852096 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1937840107282432 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6345427569664 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚19138818 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1937840133455872 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚252 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚523994799149056 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1944185534852096 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 115933507 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚532180919011328 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚57506961 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627LñkJ'Ž‚P ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6346772959232 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1937003034013696 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 115923528 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚58426540 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚57506967 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19139070 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚523995167240192 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1937840133455872 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1937840107282432 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚36451212 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚257 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1944186906415104 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚36450955 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1943337605165815 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚57506967 $ _timestamp‚1.784301005527523e+09 trainer/global_step‚6749 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6346772959232 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚17312137 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6335915752441 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1937003059441664 _step‚553 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1944185534852096 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1937840107282432 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6345427569664 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚19138818 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1937840133455872 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚252 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚523994799149056 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1944185534852096 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 115933507 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚532180919011328 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚57506961 ?9DeviceStatsMonitor.on_train_batch_start/num_!>œ¾alloc_retries‚0 JDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚58435032 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚36456074 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1943616410410237 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 115950205 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚57515173 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1944465832187904 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1938118134185984 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1944464460617728 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 % _timestamp‚1.7843010069329824e+09 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6346352605184 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚532260007301120 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6346352605184 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚17314685 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1943617780438639 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚252 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚17314680 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8187353653248 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚11191.150470679 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1944465832187904 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚58435032 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6347698001920 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19141394 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚532259604680704 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19141641 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8187319123968 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚58425045 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1937280941219840 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚57515173 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚524072285556736 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402!ÄB×'“N QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚36456074 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1943616410410237 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 115950205 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚57515173 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1944465832187904 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1938118134185984 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1944464460617728 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 % _timestamp‚1.7843010069329824e+09 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6346352605184 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚532260007301120 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6346352605184 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚17314685 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1943617780438639 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚252 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚17314680 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8187353653248 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚11191.150470679 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1944465832187904 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚58435032 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6347698001920 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19141394 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚532259604680704 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19141641 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8187319123968 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚58425045 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1937280941219840 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚57515173 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚524072285556736 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 115940212 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚57515167 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1937280915791872 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚58435032ª&X¶¼4”/ _wandbruntime‚11191  _runtime‚11191Ï&H* •… epoch‚16 trainer/global_step‚6749 _runtime‚11191.226402392 _step‚555 1loss/SRqqt2/detection_loss‚0.5954469170544502 0loss/FRt2/assignment_loss‚0.5910584208640185 2loss/SRqqt1/assignment_loss‚0.5374852533875233 2loss/FBt2/assignment_loss‚0.008257189570970035 1loss/SRqqt1/detection_loss‚0.8292375066771339 0loss/FBt2/detection_loss‚0.40326278212897737 $loss/total_loss‚7.54134801996884 % _timestamp‚1.7843010070092132e+09 /loss/FBt1/detection_loss‚0.5668114216029141 0loss/FRt1/assignment_loss‚1.4987167250172198 3loss/SRqqt2/assignment_loss‚0.49847843556753996 2loss/FBt1/assignment_loss‚0.008110198165675144 .loss/FRt1/detection_loss‚0.582474086125965 /loss/FRt2/detection_loss‚1.4220090838064525¢´³C–Š trainer/global_step‚6749 _runtime‚11191.226402392 _step‚555 1loss/SRqqt2/detection_loss‚0.5954469170544502 0loss/FRt2/assignment_loss‚0.5910584208640185 2loss/SRqqt1/assignment_loss‚0.5374852533875233 2loss/FBt2/assignment_loss‚0.008257189570970035 1loss/SRqqt1/detection_loss‚0.8292375066771339 0loss/FBt2/detection_loss‚0.40326278212897737 $loss/total_loss‚7.54134801996884 % _timestamp‚1.7843010070092132e+09 /loss/FBt1/detection_loss‚0.5668114216029141 0loss/FRt1/assignment_loss‚1.4987167250172198 3loss/SRqqt2/assignment_loss‚0.49847843556753996 2loss/FBt1/assignment_loss‚0.008110198165675144 .loss/FRt1/detection_loss‚0.582474086125965 /loss/FRt2/detection_loss‚1.4220090838064525 epoch‚16«Ùu=¯ž—‚(:“ Ü‹éÒéÜ€ø gpu.0.gpu‚89.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚264.23' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚52.846000000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0f1…͘‚(:„ Ü‹éÒÒ¿©ù# disk.nvme0c0n1.in‚ 25456.9609375 network.recv‚ 2.806208e+06 disk.dm-1.out‚ 574.6015625 disk.drbd2165.out‚0 disk.rbd0.in‚0 disk.sda1.out‚0 disk.sda3.out‚ 574.6015625 disk.sda2.in‚0 proc.cpu.threads‚17$ disk./.usageGB‚550.6291007995605 disk.rbd3.out‚0$ memory_percent‚93.07644367218018# proc.memory.rssMB‚ 2767.43359375* proc.memory.percent‚16.891074180603027 disk.rbd3.in‚0' disk.nvme0n1.out‚107949.35107421875 disk.rbd1.out‚ 29.763671875 disk.sda.in‚0 disk.sda1.in‚0 disk.drbd2165.in‚0) disk.nvme0c0n1.out‚107949.35107421875! disk.nvme0n1.in‚ 25456.9609375$ disk.dm-0.out‚107949.35107421875) proc.memory.availableMB‚ 1134.35546875 cpu‚25.536835584825784 network.sent‚ 1.2894377e+07* disk./.usagePercent‚36.953180963246986 disk.dm-1.in‚0 disk.sda2.out‚0 disk.rbd1.in‚ 0.47314453125 disk.sda.out‚ 574.6015625 disk.rbd0.out‚0 disk.dm-0.in‚ 25456.9609375 disk.sda3.in‚0 +ÌKŸ™‚(:” ë‹éÒý£áö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚283.559' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚56.711800000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0bQ”š‚(:ƒ ë‹éÒõÛØú) disk./.usagePercent‚36.95361617605519 disk.rbd0.out‚0 disk.drbd2165.in‚0! disk.nvme0n1.in‚ 25457.1640625 disk.dm-1.in‚0 disk.dm-1.out‚ 575.91015625 disk.sda1.in‚0 disk.sda3.out‚ 575.91015625$ memory_percent‚93.07641983032227 disk.rbd1.in‚ 0.47314453125 disk.rbd1.out‚ 29.826171875$ disk.dm-0.out‚107960.16748046875' disk.nvme0n1.out‚107960.16748046875 disk.sda2.in‚0 network.sent‚ 1.2946388e+07 disk.sda.out‚ 575.91015625 disk.sda3.in‚0 disk.rbd3.in‚0* proc.memory.percent‚16.891074180603027 disk.rbd0.in‚0 disk.dm-0.in‚ 25457.1640625 disk.rbd3.out‚0 disk.sda2.out‚0# proc.memory.rssMB‚ 2767.43359375 cpu‚25.53662439750704 disk.sda.in‚0 disk.drbd2165.out‚0# disk.nvme0c0n1.in‚ 25457.1640625) disk.nvme0c0n1.out‚107960.16748046875 network.recv‚ 2.810877e+06$ disk./.usageGB‚550.6355857849121 disk.sda1.out‚0 proc.cpu.threads‚17' proc.memory.availableMB‚ 1134.359375UbáÓž›‚(:“ ú‹éÒüÇ·ö gpu.0.gpu‚99.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚46.0 gpu.0.powerWatts‚339.809' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚67.96180000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ä{£Óœ‚(:„ ú‹éÒ­‰‡ù$ disk.dm-0.out‚107973.99560546875 disk.rbd3.in‚0' disk.nvme0n1.out‚107973.99560546875 disk.rbd1.in‚ 0.47314453125 disk.sda.in‚0 disk.dm-1.out‚ 576.38671875 disk.rbd0.in‚0$ memory_percent‚93.07646751403809 disk.sda1.out‚0! disk.nvme0n1.in‚ 25457.5859375( proc.memory.availableMB‚ 1134.3515625* proc.memory.percent‚16.891074180603027 cpu‚25.536450027956942 network.sent‚ 1.2951966e+07 disk.sda1.in‚0 disk.sda3.in‚0 disk.sda3.out‚ 576.38671875 disk.rbd3.out‚0 network.recv‚ 2.814382e+06 disk.drbd2165.in‚0) disk.nvme0c0n1.out‚107973.99560546875# disk./.usageGB‚550.642993927002 disk.dm-0.in‚ 25457.5859375 disk.drbd2165.out‚0 disk.sda2.out‚0# proc.memory.rssMB‚ 2767.43359375 proc.cpu.threads‚17 disk.rbd0.out‚0) disk./.usagePercent‚36.95411334268669 disk.rbd1.out‚ 29.826171875 disk.dm-1.in‚0# disk.nvme0c0n1.in‚ 25457.5859375 disk.sda2.in‚0 disk.sda.out‚ 576.386718753fo“‚(:ˆ ‰ŒéÒ›¢ºö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚265.005' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ânF ž‚(:þ ‰ŒéÒ´¡‘ù disk.rbd3.out‚0 network.sent‚ 1.2957037e+07$ disk./.usageGB‚550.5535087585449 disk.sda.out‚ 576.8984375 disk.dm-1.in‚0" disk.dm-0.out‚107988.029296875 disk.sda3.in‚0 disk.sda3.out‚ 576.8984375% disk.nvme0n1.out‚107988.029296875 disk.rbd1.in‚ 0.47314453125 proc.cpu.threads‚17$ disk.nvme0c0n1.in‚25668.49609375( proc.memory.availableMB‚ 1134.6015625 disk.dm-0.in‚25668.49609375# proc.memory.rssMB‚ 2767.43359375* proc.memory.percent‚16.891074180603027 cpu‚25.5362237397279 disk.rbd3.in‚0 disk.sda2.out‚0 network.recv‚ 2.817957e+06 disk.sda1.in‚0 disk.drbd2165.out‚0 disk.sda.in‚0 disk.dm-1.out‚ 576.8984375 disk.rbd0.out‚0 disk.sda1.out‚0 disk.rbd0.in‚0 disk.drbd2165.in‚0' disk.nvme0c0n1.out‚107988.029296875" disk.nvme0n1.in‚25668.49609375* disk./.usagePercent‚36.948107917948484 disk.sda2.in‚0 disk.rbd1.out‚ 29.904296875$ memory_percent‚93.07494163513184«åIÝ”Ÿ‚(:‰ ˜ŒéÒå…Þö gpu.0.gpu‚90.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚46.0 gpu.0.powerWatts‚334.442' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚66.8884 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0'4ýô ‚(:ý ˜ŒéÒ¥·ƒù disk.sda3.in‚0 disk.drbd2165.out‚0! disk.nvme0n1.in‚ 25668.7109375 disk.drbd2165.in‚0) disk.nvme0c0n1.out‚107999.41748046875 disk.rbd0.out‚0* proc.memory.percent‚16.891074180603027 proc.cpu.threads‚17$ disk.dm-0.out‚107999.41748046875 disk.sda1.in‚0 disk.sda2.out‚0 disk.rbd1.in‚ 0.47314453125 disk.rbd0.in‚0# proc.memory.rssMB‚ 2767.43359375 disk.dm-1.out‚ 577.5546875 cpu‚25.53606534965072) disk./.usagePercent‚36.94858025184915 disk.sda3.out‚ 577.5546875 disk.rbd3.in‚0 disk.sda.out‚ 577.5546875 network.sent‚ 1.2962058e+07 disk./.usageGB‚ 550.560546875 disk.sda1.out‚0# disk.nvme0c0n1.in‚ 25668.7109375 disk.sda.in‚0 disk.dm-0.in‚ 25668.7109375 network.recv‚ 2.821187e+06 disk.rbd3.out‚0' disk.nvme0n1.out‚107999.41748046875 disk.sda2.in‚0 disk.rbd1.out‚ 29.904296875 disk.dm-1.in‚0$ memory_percent‚93.07646751403809( proc.memory.availableMB‚ 1134.3515625ÙiÒ4¡/ _wandbruntime‚11269  _runtime‚11269ÙpøȢ & lr-AdamW/pg2‚0.0005252961863094225 trainer/global_step‚6799 % _timestamp‚1.7843010854969773e+09 _runtime‚11269.714156891 _step‚556 & lr-AdamW/pg1‚0.00052529618630942253shꣂ trainer/global_step‚6799 % _timestamp‚1.7843010854969773e+09 _runtime‚11269.714156891 _step‚556 & lr-AdamW/pg1‚0.0005252961863094225 & lr-AdamW/pg2‚0.0005252961863094225¬„X"†4¤/ _wandbruntime‚11269  _runtime‚11269ÞA¥ýO VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1958131887288320 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6391743500288 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚57917267 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚36714600 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1958131887288320 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6391743500288 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 400530432 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6393088889856 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚57917261 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚527869487624192 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1957277931187723 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6380808265227 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚527869119533056 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8246021730304 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚32439296 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 _step‚557 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚536115541793792 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8246054169600 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1950897148350464 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6393088889856 $ _timestamp‚1.784301085498126e+09 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚17439287 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 116759580 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚58852292 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚249 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚58842319 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚19275059 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚36714346 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚536115141263360 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640,å|¤(¦‚P XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19275308 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1958133258851328 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1958133258851328 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1951740169961472 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6393088889856 $ _timestamp‚1.784301085498126e+09 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚17439287 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 116759580 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚58852292 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚249 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚58842319 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚19275059 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚36714346 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚536115141263360 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1958131887288320 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6391743500288 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚57917267 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚36714600 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1958131887288320 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6391743500288 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 400530432 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6393088889856 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚57917261 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚527869487624192 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1957277931187723 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6380808265227 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚527869119533056 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8246021730304 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚32439296 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 _step‚557 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚536115541793792 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8246054169600 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1950897148350464 JDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6392668535808 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 116776264 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 116786257 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 _runtime‚11271.11197524 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚57925467 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚58860784 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6392668535808 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚536193832527872 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6394013932544 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚58860784 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚36719633 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 trainer/global_step‚6799 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚57925467 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚57925473 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6394013932544 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1958412184624128 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8247259019264 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _step‚558 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19277803 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1952018144518144 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7843010868945196e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 116786257 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1958410813053952 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚527946605940736GB1­°©ŒN UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6394013932544 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚58860784 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚36719633 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 trainer/global_step‚6799 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚57925467 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚57925473 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6394013932544 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1958412184624128 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8247259019264 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _step‚558 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19277803 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1952018144518144 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7843010868945196e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 116786257 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1958410813053952 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚527946605940736 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1952018144518144 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚36719874 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚32432128 JDDeviceStatsMonitor.on_train_batch_end/oÚÓbversize_allocations.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6392668535808 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 116776264 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 116786257 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 _runtime‚11271.11197524 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚57925467 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚58860784 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6392668535808 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚536193832527872®5,o•4ª/ _wandbruntime‚11271  _runtime‚11271J¡Æ€ «ƒ 0loss/FRt1/assignment_loss‚1.5237183908832224 0loss/SRqqt1/detection_loss‚0.783181243032077 /loss/FBt2/detection_loss‚0.5002704663119603 trainer/global_step‚6799 _step‚559 0loss/FRt2/assignment_loss‚0.6905925206840038 2loss/SRqqt2/assignment_loss‚0.5294299360320687 2loss/FBt2/assignment_loss‚0.004750920189018156 /loss/FBt1/detection_loss‚0.5891441646032035 _runtime‚11271.194155561 2loss/SRqqt1/assignment_loss‚0.4451761538460357 3loss/FBt1/assignment_loss‚0.0019017672976390898 .loss/FRt2/detection_loss‚1.629010412064381 % _timestamp‚1.7843010869770145e+09 .loss/FRt1/detection_loss‚0.551667079660017 1loss/SRqqt2/detection_loss‚0.6124557740011487 %loss/total_loss‚7.861298828604775 epoch‚16Òk¬ˆ 0loss/FRt1/assignment_loss‚1.5237183908832224 0loss/SRqqt1/detection_loss‚0.783181243032077 /loss/FBt2/detection_loss‚0.5002704663119603 trainer/global_step‚6799 _step‚559 0loss/FRt2/assignment_loss‚0.6905925206840038 2loss/SRqqt2/assignment_loss‚0.5294299360320687 2loss/FBt2/assignment_loss‚0.004750920189018156 /loss/FBt1/detection_loss‚0.5891441646032035 _runtime‚11271.194155561 2loss/SRqqt1/assignment_loss‚0.4451761538460357 3loss/FBt1/assignment_loss‚0.0019017672976390898 .loss/FRt2/detection_loss‚1.629010412064381 % _timestamp‚1.7843010869770145e+09 .loss/FRt1/detection_loss‚0.551667079660017 1loss/SRqqt2/detection_loss‚0.6124557740011487 %loss/total_loss‚7.861298828604775 epoch‚16¯ »©Ô”­‚(:‰ §ŒéÒ®óŠø gpu.0.gpu‚99.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚39.0 gpu.0.powerWatts‚113.129' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.6258 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ü”‡…®‚(:ù §ŒéÒ®‡äù disk.rbd0.in‚0 disk.dm-0.in‚25669.12109375 disk.sda1.in‚0) proc.memory.percent‚17.49553680419922 disk.sda.out‚577.9375 disk.drbd2165.out‚0 disk.rbd3.out‚0" disk.nvme0n1.in‚25669.12109375 proc.memory.rssMB‚ 2866.46875 disk.rbd1.in‚ 0.47314453125 disk.sda1.out‚0$ disk./.usageGB‚550.5673141479492 disk.dm-1.out‚577.9375 disk.rbd3.in‚0) disk.nvme0c0n1.out‚108011.87744140625) proc.memory.availableMB‚ 1035.19921875 network.recv‚ 2.824758e+06 disk.sda.in‚0 disk.dm-1.in‚0 disk.drbd2165.in‚0 cpu‚25.53793821697627 disk.rbd0.out‚0$ disk.dm-0.out‚108011.87744140625' disk.nvme0n1.out‚108011.87744140625 disk.rbd1.out‚ 29.966796875 disk.sda2.out‚0 disk.sda3.in‚0$ disk.nvme0c0n1.in‚25669.12109375 disk.sda2.in‚0$ memory_percent‚93.68164539337158 proc.cpu.threads‚17 network.sent‚ 1.2967119e+07* disk./.usagePercent‚36.949034409214896 disk.sda3.out‚577.9375Dñb®”¯‚(:‰ ¶ŒéÒ”ôö gpu.0.gpu‚68.0 gpu.0.memory‚20, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚40.0 gpu.0.powerWatts‚123.364' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.6728 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0+t °‚(:€ ¶ŒéÒæ“Äú network.sent‚ 1.3029371e+07$ disk./.usageGB‚550.5719032287598 disk.dm-1.in‚0 disk.sda3.in‚0 disk.rbd3.in‚0 disk.sda2.out‚0 disk.sda.out‚ 578.12109375 network.recv‚ 2.829466e+06 disk.rbd0.in‚0 disk.dm-0.in‚25669.33984375# disk.dm-0.out‚108022.2216796875 cpu‚25.541241849005612) disk./.usagePercent‚36.94934238627859 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda3.out‚ 578.12109375 disk.drbd2165.out‚0" disk.nvme0n1.in‚25669.33984375 disk.rbd1.in‚ 0.47314453125 disk.rbd1.out‚ 29.966796875 disk.sda.in‚0* proc.memory.percent‚17.497539520263672 disk.dm-1.out‚ 578.12109375 disk.sda1.in‚0 disk.rbd3.out‚0$ disk.nvme0c0n1.in‚25669.33984375( disk.nvme0c0n1.out‚108022.2216796875% proc.memory.availableMB‚ 1034.6875 disk.rbd0.out‚0& disk.nvme0n1.out‚108022.2216796875$ memory_percent‚93.68476867675781 proc.cpu.threads‚17 disk.drbd2165.in‚0! proc.memory.rssMB‚ 2866.796875M„¨4±/ _wandbruntime‚11297  _runtime‚11297I2<¶ç=²á{ 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6794134974479675 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9889451265335083 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.2731499969959259 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.46064817905426025 /particle/accuracy_4_of_4‚0.9505739808082581 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.038885388523340225 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 6 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.260284423828125 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.3350023031234741 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.25227224826812744 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8627544641494751 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6569612622261047 *jet/accuracy_3_of_3‚0.0678483322262764 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2723780870437622 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9974919557571411 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.5732911229133606 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.24119073152542114 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 particle/accuracy_1_of_5‚1 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6529412269592285 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6538980007171631 , Purity/0FRt2SRqqt*FBt/FBt_purity‚0.96875 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 particle/accuracy_5_of_5‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6472569704055786 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7926681041717529 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.0833333358168602 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9583333134651184 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9947671294212341 /"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.421875 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8006047010421753 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 ,jet/accuracy_4_of_4‚0.013899499550461769 particle/accuracy_3_of_5‚1 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5735671520233154 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 8"Purity/*FRt*SRqqt2FBt/event_purity‚0.863677978515625 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.5907489657402039 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7399647831916809 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 8"Purity/0FRt*SRqqt0FBt/event_purity‚0.545141875743866 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 6 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.273772656917572 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 )jet/accuracy_1_of_2‚0.771665096282959 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.07692307978868484 % _timestamp‚1.7843011133988724e+09 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.2073085606098175 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.26244065165519714 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9814814329147339 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9873974919319153 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7144568562507629 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7194822430610657 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9250200390815735 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8100529909133911 9"Purity/2FRt*SRqqt1FBt/event_purity‚0.1691666692495346 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9931973218917847 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.6797081232070923 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9992193579673767 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6562555432319641 particle/accuracy_1_of_1‚1 _step‚560 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7905706763267517 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 epoch‚16 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6064230799674988 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6799899339675903 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6558070182800293 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.21220600605010986 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.07692307978868484 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3198700547218323 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.22875340282917023 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.15033812820911407 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 *jet/accuracy_2_of_2‚0.1396089792251587 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6848958730697632 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.030195971950888634 particle/accuracy_1_of_3‚1 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.5714285969734192 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9962392449378967 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5533332824707031 6 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.255983829498291 *jet/accuracy_2_of_4‚0.7835499048233032 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.15311269462108612 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9966624975204468 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.6759387254714966 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.23518267273902893 8"Purity/*FRt*SRqqt0FBt/event_purity‚0.281962513923645 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6035784482955933 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3403683602809906 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.19989974796772003 particle/accuracy_2_of_2‚1 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 particle/accuracy_4_of_5‚1 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9992193579673767 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7857142686843872 jet/accuracy_1_of_5‚1 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9862701892852783 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.6495869755744934 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9984122514724731 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.19767822325229645 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.21230129897594452 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9007157683372498 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.6541121006011963 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4650857150554657 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6148669123649597 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3363739252090454 *jet/accuracy_1_of_4‚0.9695242047309875 /particle/accuracy_3_of_4‚0.9936224818229675 ) Purity/2FRt*SRqqt2FBt/FRt_purity‚0.75 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9984462261199951 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9854843616485596 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7260202169418335 (particle/accuracy‚0.8480902910232544 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.2536255419254303 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9955548048019409 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.5957292914390564 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7731481194496155 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.07543392479419708 *"Purity/2FRt*SRqqt2FBt/event_purity‚0.5 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5687565803527832 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.26244065165519714 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9169871211051941 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.0833333358168602 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9965986609458923 'particle/f_score‚0.6428419947624207 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6046760678291321 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9946774244308472 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.2754203975200653 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.23796780407428741 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9814814329147339 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7731481194496155 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.24960772693157196 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.5660532712936401 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5683568120002747 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.48010003566741943 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 particle/accuracy_2_of_3‚1 6"Purity/*FRt*SRqqt1FBt/event_purity‚0.7428018450737 6 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.998015820980072 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 _runtime‚11297.616697587 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7912320494651794 8"Purity/0FRt1SRqqt1FBt/event_purity‚0.545729398727417 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.05891183391213417 +jet/accuracy_3_of_4‚0.08711894601583481 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.34452730417251587 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 , Purity/0FRt2SRqqt1FBt/FBt_purity‚0.96875 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 particle/accuracy_1_of_4‚1 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.4010416865348816 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5747125744819641 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9747950434684753 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5683568120002747 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9941260814666748 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9615384340286255 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6205047369003296 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5713609457015991 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.7855651378631592 particle/accuracy_1_of_2‚1 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6158018708229065 6 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.995476484298706 7 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.0963035300374031 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.09851446747779846 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.09170838445425034 +jet/accuracy_1_of_1‚0.31912460923194885 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 )loss/val_total_loss‚7.985574890423043 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6510171294212341 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.09610002487897873 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.09842091798782349 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.09170838445425034 7validation_average_jet_accuracy‚0.36016955971717834 particle/accuracy_2_of_5‚1 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.05000000074505806 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.05000000074505806 particle/accuracy_3_of_3‚1 jet/accuracy_5_of_5‚0 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.4166666567325592 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9948779344558716 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6749089360237122 jet/accuracy_4_of_5‚0 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6274787783622742 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03406568244099617 *jet/accuracy_2_of_3‚0.5079657435417175 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 jet/accuracy_2_of_5‚1 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7270423769950867 particle/accuracy_2_of_4‚1 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.31853151321411133 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.2049047201871872 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9986394047737122 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 +particle/specificity‚0.9059207439422607 trainer/global_step‚6799 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.24622857570648193 *jet/accuracy_1_of_3‚0.9134830236434937 +particle/sensitivity‚0.6366313695907593 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.044302862137556076 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.23125796020030975 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9993734955787659 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.2176825851202011 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.7855651378631592 8"Purity/*FRt1SRqqt1FBt/event_purity‚0.428792268037796 5 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.47078737616539 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6481547355651855 jet/accuracy_3_of_5‚1¿z×°´³æ{ =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9992193579673767 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6562555432319641 particle/accuracy_1_of_1‚1 _step‚560 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7905706763267517 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 epoch‚16 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6064230799674988 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6799899339675903 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6558070182800293 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.21220600605010986 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.07692307978868484 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3198700547218323 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.22875340282917023 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.15033812820911407 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 *jet/accuracy_2_of_2‚0.1396089792251587 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6848958730697632 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.030195971950888634 particle/accuracy_1_of_3‚1 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.5714285969734192 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9962392449378967 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5533332824707031 6 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.255983829498291 *jet/accuracy_2_of_4‚0.7835499048233032 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.15311269462108612 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9966624975204468 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.6759387254714966 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.23518267273902893 8"Purity/*FRt*SRqqt0FBt/event_purity‚0.281962513923645 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6035784482955933 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3403683602809906 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.19989974796772003 particle/accuracy_2_of_2‚1 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 particle/accuracy_4_of_5‚1 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9992193579673767 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7857142686843872 jet/accuracy_1_of_5‚1 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9862701892852783 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.6495869755744934 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9984122514724731 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.19767822325229645 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.21230129897594452 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9007157683372498 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.6541121006011963 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4650857150554657 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6148669123649597 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3363739252090454 *jet/accuracy_1_of_4‚0.9695242047309875 /particle/accuracy_3_of_4‚0.9936224818229675 ) Purity/2FRt*SRqqt2FBt/FRt_purity‚0.75 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9984462261199951 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9854843616485596 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7260202169418335 (particle/accuracy‚0.8480902910232544 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.2536255419254303 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9955548048019409 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.5957292914390564 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7731481194496155 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.07543392479419708 *"Purity/2FRt*SRqqt2FBt/event_purity‚0.5 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5687565803527832 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.26244065165519714 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9169871211051941 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.0833333358168602 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9965986609458923 'particle/f_score‚0.6428419947624207 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6046760678291321 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9946774244308472 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.2754203975200653 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.23796780407428741 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9814814329147339 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7731481194496155 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.24960772693157196 9"Pv”ž€8&urity/0FRt1SRqqt*FBt/event_purity‚0.5660532712936401 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5683568120002747 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.48010003566741943 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 particle/accuracy_2_of_3‚1 6"Purity/*FRt*SRqqt1FBt/event_purity‚0.7428018450737 6 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.998015820980072 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 _runtime‚11297.616697587 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7912320494651794 8"Purity/0FRt1SRqqt1FBt/event_purity‚0.545729398727417 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.05891183391213417 +jet/accuracy_3_of_4‚0.08711894601583481 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.34452730417251587 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 , Purity/0FRt2SRqqt1FBt/FBt_purity‚0.96875 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 particle/accuracy_1_of_4‚1 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.4010416865348816 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5747125744819641 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9747950434684753 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5683568120002747 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9941260814666748 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9615384340286255 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6205047369003296 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5713609457015991 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.7855651378631592 particle/accuracy_1_of_2‚1 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6158018708229065 6 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.995476484298706 7 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.0963035300374031 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.09851446747779846 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.09170838445425034 +jet/accuracy_1_of_1‚0.31912460923194885 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 )loss/val_total_loss‚7.985574890423043 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6510171294212341 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.09610002487897873 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.09842091798782349 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.09170838445425034 7validation_average_jet_accuracy‚0.36016955971717834 particle/accuracy_2_of_5‚1 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.05000000074505806 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.05000000074505806 particle/accuracy_3_of_3‚1 jet/accuracy_5_of_5‚0 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.4166666567325592 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9948779344558716 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6749089360237122 jet/accuracy_4_of_5‚0 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6274787783622742 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03406568244099617 *jet/accuracy_2_of_3‚0.5079657435417175 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 jet/accuracy_2_of_5‚1 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7270423769950867 particle/accuracy_2_of_4‚1 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.31853151321411133 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.2049047201871872 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9986394047737122 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 +particle/specificity‚0.9059207439422607 trainer/global_step‚6799 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.24622857570648193 *jet/accuracy_1_of_3‚0.9134830236434937 +particle/sensitivity‚0.6366313695907593 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.044302862137556076 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.23125796020030975 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9993734955787659 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.2176825851202011 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.7855651378631592 8"Purity/*FRt1SRqqt1FBt/event_purity‚0.428792268037796 5 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.47078737616539 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6481547355651855 jet/accuracy_3_of_5‚1 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6794134974479675 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9889451265335083 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.2731499969959259 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.46064817905426025 /particle/accuracy_4_of_4‚0.9505739808082581 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.038885388523340225 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 6 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.260284423828125 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.3350023031234741 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.25227224826812744 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8627544641494751 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6569612622261047 *jet/accuracy_3_of_3‚0.0678483322262764 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2723780870437622 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9974919557571411 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.5732911229133606 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.24119073152542114 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 particle/accuracy_1_of_5‚1 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6529412269592285 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6538980007171631 , Purity/0FRt2SRqqt*FBt/FBt_purity‚0.96875 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 particle/accuracy_5_of_5‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6472569704055786 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7926681041717529 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.0833333358168602 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9583333134651184 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9947671294212341 /"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.421875 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8006047010421753 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 ,jet/accuracy_4_of_4‚0.013899499550461769 particle/accuracy_3_of_5‚1 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5735671520233154 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 8"Purity/*FRt*SRqqt2FBt/event_purity‚0.863677978515625 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.5907489657402039 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7399647831916809 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 8"Purity/0FRt*SRqqt0FBt/event_purity‚0.545141875743866 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 6 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.273772656917572 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 )jet/accuracy_1_of_2‚0.771665096282959 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.07692307978868484 % _timestamp‚1.7843011133988724e+09 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.2073085606098175 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.26244065165519714 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9814814329147339 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9873974919319153 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7144568562507629 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7194822430610657 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9250200390815735 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8100529909133911 9"Purity/2FRt*SRqqt1FBt/event_purity‚0.1691666692495346 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9931973218917847 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.6797081232070923°ç¦6[ž´‚(:“ ÅŒéÒ¹Øâ÷ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚264.23' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚52.846000000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0E®"I µ‚(:ÿ ÅŒéÒ¤¿Çø% disk.nvme0n1.out‚108029.966796875 disk.dm-1.in‚0" disk.nvme0n1.in‚25812.02734375 disk.sda2.out‚0 disk.dm-1.out‚ 578.5390625) disk./.usagePercent‚36.94997267682788$ disk./.usageGB‚550.5812950134277 disk.sda.out‚ 578.5390625 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda3.out‚ 578.5390625 network.recv‚ 2.832736e+06 disk.rbd1.out‚ 30.107421875 disk.drbd2165.in‚0 disk.drbd2165.out‚0* proc.memory.percent‚17.519307136535645 network.sent‚ 1.303496e+07 disk.sda1.out‚0 cpu‚25.541704481738126 proc.cpu.threads‚17" disk.dm-0.out‚108029.966796875 disk.sda1.in‚0 disk.sda3.in‚0 disk.rbd3.in‚0$ disk.nvme0c0n1.in‚25812.02734375) proc.memory.availableMB‚ 1030.79296875# proc.memory.rssMB‚ 2870.36328125 disk.sda2.in‚0 disk.sda.in‚0 disk.rbd3.out‚0 disk.rbd1.in‚ 0.47314453125 disk.dm-0.in‚25812.02734375' disk.nvme0c0n1.out‚108029.966796875$ memory_percent‚93.70853900909424mÒŽŸ¶‚(:” ÔŒéÒÖ¾âö gpu.0.gpu‚46.0 gpu.0.memory‚14, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚38.0 gpu.0.powerWatts‚110.415' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.083000000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0£õ™³ ·‚(:€ ÔŒéÒ–Ò‹ù& disk.nvme0n1.out‚108045.1279296875 disk.sda.out‚ 578.8203125 disk.dm-0.in‚25825.88671875 disk.sda1.in‚0$ disk./.usageGB‚550.5851974487305 disk.sda3.in‚0 disk.drbd2165.in‚0( proc.memory.availableMB‚ 1030.2890625 disk.dm-1.out‚ 578.8203125 disk.rbd0.in‚0 disk.sda1.out‚0 disk.rbd3.out‚0 disk.sda2.in‚0 cpu‚25.54146994085641 disk.sda.in‚0 disk.dm-1.in‚0 disk.sda2.out‚0 disk.rbd1.out‚ 30.107421875# disk.dm-0.out‚108045.1279296875 disk.sda3.out‚ 578.8203125 disk.drbd2165.out‚0" disk.nvme0n1.in‚25825.88671875 network.recv‚ 2.8371e+06* disk./.usagePercent‚36.950234572535415$ disk.nvme0c0n1.in‚25825.88671875# proc.memory.rssMB‚ 2870.36328125* proc.memory.percent‚17.519307136535645 network.sent‚ 1.3092606e+07$ memory_percent‚93.71161460876465 proc.cpu.threads‚17 disk.rbd1.in‚ 0.47314453125 disk.rbd0.out‚0 disk.rbd3.in‚0( disk.nvme0c0n1.out‚108045.1279296875>í©Ÿ¸‚(:” ãŒéÒ‡ç§÷ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚287.087' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚57.417399999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0"½ºþ¹‚(:ó ãŒéÒúöªù% disk.nvme0n1.out‚108053.455078125 disk.rbd1.in‚ 0.47314453125 disk.rbd0.in‚0* proc.memory.percent‚17.519307136535645 disk.rbd3.out‚0 network.recv‚ 2.840703e+06$ disk./.usageGB‚550.5921630859375 disk.dm-1.in‚0 disk.sda3.in‚0 disk.sda3.out‚579.125 disk.drbd2165.in‚0) disk./.usagePercent‚36.95070204229294" disk.nvme0n1.in‚25826.10546875) proc.memory.availableMB‚ 1030.28515625 network.sent‚ 1.309768e+07 disk.rbd3.in‚0# proc.memory.rssMB‚ 2870.36328125 cpu‚25.541304075179447' disk.nvme0c0n1.out‚108053.455078125 disk.sda1.out‚0 disk.sda2.out‚0 disk.sda.in‚0 disk.dm-1.out‚579.125 disk.rbd0.out‚0 disk.dm-0.in‚25826.10546875 disk.sda1.in‚0 proc.cpu.threads‚17$ disk.nvme0c0n1.in‚25826.10546875 disk.sda2.in‚0 disk.sda.out‚579.125" disk.dm-0.out‚108053.455078125$ memory_percent‚93.71163845062256 disk.rbd1.out‚ 30.169921875 disk.drbd2165.out‚0†õ€ÃŸº‚(:” òŒéÒ¿ÌÎø gpu.0.gpu‚74.0 gpu.0.memory‚22, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚38.0 gpu.0.powerWatts‚110.415' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.083000000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0;ß:»‚(:‡ òŒéÒñ†Ðü$ disk.dm-0.out‚108068.00927734375 disk.sda3.out‚ 580.19921875" disk.nvme0n1.in‚25826.52734375# disk./.usageGB‚550.596305847168 disk.dm-1.in‚0 disk.rbd3.out‚0$ disk.nvme0c0n1.in‚25826.52734375 cpu‚25.54112436968207 disk.dm-1.out‚ 580.19921875 disk.sda3.in‚0 disk.rbd3.in‚0' disk.nvme0n1.out‚108068.00927734375) proc.memory.availableMB‚ 1030.52734375 network.recv‚ 2.844004e+06 disk.rbd0.out‚0 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.sda2.out‚0 disk.rbd1.out‚ 30.169921875$ memory_percent‚93.71016025543213 disk.rbd0.in‚0) disk.nvme0c0n1.out‚108068.00927734375 disk.rbd1.in‚ 0.47314453125 proc.cpu.threads‚17) disk./.usagePercent‚36.95098006647512 disk.sda.out‚ 580.19921875 network.sent‚ 1.3102711e+07 disk.sda.in‚0 disk.sda1.in‚0 disk.sda2.in‚0* proc.memory.percent‚17.519307136535645 disk.dm-0.in‚25826.52734375 disk.sda1.out‚0# proc.memory.rssMB‚ 2870.36328125îUúª”¼‚(:‰ éÒÉ€éö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚260.476' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.0952 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0wèh½‚(:ˆ éÒË…ãø disk.rbd1.in‚ 0.47314453125 disk.sda.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 581.87109375 disk.sda1.out‚0 disk.sda3.in‚0 disk.rbd3.out‚0$ disk.nvme0c0n1.in‚25826.73046875) disk./.usagePercent‚36.95125425054429 disk.sda.out‚ 581.87109375 disk.sda1.in‚0 disk.sda3.out‚ 581.87109375 disk.drbd2165.out‚0$ memory_percent‚93.71163845062256 cpu‚25.540890786676247 network.recv‚ 2.847474e+06 disk.rbd0.out‚0) disk.nvme0c0n1.out‚108080.12353515625' disk.nvme0n1.out‚108080.12353515625* proc.memory.percent‚17.519307136535645 disk.dm-0.in‚25826.73046875 disk.rbd1.out‚ 30.248046875 disk.rbd0.in‚0$ disk.dm-0.out‚108080.12353515625" disk.nvme0n1.in‚25826.73046875 disk.sda2.in‚0 proc.cpu.threads‚17$ disk./.usageGB‚550.6003913879395 network.sent‚ 1.310723e+07 disk.drbd2165.in‚0 disk.rbd3.in‚0 disk.sda2.out‚0) proc.memory.availableMB‚ 1030.28515625# proc.memory.rssMB‚ 2870.36328125A¯§‘4¾/ _wandbruntime‚11375  _runtime‚11375úïÆ¿À _step‚561 % lr-AdamW/pg1‚0.000522919944541473 % lr-AdamW/pg2‚0.000522919944541473 trainer/global_step‚6849 % _timestamp‚1.7843011910582185e+09 _runtime‚11375.275387744]¢ñËÀÅ % lr-AdamW/pg1‚0.000522919944541473 % lr-AdamW/pg2‚0.000522919944541473 trainer/global_step‚6849 % _timestamp‚1.7843011910582185e+09 _runtime‚11375.275387744 _step‚561±Gù4Á/ _wandbruntime‚11375  _runtime‚11375š•D(ÂýO MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1969468187901952 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚531791552163840 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1975912063771136 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6434219110094 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 117786620 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6443902042624 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚59327053 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚59337026 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚258 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 trainer/global_step‚6849 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8313299010048 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1968617146191872 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚59337026 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1968617120763904 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19449241 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚58459567 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 117796599 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚58459567 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚17575360 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚19448983 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6432874509772 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1969468161728512 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 _step‚562 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚531791920254976 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6445247432192 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1975913435334144 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚37024601 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8313333546496 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚11375.27669391 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6445247432192 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚1975049995273676 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1969468161728512 % _timestamp‚1.7843011910593624e+09 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚59327053 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚1969468187901952 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚58459573 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0¿a#ÂP ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1969468187901952 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚531791552163840 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1975912063771136 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6434219110094 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 117786620 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6443902042624 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚59327053 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚59337026 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚258 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 trainer/global_step‚6849 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8313299010048 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1968617146191872 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚59337026 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1968617120763904 WGDeviceStatsMoniŽDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚58459573 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648²C2²4Ä/ _wandbruntime‚11376  _runtime‚11376¤Šæ'ňN [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1975328800518098 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚59345518 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚59335531 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 117813297 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚17577903 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚531869406662656 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6435142576452 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚37029608 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚59335531 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19451710 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚59345518 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚58467773 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1976192361106944 trainer/global_step‚6849 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1969746162458624 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1976192361106944 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚58467779 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8314498501120 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 117813297 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1969746162458624 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6444827078144 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1975330170546500 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6433797976018 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚17577898 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚531869038571520 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚540183537072640 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 117813297 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚17577903 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚531869406662656 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6435142576452 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚37029608 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚59335531 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19451710 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚59345518 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚58467773 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1976192361106944 trainer/global_step‚6849 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1969746162458624 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1976192361106944 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚58467779 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8314498501120 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 117813297 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1969746162458624 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6444827078144 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1975330170546500 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6433797976018 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚17577898 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚531869038571520 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚540183537072640 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 EDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8372944859648 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚58869867 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6491563362816 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1989858416207360 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8372981493248 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚17702505 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚37291475 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1982511209672704 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚59762778 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚59762778 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 _step‚566 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6490217973248 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚17702510 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404724736 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚535665872547840 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1982511235100672 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6480456223202 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1989859787770368 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1989858416207360 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6479111622880 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚59752805 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1989859787770368 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚535666240638976 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚59752805 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚544038817407488 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 118622672 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36633600 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6491563362816 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚1989858416207360 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8372981493248 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚17702505 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚37291475 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1982511209672704 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚59762778 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚59762778 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 _step‚566 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6490217973248 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚17702510 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404724736 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚535665872547840 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1982511235100672 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6480456223202 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚1989859787770368 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚1989858416207360 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6479111622880 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚59752805 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚1989859787770368 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚535666240638976 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚59752805 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚544038817407488 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 118622672 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36633600 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8372944859648 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚58869867¶¼L4Ú/ _wandbruntime‚11455  _runtime‚11455öhÆÕ'ÛˆN YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6492488405504 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 118639356 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8374163936768 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36626432 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1983646225137664 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚267 trainer/global_step‚6899 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚17705053 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚37296509 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚544117890983424 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1983646198964224 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8374127310336 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1990137341972992 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6492488405504 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6491143008768 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1990138713543168 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1982789091450880 % _timestamp‚1.7843012712045994e+09 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1983646198964224 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19591728 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1989270496568408 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 118639356 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚17705048 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1990138713543168 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚58878079 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6480035089126 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚37296781 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚59771270 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1983646225137664 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚59771270 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1989269126540006 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚59761283 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6491143008768 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚272 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 118649349 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚58878073 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚59761283 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚535743358955520 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚535743727046656 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚58878079 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 _runtime‚11455.422057389 _step‚567 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926ý"ÇCÜN REDeviceStatsMonitor.on_train_batch_end/allocation.small_po3ùïžÐ&ol.allocated‚59771270 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚1989269126540006 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚59761283 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6491143008768 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚272 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 118649349 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚58878073 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚59761283 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚535743358955520 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚535743727046656 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚58878079 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 _runtime‚11455.422057389 _step‚567 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6492488405504 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 118639356 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8374163936768 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36626432 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚1983646225137664 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚267 trainer/global_step‚6899 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚17705053 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚37296509 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚544117890983424 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1983646198964224 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8374127310336 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚1990137341972992 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6492488405504 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6491143008768 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚1990138713543168 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1982789091450880 % _timestamp‚1.7843012712045994e+09 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1983646198964224 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19591728 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚1989270496568408 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 118639356 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚17705048 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚1990138713543168 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚58878079 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6480035089126 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚37296781 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚59771270 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1983646225137664 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175·†œ=4Ý/ _wandbruntime‚11455  _runtime‚11455ÝŠ Þ† 0loss/FRt1/assignment_loss‚1.4732719258478677 0loss/FRt2/assignment_loss‚0.6782765428123572 3loss/SRqqt2/assignment_loss‚0.48103754678794763 % _timestamp‚1.7843012712909703e+09 3loss/SRqqt1/assignment_loss‚0.46899829932637643 1loss/FBt2/assignment_loss‚0.03684227206472302 %loss/total_loss‚7.951179035492843 _runtime‚11455.507909941 1loss/FBt1/assignment_loss‚0.06959092784153142 /loss/FRt2/detection_loss‚1.5328297624888514 1loss/SRqqt2/detection_loss‚0.5780843916245205 epoch‚17 trainer/global_step‚6899 /loss/FRt1/detection_loss‚0.5444984410939391 1loss/SRqqt1/detection_loss‚0.9798965234736431 /loss/FBt1/detection_loss‚0.7288096186450936 0loss/FBt2/detection_loss‚0.37904278348599163 _step‚568Òß‹ trainer/global_step‚6899 /loss/FRt1/detection_loss‚0.5444984410939391 1loss/SRqqt1/detection_loss‚0.9798965234736431 /loss/FBt1/detection_loss‚0.7288096186450936 0loss/FBt2/detection_loss‚0.37904278348599163 _step‚568 0loss/FRt1/assignment_loss‚1.4732719258478677 0loss/FRt2/assignment_loss‚0.6782765428123572 3loss/SRqqt2/assignment_loss‚0.48103754678794763 % _timestamp‚1.7843012712909703e+09 3loss/SRqqt1/assignment_loss‚0.46899829932637643 1loss/FBt2/assignment_loss‚0.03684227206472302 %loss/total_loss‚7.951179035492843 _runtime‚11455.507909941 1loss/FBt1/assignment_loss‚0.06959092784153142 /loss/FRt2/detection_loss‚1.5328297624888514 1loss/SRqqt2/detection_loss‚0.5780843916245205 epoch‚17¸ƒi””à‚(:‰ ÛéÒ´òø gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚257.762' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚51.5524 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0½ß0á‚(:‰ ÛéÒìæáù disk.sda3.out‚ 583.59765625 disk.sda1.in‚0 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.rbd1.out‚ 30.451171875$ memory_percent‚93.70510578155518$ disk.dm-0.out‚108147.85888671875 disk.drbd2165.in‚0 cpu‚25.539651279598154 disk.dm-0.in‚26649.62109375 disk.dm-1.out‚ 583.59765625) disk.nvme0c0n1.out‚108147.85888671875' disk.nvme0n1.out‚108147.85888671875 disk.drbd2165.out‚0$ disk.nvme0c0n1.in‚26649.62109375" disk.nvme0n1.in‚26649.62109375 disk.rbd1.in‚ 0.47314453125) proc.memory.availableMB‚ 1031.35546875# proc.memory.rssMB‚ 2870.36328125 proc.cpu.threads‚17) disk./.usagePercent‚36.95403346833601$ disk./.usageGB‚550.6418037414551 disk.sda.in‚0 disk.sda.out‚ 583.59765625 disk.rbd3.in‚0 network.sent‚ 1.3196145e+07 disk.sda1.out‚0 disk.rbd3.out‚0 disk.sda2.in‚0 disk.sda2.out‚0* proc.memory.percent‚17.519307136535645 disk.rbd0.out‚0 network.recv‚ 2.868689e+06 disk.sda3.in‚0‹ ¬àžâ‚(:“ êéÒÜõÂö gpu.0.gpu‚73.0 gpu.0.memory‚24, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚333.618' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚66.72359999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0rhÏã‚(:… êéÒ†‹€ú disk.sda3.out‚ 583.73046875* proc.memory.percent‚17.519307136535645 disk.sda2.out‚0 disk.dm-1.in‚0 disk.rbd0.in‚0$ memory_percent‚93.70510578155518& disk.nvme0n1.out‚108162.5302734375 disk.rbd1.out‚ 30.451171875 network.sent‚ 1.3258353e+07) disk./.usagePercent‚36.95449095379969" disk.nvme0n1.in‚26650.04296875 disk.sda.in‚0$ disk.nvme0c0n1.in‚26650.04296875 cpu‚25.53946801743172 disk.sda2.in‚0 network.recv‚ 2.872467e+06( disk.nvme0c0n1.out‚108162.5302734375 disk.dm-1.out‚ 583.73046875 proc.cpu.threads‚17 disk.rbd1.in‚ 0.47314453125 disk.dm-0.in‚26650.04296875# disk.dm-0.out‚108162.5302734375 disk.sda3.in‚0 disk.drbd2165.in‚0 disk.rbd3.out‚0 disk.sda1.in‚0 disk.drbd2165.out‚0# proc.memory.rssMB‚ 2870.36328125 disk.sda.out‚ 583.73046875 disk.sda1.out‚0) proc.memory.availableMB‚ 1031.35546875$ disk./.usageGB‚550.6486206054688 disk.rbd3.in‚0 disk.rbd0.out‚0ÀƆǔä‚(:‰ ùéÒ¯Ûö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚285.532' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚57.1064 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0µoC¾ å‚(:þ ùéÒâÆ¿ù' disk.nvme0n1.out‚108170.24658203125 disk.rbd1.in‚ 0.47314453125 disk.rbd1.out‚ 30.529296875# proc.memory.rssMB‚ 2870.36328125 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.sda3.in‚0" disk.nvme0n1.in‚26650.26171875 disk.sda2.out‚0) proc.memory.availableMB‚ 1031.34765625 proc.cpu.threads‚17 disk.rbd0.out‚0 disk.sda1.in‚0 disk.rbd3.in‚0 disk.sda2.in‚0 cpu‚25.5392452167961 network.sent‚ 1.3263217e+07 disk.sda.out‚ 583.921875 disk.sda1.out‚0 disk.drbd2165.in‚0" memory_percent‚93.705153465271 disk.dm-1.out‚ 583.921875 disk.drbd2165.out‚0 disk.dm-0.in‚26650.26171875$ disk.dm-0.out‚108170.24658203125$ disk.nvme0c0n1.in‚26650.26171875( disk./.usagePercent‚36.9548109632175 disk.sda3.out‚ 583.921875$ disk./.usageGB‚550.6533889770508 disk.sda.in‚0) disk.nvme0c0n1.out‚108170.24658203125* proc.memory.percent‚17.519307136535645 network.recv‚ 2.876225e+06 disk.rbd3.out‚0à6æ‚(:… ˆŽéÒÆòðö gpu.0.gpu‚98.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚293.8' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚58.76 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 SÐç‚(:… ˆŽéÒÝ¡øù# disk.dm-0.out‚108181.9638671875 disk.drbd2165.out‚0 proc.cpu.threads‚17 disk.sda.out‚ 584.23046875 disk.sda2.in‚0 disk.rbd0.out‚0) proc.memory.availableMB‚ 1031.33203125 disk.sda.in‚0 disk.sda3.in‚0 disk.drbd2165.in‚0( disk.nvme0c0n1.out‚108181.9638671875 disk.sda2.out‚0 disk.rbd1.in‚ 0.47314453125 disk.rbd1.out‚ 30.529296875 disk.sda3.out‚ 584.23046875$ memory_percent‚93.70524883270264 network.sent‚ 1.3268239e+07 disk.dm-1.in‚0 disk.rbd3.in‚0 disk.rbd3.out‚0* proc.memory.percent‚17.519307136535645 cpu‚25.53906198002774 network.recv‚ 2.879541e+06) disk./.usagePercent‚36.95543254951063 disk.dm-1.out‚ 584.23046875 disk.dm-0.in‚26737.65234375 disk.sda1.out‚0# proc.memory.rssMB‚ 2870.36328125$ disk./.usageGB‚550.6626510620117$ disk.nvme0c0n1.in‚26737.65234375" disk.nvme0n1.in‚26737.65234375& disk.nvme0n1.out‚108181.9638671875 disk.rbd0.in‚0 disk.sda1.in‚0‘?bžè‚(:“ —ŽéÒþç“÷ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚275.782' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚55.15639999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ü™Ìlé‚(:û —ŽéÒõè‘ú disk.drbd2165.out‚0% disk.nvme0c0n1.out‚108195.5546875" disk.nvme0n1.in‚26737.85546875 disk.rbd0.out‚0# proc.memory.rssMB‚ 2870.36328125$ disk./.usageGB‚550.6668663024902 disk.rbd1.in‚ 0.47314453125 disk.rbd0.in‚0 disk.sda3.out‚ 584.64453125' proc.memory.availableMB‚ 1031.546875 disk.sda2.in‚0 network.recv‚ 2.883413e+06 disk.drbd2165.in‚0 disk.dm-0.in‚26737.85546875 disk.dm-0.out‚108195.5546875 network.sent‚ 1.3273398e+07 disk.sda1.out‚0 disk.sda1.in‚0$ disk.nvme0c0n1.in‚26737.85546875 disk.rbd1.out‚ 30.591796875 disk.dm-1.in‚0* proc.memory.percent‚17.519307136535645 cpu‚25.538858760435645# disk.nvme0n1.out‚108195.5546875 disk.sda2.out‚0 disk.dm-1.out‚ 584.64453125$ memory_percent‚93.70393753051758 proc.cpu.threads‚17 disk.rbd3.in‚0 disk.sda3.in‚0 disk.rbd3.out‚0 disk.sda.in‚0 disk.sda.out‚ 584.64453125) disk./.usagePercent‚36.95571543783596¯ “k4ê/ _wandbruntime‚11532  _runtime‚11532õ+âÆëÀ _step‚569 % lr-AdamW/pg1‚0.000518133510044508 % lr-AdamW/pg2‚0.000518133510044508 trainer/global_step‚6949 % _timestamp‚1.7843013483662882e+09 _runtime‚11532.583442687’Ð\ËìÅ _runtime‚11532.583442687 _step‚569 % lr-AdamW/pg1‚0.000518133510044508 % lr-AdamW/pg2‚0.000518133510044508 trainer/global_step‚6949 % _timestamp‚1.7843013483662882e+09¹•ð¸p4í/ _wandbruntime‚11532  _runtime‚11532oH¶Í(îüO P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36633600 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1997268260913152 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚37557432 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚19727777 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚547972446315008 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19728054 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 _runtime‚11532.584660526 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1997268234739712 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚60188530 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1997268234739712 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6537879293440 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 119458724 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚59280167 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚60178557 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚17829655 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1996405298581504 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 trainer/global_step‚6949 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6525348735988 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚59280173 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2002930647317492 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6537879293440 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚59280167 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8432253383168 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6536533903872 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚277 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6526693336310 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6536533903872 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404724736 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2003804768643584 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚17829660 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 # _timestamp‚1.78430134836735e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2003806140206592 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 119468703 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚547972851039744 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚60178557 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2003804768643584 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1996405324009472 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚570 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2003806140206592 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8432290016768 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2002932017345782 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚1997268260913152 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚37557432 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚19727777 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚547972446315008 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19728054 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 NFDeviceStatsMonitor.on¦¥æ_train_batch_start/inactive_split.small_pool.peak‚545 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 _runtime‚11532.584660526 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚1997268234739712 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚60188530 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚1997268234739712 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6537879293440 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 119458724 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚59280167 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚60178557 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚17829655 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚1996405298581504 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 trainer/global_step‚6949 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6525348735988 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚59280173 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2002930647317492 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6537879293440 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚59280167 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8432253383168 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6536533903872 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚277 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6526693336310 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6536533903872 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404724736 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2003804768643584 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚17829660 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 # _timestamp‚1.78430134836735e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2003806140206592 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 119468703 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚547972851039744 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚60178557 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2003804768643584 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚1996405324009472 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154ºãwÉ4ð/ _wandbruntime‚11534  _runtime‚11534?(¶È 'ñ†N J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚59288373 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 119475408 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚59288379 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚60187035 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚59288379 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚60187035 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6537458939392 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚262 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 119475408 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1997546235469824 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404717568 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 trainer/global_step‚6949 # _timestamp‚1.78430134994017e+09 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2004085065979392 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚539617679339520 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 _runtime‚11534.157845546 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚60197022 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2004085065979392 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2004083694409216 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 119485401 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1996683180359680 _step‚571 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2003210822590316 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚59288373 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8433436258816 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚17832203 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚548051520315904 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚60197022 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚548051115598336 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1997546235469824 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚267 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1997546261643264 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚37562724 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0ÿˆß¨'ò‹N E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚1996683205787648 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚548051520315904 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚60197022 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚548051115598336 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚1997546235469824 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚267 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚1997546261643264 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚37562724 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚59288373 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 119475408 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚59288379 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚60187035 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚59288379 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚60187035 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6537458939392 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚262 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 119475408 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚1997546235469824 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404717568 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 trainer/global_step‚6949 # _timestamp‚1.78430134994017e+09 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2004085065979392 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚539617679339520 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 _runtime‚11534.157845546 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚60197022 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2004085065979392 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2004083694409216 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 119485401 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚1996683180359680 _step‚571 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2003210822590316 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚59288373 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8433436258816 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚17832203 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6572930449418 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚17956810 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19866752 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚551907002789376 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚543414513315840 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2011168271245312 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚551906598064640 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8492121382400 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6571585849096 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 120304755 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚59690473 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2011168297418752 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2016870973339400 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2017751121079808 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚60604309 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚260 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚60604309 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚17956805 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 % _timestamp‚1.7843014271045365e+09 _runtime‚11611.321857444 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚60614282 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8492084748800 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚59690467 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2016872343367690 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 _step‚574 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚19866492 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6572930449418 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚17956810 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19866752 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚551907002789376 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚543414513315840 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2011168271245312 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚551906598064640 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8492121382400 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6571585849096 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 120304755 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚59690473 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2011168297418752 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2016870973339400 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2017751121079808 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚60604309 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚260 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚60604309 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚17956805 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 % _timestamp‚1.7843014271045365e+09 _runtime‚11611.321857444 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚60614282 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8492084748800 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚59690467 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2016872343367690 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 _step‚574 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚19866492 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚59698679 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚59698673 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚543492367814656 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 120321453 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8493312442880 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8493277913600 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19869483 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6585120266752 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2011446271975424 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19869230 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.a,jxIj llocated‚60622774 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 _step‚575 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2011446271975424 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚17959348 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 % _timestamp‚1.7843014284898016e+09 _runtime‚11612.707194139 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚37828578 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚253 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2011446298148864 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2018031418415616 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6583774870016 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 120311460 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 120311460 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚60622774 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚258 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚551985680257536 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6572509315342 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2017151148612224 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2017149778583822 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6573853915776 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2010577294696448À'I»'ŠN UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6583774870016 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚59698679 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚59698673 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚59698679 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚59698673 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚543492367814656 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 120321453 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8493312442880 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8493277913600 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19869483 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6585120266752 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2011446271975424 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19869230 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚60622774 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 _step‚575 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2011446271975424 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚17959348 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 % _timestamp‚1.7843014284898016e+09 _runtime‚11612.707194139 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚37828578 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚253 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2011446298148864 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2018031418415616 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6583774870016 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 120311460 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 120311460 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚60622774 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚258 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚551985680257536 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6572509315342 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2017151148612224 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2017149778583822 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6573853915776 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2010577294696448 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2011446298148864 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚543491999723520 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2018030046845440 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚551985277637120 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2018031418415616 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚37828836 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648¿×râ4‹/ _wandbruntime‚11612  _runtime‚11612— ÒŒ‰ 1loss/SRqqt1/detection_loss‚0.7529679893599158 /loss/FBt1/detection_loss‚0.6297112926105527 /loss/FBt2/detection_loss‚0.5167642624962328 _runtime‚11612.793341297 4loss/FBt2/assignment_loss‚0.00041173482716416743 /loss/FRt2/detection_loss‚1.5046442547356136 %loss/total_loss‚7.634810123897381 0loss/FRt2/assignment_loss‚0.5207961158522332 2loss/SRqqt1/assignment_loss‚0.5057201376596757 2loss/SRqqt2/assignment_loss‚0.4757973120069255 0loss/FRt1/detection_loss‚0.49557149386271215 epoch‚17 trainer/global_step‚6999 _step‚576 0loss/FRt1/assignment_loss‚1.6043154902249324 1loss/SRqqt2/detection_loss‚0.6280419715742893 % _timestamp‚1.7843014285763958e+09 3loss/FBt1/assignment_loss‚6.806868713426477e-05Ürø”Ž 0loss/FRt2/assignment_loss‚0.5207961158522332 2loss/SRqqt1/assignment_loss‚0.5057201376596757 2loss/SRqqt2/assignment_loss‚0.4757973120069255 0loss/FRt1/detection_loss‚0.49557149386271215 epoch‚17 trainer/global_step‚6999 _step‚576 0loss/FRt1/assignment_loss‚1.6043154902249324 1loss/SRqqt2/detection_loss‚0.6280419715742893 % _timestamp‚1.7843014285763958e+09 3loss/FBt1/assignment_loss‚6.806868713426477e-05 1loss/SRqqt1/detection_loss‚0.7529679893599158 /loss/FBt1/detection_loss‚0.6297112926105527 /loss/FBt2/detection_loss‚0.5167642624962328 _runtime‚11612.793341297 4loss/FBt2/assignment_loss‚0.00041173482716416743 /loss/FRt2/detection_loss‚1.5046442547356136 %loss/total_loss‚7.634810123897381À†X VžŽ‚(:“ €éÒÚÊŸ÷ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚273.684' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚54.73680000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 k§‚(:‡ €éÒÊëìø proc.cpu.threads‚17 network.sent‚ 1.3367042e+07 disk.rbd1.in‚ 0.47314453125$ disk.dm-0.out‚108275.95068359375" disk.nvme0n1.in‚26831.29296875 disk.rbd1.out‚ 30.810546875* proc.memory.percent‚17.519307136535645 cpu‚25.537462692579492 disk.dm-1.in‚0$ memory_percent‚93.70818138122559) disk./.usagePercent‚36.95863801984685 disk.dm-1.out‚ 586.78515625 disk.dm-0.in‚26831.29296875 disk.sda3.in‚0 disk.sda3.out‚ 586.78515625' disk.nvme0n1.out‚108275.95068359375 disk.sda1.in‚0 disk.rbd3.out‚0$ disk.nvme0c0n1.in‚26831.29296875 disk.sda2.out‚0( proc.memory.availableMB‚ 1030.8515625# proc.memory.rssMB‚ 2870.36328125 disk.drbd2165.out‚0 disk.rbd3.in‚0) disk.nvme0c0n1.out‚108275.95068359375 disk.sda.in‚0 disk.sda.out‚ 586.78515625$ disk./.usageGB‚550.7104148864746 disk.rbd0.out‚0 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.sda2.in‚0 network.recv‚ 2.90826e+06 disk.rbd0.in‚0mºÈŸ‚(:” éÒ“Ìö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚301.343' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚60.268600000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0÷UÖ±‘‚(:ø éÒÝ»°ù disk.dm-0.in‚ 26924.3828125# disk.dm-0.out‚108298.6728515625 disk.sda1.in‚0 disk.drbd2165.in‚0 disk.rbd1.out‚ 30.810546875$ memory_percent‚93.70815753936768 disk.sda3.out‚589.3125 disk.sda2.in‚0# proc.memory.rssMB‚ 2870.36328125 disk.sda1.out‚0* disk./.usagePercent‚36.959365849266696 disk.rbd0.out‚0 disk.rbd3.in‚0! disk.nvme0n1.in‚ 26924.3828125 disk.dm-1.out‚589.3125 disk.rbd3.out‚0& disk.nvme0n1.out‚108298.6728515625 network.sent‚ 1.3430221e+07 disk.sda.out‚589.3125 disk.rbd0.in‚0 disk.rbd1.in‚ 0.47314453125) proc.memory.availableMB‚ 1030.85546875 proc.cpu.threads‚17 network.recv‚ 2.912772e+06 disk.dm-1.in‚0 disk.sda3.in‚0 disk.drbd2165.out‚0# disk.nvme0c0n1.in‚ 26924.3828125( disk.nvme0c0n1.out‚108298.6728515625 cpu‚25.537304884401916 disk.sda2.out‚0* proc.memory.percent‚17.519307136535645$ disk./.usageGB‚550.7212600708008 disk.sda.in‚0w8« Ÿ’‚(:” žéÒ’û‹÷ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚279.015' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚55.803000000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0}xŸ‚“‚(:ƒ žéÒåÃèù# disk.nvme0c0n1.in‚ 26924.5859375' disk.nvme0n1.out‚108318.26806640625 disk.rbd1.in‚ 0.47314453125 disk.rbd0.out‚0 disk.dm-0.in‚ 26924.5859375 cpu‚25.537083709747733 proc.cpu.threads‚17$ disk./.usageGB‚550.7286262512207 disk.sda.out‚ 592.0703125 disk.dm-1.in‚0 disk.drbd2165.out‚0 network.recv‚ 2.915841e+06 disk.rbd3.in‚0! disk.nvme0n1.in‚ 26924.5859375 disk.sda1.out‚0 disk.drbd2165.in‚0* proc.memory.percent‚17.519307136535645) disk./.usagePercent‚36.95986019981532 disk.rbd3.out‚0 disk.sda2.out‚0 disk.sda3.out‚ 592.0703125 network.sent‚ 1.3435303e+07$ disk.dm-0.out‚108318.26806640625 disk.sda2.in‚0 disk.sda3.in‚0$ memory_percent‚93.70973110198975# proc.memory.rssMB‚ 2870.36328125 disk.rbd1.out‚ 30.873046875) disk.nvme0c0n1.out‚108318.26806640625 disk.sda.in‚0 disk.rbd0.in‚0 disk.sda1.in‚0 disk.dm-1.out‚ 592.0703125) proc.memory.availableMB‚ 1030.59765625©¯ös‘”‚(:† ­éÒèñö gpu.0.gpu‚91.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚338.35' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚67.67 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‰øb•‚(:‡ ­éÒ¨Þæø* disk./.usagePercent‚36.960380663132426 disk.dm-1.in‚0) proc.memory.availableMB‚ 1030.59765625 disk.sda1.out‚0 disk.sda3.in‚0 disk.rbd0.in‚0 disk.dm-0.in‚ 26925.0078125 disk.rbd3.out‚0# disk.nvme0c0n1.in‚ 26925.0078125! disk.nvme0n1.in‚ 26925.0078125 disk.drbd2165.in‚0 disk.sda.out‚ 594.10546875 cpu‚25.536912459826624 disk.dm-1.out‚ 594.10546875 disk.rbd0.out‚0 disk.sda1.in‚0# proc.memory.rssMB‚ 2870.36328125 network.sent‚ 1.3440323e+07 disk.sda2.in‚0 disk.sda.in‚0$ disk./.usageGB‚550.7363815307617$ disk.dm-0.out‚108336.59130859375 disk.rbd3.in‚0' disk.nvme0n1.out‚108336.59130859375* proc.memory.percent‚17.519307136535645 disk.sda3.out‚ 594.10546875 disk.drbd2165.out‚0 disk.sda2.out‚0 disk.rbd1.out‚ 30.873046875 proc.cpu.threads‚17 network.recv‚ 2.919396e+06) disk.nvme0c0n1.out‚108336.59130859375 disk.rbd1.in‚ 0.47314453125$ memory_percent‚93.70973110198975¨´Gž–‚(:“ ¼éÒ¤¤²÷ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚262.054' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚52.41079999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0±¦ãÎ —‚(:‚ ¼éÒàá×ø disk.sda3.in‚0) disk./.usagePercent‚36.96046898573174 disk.sda3.out‚ 596.8046875 disk.rbd3.out‚0# disk.nvme0c0n1.in‚ 26925.2265625 disk.sda2.in‚0 disk.dm-1.in‚0) disk.nvme0c0n1.out‚108359.64892578125 disk.sda.out‚ 596.8046875 disk.rbd0.in‚0* proc.memory.percent‚17.519307136535645' disk.nvme0n1.out‚108359.64892578125 disk.sda.in‚0 disk.dm-1.out‚ 596.8046875) proc.memory.availableMB‚ 1030.75390625 proc.cpu.threads‚17$ disk./.usageGB‚550.7376976013184 disk.dm-0.in‚ 26925.2265625 disk.drbd2165.out‚0 disk.rbd1.in‚ 0.47314453125# proc.memory.rssMB‚ 2870.36328125 disk.rbd0.out‚0 disk.sda1.in‚0 disk.sda2.out‚0$ memory_percent‚93.70877742767334 network.sent‚ 1.3445387e+07 network.recv‚ 2.922766e+06 disk.sda1.out‚0 disk.drbd2165.in‚0! disk.nvme0n1.in‚ 26925.2265625$ disk.dm-0.out‚108359.64892578125 disk.rbd3.in‚0 disk.rbd1.out‚ 30.951171875 cpu‚25.53669488531991Dä 4˜/ _wandbruntime‚11690  _runtime‚11690ºS¸Ç™Á trainer/global_step‚7049 $ _timestamp‚1.784301505949397e+09 _runtime‚11690.166565848 _step‚577 & lr-AdamW/pg1‚0.0005133028335358755 & lr-AdamW/pg2‚0.0005133028335358755û vÌšÆ $ _timestamp‚1.784301505949397e+09 _runtime‚11690.166565848 _step‚577 & lr-AdamW/pg1‚0.0005133028335358755 & lr-AdamW/pg2‚0.0005133028335358755 trainer/global_step‚7049Á°ÖŒ4›/ _wandbruntime‚11690  _runtime‚11690+ZØX'œþO ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2031698845079040 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2024193476399104 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 121130828 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚60100767 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8551796156928 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2025068333924352 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚60100767 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2031698845079040 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 _runtime‚11690.167924994 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚555840997947904 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚547288833699840 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2030812669389598 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚38088651 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚61040034 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20004696 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2031697473516032 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6629165765120 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2024193501827072 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6630511154688 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚61030061 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚38088904 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20004944 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 % _timestamp‚1.7843015059505007e+09 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚61040034 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2025068307750912 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚547289201790976 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2025068307750912 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 trainer/global_step‚7049 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2031697473516032 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚18083960 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚555840595320320 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 O?DeviceStatsMonitor.Õ^9¬on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2025068333924352 _step‚578rM)Ñ (ƒP C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2025068307750912 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚547289201790976 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2025068307750912 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 trainer/global_step‚7049 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2031697473516032 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚18083960 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚555840595320320 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2025068333924352 _step‚578 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2031698845079040 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2024193476399104 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 121130828 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚60100767 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8551796156928 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2025068333924352 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚60100767 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2031698845079040 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 _runtime‚11690.167924994 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚555840997947904 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚547288833699840 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2030812669389598 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚38088651 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚61040034 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20004696 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2031697473516032 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6629165765120 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2024193501827072 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6630511154688 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚61030061 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚38088904 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20004944 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 % _timestamp‚1.7843015059505007e+09 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚61040034 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008Âå˜,04ž/ _wandbruntime‚11691  _runtime‚116917.P'ŸˆN ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2031977770851840 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 121157505 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚18086503 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8552953655808 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6618746428450 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚20007675 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 121157505 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8552988185088 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚61048526 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2025346334654464 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 % _timestamp‚1.7843015073370302e+09 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚555919273763328 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2025346308481024 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚18086498 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚244 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2024471383605248 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2031977770851840 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6631436197376 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 _runtime‚11691.554380614 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 121147512 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2031091474634132 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚60108973 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚60108979 trainer/global_step‚7049 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚61038539 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚249 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚38093929 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 QDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2031977770851840 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 121157505 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚18086503 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8552953655808 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6618746428450 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚20007675 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 121157505 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8552988185088 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚61048526 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2025346334654464 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 % _timestamp‚1.7843015073370302e+09 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚555919273763328 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2025346308481024 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚18086498 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚244 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2024471383605248 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2031977770851840 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6631436197376 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 _runtime‚11691.554380614 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 121147512 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2031091474634132 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚60108973 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚60108979 trainer/global_step‚7049 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚61038539 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚249 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚38093929 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640ÃÉYò4¡/ _wandbruntime‚11691  _runtime‚11691A¶ ¢† 0loss/FRt2/assignment_loss‚0.6045705058014802 3loss/FBt2/assignment_loss‚0.0006911893309599557 /loss/FBt2/detection_loss‚0.5080453372629429 epoch‚17 % _timestamp‚1.7843015074232912e+09 _runtime‚11691.640223542 _step‚580 3loss/SRqqt2/assignment_loss‚0.48849666472701314 /loss/FRt1/detection_loss‚0.5583725535333393 /loss/FRt2/detection_loss‚1.6134657278733135 %loss/total_loss‚8.358188764169725 3loss/FBt1/assignment_loss‚0.0013968797377261984 1loss/SRqqt1/detection_loss‚0.8588131102733314 1loss/SRqqt1/assignment_loss‚0.521638086714845 0loss/SRqqt2/detection_loss‚0.597074665585376 /loss/FBt1/detection_loss‚1.1225954647143852 trainer/global_step‚7049 0loss/FRt1/assignment_loss‚1.4830285786150124 |p£‹ % _timestamp‚1.7843015074232912e+09 _runtime‚11691.640223542 _step‚580 3loss/SRqqt2/assignment_loss‚0.48849666472701314 /loss/FRt1/detection_loss‚0.5583725535333393 /loss/FRt2/detection_loss‚1.6134657278733135 %loss/total_loss‚8.358188764169725 3loss/FBt1/assignment_loss‚0.0013968797377261984 1loss/SRqqt1/detection_loss‚0.8588131102733314 1loss/SRqqt1/assignment_loss‚0.521638086714845 0loss/SRqqt2/detection_loss‚0.597074665585376 /loss/FBt1/detection_loss‚1.1225954647143852 trainer/global_step‚7049 0loss/FRt1/assignment_loss‚1.4830285786150124 0loss/FRt2/assignment_loss‚0.6045705058014802 3loss/FBt2/assignment_loss‚0.0006911893309599557 /loss/FBt2/detection_loss‚0.5080453372629429 epoch‚17Ä'–p<ž¤‚(:“ ËéÒ§†ãö gpu.0.gpu‚93.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚341.161' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚68.23219999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ç1· Á¥‚(:ÿ ËéÒÀÒþ÷) disk./.usagePercent‚36.96091239078104 disk.sda.in‚0 disk.dm-1.out‚ 597.4765625 disk.rbd0.in‚0 disk.sda3.in‚0 disk.rbd3.in‚0" disk.nvme0cú?x’I0n1.in‚ 26996.484375 disk.nvme0n1.in‚ 26996.484375 disk.rbd3.out‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd1.out‚ 30.951171875 network.recv‚ 2.926401e+06 disk.dm-0.in‚ 26996.484375' disk.nvme0n1.out‚108367.46337890625( proc.memory.availableMB‚ 1030.7265625 disk.sda3.out‚ 597.4765625$ memory_percent‚93.70894432067871# proc.memory.rssMB‚ 2870.36328125 proc.cpu.threads‚17 disk.rbd0.out‚0 disk.sda1.out‚0* proc.memory.percent‚17.519307136535645 disk.dm-1.in‚0 disk.drbd2165.in‚0 disk.drbd2165.out‚0) disk.nvme0c0n1.out‚108367.46337890625$ disk./.usageGB‚550.7443046569824 disk.sda.out‚ 597.4765625$ disk.dm-0.out‚108367.46337890625 disk.rbd1.in‚ 0.47314453125 network.sent‚ 1.3450468e+07 disk.sda1.in‚0 cpu‚25.536523171526213ž©\Ÿ¦‚(:” ÚéÒÛœŒ÷ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚296.032' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚59.206399999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0S*ª§‚(:ý ÚéÒ³Ô÷ù network.sent‚ 1.3513401e+07& disk.nvme0n1.out‚108379.1396484375 disk.sda.out‚ 597.83203125) proc.memory.availableMB‚ 1030.91796875) disk./.usagePercent‚36.96058802923516 disk.rbd3.out‚0 disk.rbd1.out‚ 31.013671875 disk.dm-1.in‚0 disk.drbd2165.out‚0! disk.nvme0c0n1.in‚ 26996.90625 disk.sda2.in‚0 disk.rbd0.out‚0# disk.dm-0.out‚108379.1396484375( disk.nvme0c0n1.out‚108379.1396484375 disk.sda2.out‚0 disk.rbd1.in‚ 0.47314453125$ memory_percent‚93.70777606964111 network.recv‚ 2.930759e+06 disk.drbd2165.in‚0 disk.nvme0n1.in‚ 26996.90625 proc.cpu.threads‚17 disk.rbd3.in‚0 disk.rbd0.in‚0 cpu‚25.536297313966966 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda3.out‚ 597.83203125# proc.memory.rssMB‚ 2870.36328125* proc.memory.percent‚17.519307136535645 disk.dm-1.out‚ 597.83203125$ disk./.usageGB‚550.7394714355469 disk.dm-0.in‚ 26996.90625 disk.sda3.in‚0 disk.sda.in‚0ƒtŸ¤ž¨‚(:“ ééÒöÏö gpu.0.gpu‚37.0 gpu.0.memory‚12, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚339.941' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚67.98819999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0gn­[ñ©‚(:æ ééÒš¡×ù disk.nvme0c0n1.in‚ 26997.125 disk.sda2.out‚0 disk.rbd0.in‚0# memory_percent‚93.7082290649414 disk.sda1.in‚0 disk.rbd3.in‚0( disk.nvme0c0n1.out‚108393.9580078125 disk.nvme0n1.in‚ 26997.125& disk.nvme0n1.out‚108393.9580078125# proc.memory.rssMB‚ 2870.36328125 network.recv‚ 2.934132e+06# disk.dm-0.out‚108393.9580078125 disk.drbd2165.in‚0 disk.drbd2165.out‚0 cpu‚25.536124516005668 disk.sda1.out‚0 disk.sda3.in‚0 disk.rbd1.out‚ 31.013671875& proc.memory.availableMB‚ 1030.84375* proc.memory.percent‚17.519307136535645 proc.cpu.threads‚17 disk.sda3.out‚598.0625 disk.rbd0.out‚0 network.sent‚ 1.351847e+07 disk.sda2.in‚0 disk.sda.out‚598.0625 disk.dm-1.in‚0$ disk./.usageGB‚550.7425003051758 disk.rbd3.out‚0 disk.rbd1.in‚ 0.47314453125 disk.sda.in‚0 disk.dm-1.out‚598.0625) disk./.usagePercent‚36.96079129921734 disk.dm-0.in‚ 26997.125™/šºžª‚(:“ øéÒó™Íö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚283.228' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚56.64560000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ïøéþ«‚(:ó øéÒÖ®Þù disk.dm-1.out‚600.375 disk.sda.in‚0 disk.sda.out‚600.375* proc.memory.percent‚17.519307136535645 proc.cpu.threads‚17* disk./.usagePercent‚36.961546777450884 disk.dm-1.in‚0 disk.sda1.in‚0 disk.drbd2165.in‚0 disk.sda2.out‚0 disk.dm-0.in‚ 26997.328125 disk.rbd1.in‚ 0.47314453125$ disk./.usageGB‚550.7537574768066 disk.sda3.in‚0 disk.rbd3.out‚0" disk.nvme0c0n1.in‚ 26997.328125) disk.nvme0c0n1.out‚108418.02685546875 disk.sda2.in‚0 disk.rbd0.out‚0$ disk.dm-0.out‚108418.02685546875 disk.sda1.out‚0 disk.drbd2165.out‚0 disk.rbd1.out‚ 31.091796875$ memory_percent‚93.70698928833008# proc.memory.rssMB‚ 2870.36328125 cpu‚25.535908254001086 disk.rbd0.in‚0 disk.sda3.out‚600.375 disk.rbd3.in‚0 disk.nvme0n1.in‚ 26997.328125' proc.memory.availableMB‚ 1031.046875 network.sent‚ 1.3523697e+07 network.recv‚ 2.937565e+06' disk.nvme0n1.out‚108418.02685546875OTÙž¬‚(:“ ‡éÒ—­Òö gpu.0.gpu‚99.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚329.225' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚65.84500000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‰ÑÑý­‚(:ò ‡éÒ’øÛø$ memory_percent‚93.70861053466797 disk.dm-1.in‚0 disk.rbd0.in‚0" disk.dm-0.out‚108957.193359375' disk.nvme0c0n1.out‚108957.193359375! disk.nvme0n1.in‚ 27196.6796875& proc.memory.availableMB‚ 1030.78125 disk.sda.in‚0 disk.sda1.out‚0# proc.memory.rssMB‚ 2870.36328125* proc.memory.percent‚17.519307136535645 disk.dm-0.in‚ 27196.6796875 disk.sda2.out‚0* disk./.usagePercent‚37.000505235988975 disk.rbd0.out‚0 disk.drbd2165.out‚0 disk.rbd1.in‚ 0.47314453125 network.sent‚ 1.3528745e+07 cpu‚25.535739824273538 disk.sda.out‚603.1875 disk.sda3.in‚0 disk.sda2.in‚0 proc.cpu.threads‚17 disk.dm-1.out‚603.1875 disk.sda1.in‚0 disk.sda3.out‚603.1875 disk.drbd2165.in‚0% disk.nvme0n1.out‚108957.193359375 network.recv‚ 2.941041e+06$ disk./.usageGB‚551.3342666625977 disk.rbd3.in‚0 disk.rbd3.out‚0# disk.nvme0c0n1.in‚ 27196.6796875 disk.rbd1.out‚ 31.091796875È,w 4®/ _wandbruntime‚11768  _runtime‚11768¥E€¥È¯Â & lr-AdamW/pg1‚0.0005108712919173217 & lr-AdamW/pg2‚0.0005108712919173217 trainer/global_step‚7099 % _timestamp‚1.7843015845818965e+09 _runtime‚11768.799080764 _step‚581ìÝ Í°Ç trainer/global_step‚7099 % _timestamp‚1.7843015845818965e+09 _runtime‚11768.799080764 _step‚581 & lr-AdamW/pg1‚0.0005108712919173217 & lr-AdamW/pg2‚0.0005108712919173217ÅÔ~êå4±/ _wandbruntime‚11768  _runtime‚11768#y(²þO BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2044751625383216 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6675481695744 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 trainer/global_step‚7099 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚60511073 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 _step‚582 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚61465786 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2038087565307904 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 121966880 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8611767224832 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚18211105 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚551163154083840 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2045645197515264 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2038968344256512 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚61455813 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚60511067 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8611732688384 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2045643825952256 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚559775289399808 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚18211110 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚61455813 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6664060075312 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚38353839 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚60511073 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚559774886772224 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6676827085312 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 % _timestamp‚1.7843015845829484e+09 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6665404675634 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2045645197515264 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚259 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 _runtime‚11768.800281651 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 121976859 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚60511067 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2038968370429952 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚38354098 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2044752995411506 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚61465786 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2038968370429952 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2044751625383216 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6675481695744 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 trainer/global_step‚7099 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚60511073 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 _step‚582 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚61465786 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2038087565307904 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 121966880 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8611767224832 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚18211105 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚551163154083840 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2045645197515264 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2038968344256512 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚61455813 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚60511067 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8611732688384 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2045643825952256 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚559775289399808 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚18211110 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚61455813 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6664060075312 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚38353839 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚60511073 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚559774886772224 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6676827085312 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 % _timestamp‚1.7843015845829484e+09 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6665404675634 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2045645197515264 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚259 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 _runtime‚11768.800281651 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 121976859 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚60511067 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2038968370429952 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚38354098 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2044752995411506 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322ÆL›×”4´/ _wandbruntime‚11770  _runtime‚11770Ó%Ÿ•µ‡N [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2038365447086080 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚38359144 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚551240640491520 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 $ _timestamp‚1.784301585967405e+09 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 121983564 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6676406731264 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 121983564 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚61474278 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2039246344986624 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8612970083840 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6677752128000 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 121993557 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2045031800656040 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2045924123288064 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6666328141992 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2045922751717888 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2039246371160064 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚20145496 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚255 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚38359399 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 _step‚583 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚18213653 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 trainer/global_step‚7099 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚559853576046080 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8612935554560 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2045030430627638 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚60519273 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2039246344986624 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2038365472514048 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚61464291 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚61464291 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 _step‚583 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚18213653 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 trainer/global_step‚7099 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚559853576046080 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8612935554560 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2045030430627638 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚60519273 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2039246344986624 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2038365472514048 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚61464291 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚61464291 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚38359144 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚551240640491520 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 $ _timestamp‚1.784301585967405e+09 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 121983564 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6676406731264 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 121983564 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚61474278 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2039246344986624 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8612970083840 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6677752128000 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 121993557 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2045031800656040 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2045924123288064 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6666328141992 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2045922751717888 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2039246371160064 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚20145496 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚255 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚38359399 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚60921367 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚38619764 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20281504 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20281264 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2051981679644672 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚245 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6710297188420 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6723143015936 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2052868380762112 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2058691951405124 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚61881565 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6721797626368 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 400530432 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚7149 % _timestamp‚1.7843016633274236e+09 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚61891538 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8671875451392 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚32439296 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚60921367 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚38619519 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚240 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚60921373 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2059590178388480 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8671843012096 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚563709718010368 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚563709317479936 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 122812911 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚555037842558976 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2052868380762112 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2059591549951488 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚18338255 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚555037474467840 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2058693321433414 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6711641788742 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6721797626368 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚61881565 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2051981654216704 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2059590178388480 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2052868406935552 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚18338260 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _runtime‚11847.544870912 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/overN,#p˜size_segments.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2059591549951488 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚60921373 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2052868406935552 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995ó•ªÑ (ɃP ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2058693321433414 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6711641788742 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6721797626368 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚61881565 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2051981654216704 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2059590178388480 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2052868406935552 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚18338260 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _runtime‚11847.544870912 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2059591549951488 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚60921373 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2052868406935552 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚60921367 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚38619764 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20281504 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20281264 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2051981679644672 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚245 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6710297188420 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6723143015936 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2052868380762112 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2058691951405124 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚61881565 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6721797626368 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 400530432 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚7149 % _timestamp‚1.7843016633274236e+09 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚61891538 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8671875451392 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚32439296 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚60921367 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚38619519 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚240 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚60921373 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2059590178388480 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8671843012096 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚563709718010368 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚563709317479936 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 122812911 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚555037842558976 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2052868380762112 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2059591549951488 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚18338255 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚555037474467840Ê«Ýl4Ê/ _wandbruntime‚11848  _runtime‚11848Þ 'ˇN D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚61900030 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚60929573 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚563788004722176 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 122819616 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 122829609 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 $ _timestamp‚1.784301664712282e+09 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚38624803 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2059869104154112 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 122819616 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚236 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚61890043 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6712565255100 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚11848.929652588 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6711220654666 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6724068058624 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚61890043 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚231 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2052259561422848 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚60929579 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2053146381492224 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚18340803 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 400523264 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚555115328966656 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8673076278784 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6724068058624 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8673043846656 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820·ð¶*'ÌŒN KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6712565255100 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚11848.929652588 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6711220654666 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6724068058624 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚61890043 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚231 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2052259561422848 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚60929579 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2053146381492224 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚18340803 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 400523264 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚555115328966656 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8673076278784 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6724068058624 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8673043846656 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚61900030 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚60929573 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚563788004722176 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 122819616 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 122829609 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 $ _timestamp‚1.784301664712282e+09 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚38624803 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2059869104154112 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 122819616 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚236 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚61890043Ë™¾š|4Í/ _wandbruntime‚11849  _runtime‚11849 ¡G§Î„ 0loss/FRt1/detection_loss‚0.47183024772701365 /loss/FBt1/detection_loss‚0.7473729441104375 %loss/total_loss‚7.880777734567678 % _timestamp‚1.7843016647992516e+09 0loss/FRt2/assignment_loss‚0.6751480675542477 2loss/SRqqt1/assignment_loss‚0.5091020082135974 1loss/FBt1/assignment_loss‚0.13074329712283944 2loss/FBt2/assignment_loss‚0.044327653673659276 0loss/FBt2/detection_loss‚0.48lÓóc192252982728956 epoch‚17 /loss/SRqqt2/detection_loss‚0.58763872237742 trainer/global_step‚7149 /loss/FRt2/detection_loss‚1.5077761017835036 1loss/SRqqt1/detection_loss‚0.8644061724592093 _runtime‚11849.01618262 _step‚588 0loss/FRt1/assignment_loss‚1.4592519902622505 3loss/SRqqt2/assignment_loss‚0.40125799945620916'Lxdω trainer/global_step‚7149 /loss/FRt2/detection_loss‚1.5077761017835036 1loss/SRqqt1/detection_loss‚0.8644061724592093 _runtime‚11849.01618262 _step‚588 0loss/FRt1/assignment_loss‚1.4592519902622505 3loss/SRqqt2/assignment_loss‚0.40125799945620916 0loss/FRt1/detection_loss‚0.47183024772701365 /loss/FBt1/detection_loss‚0.7473729441104375 %loss/total_loss‚7.880777734567678 % _timestamp‚1.7843016647992516e+09 0loss/FRt2/assignment_loss‚0.6751480675542477 2loss/SRqqt1/assignment_loss‚0.5091020082135974 1loss/FBt1/assignment_loss‚0.13074329712283944 2loss/FBt2/assignment_loss‚0.044327653673659276 0loss/FBt2/detection_loss‚0.48192252982728956 epoch‚17 /loss/SRqqt2/detection_loss‚0.58763872237742ÌÓ~p!ŸÐ‚(:” áéÒìÃø gpu.0.gpu‚97.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚267.508' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚53.501599999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0%A|Q Ñ‚(:ÿ áéÒ·è­ù proc.cpu.threads‚17 disk.sda3.out‚ 608.8984375 network.recv‚ 2.962393e+06( disk./.usagePercent‚37.2779743938397$ disk./.usageGB‚555.4687576293945 disk.sda1.in‚0" disk.nvme0n1.in‚27270.91015625 disk.sda.in‚0 disk.sda3.in‚0 disk.sda2.in‚0 disk.rbd1.out‚ 31.357421875 disk.dm-1.out‚ 608.8984375* proc.memory.percent‚17.519307136535645 disk.rbd3.out‚0 disk.drbd2165.out‚0 disk.rbd1.in‚ 0.47314453125 disk.dm-1.in‚0 disk.rbd0.out‚0 disk.drbd2165.in‚0 disk.dm-0.in‚27270.91015625 disk.sda1.out‚0$ memory_percent‚93.70803833007812 cpu‚25.53458170778745$ disk.nvme0c0n1.in‚27270.91015625) disk.nvme0c0n1.out‚112457.20751953125 disk.sda2.out‚0 disk.rbd0.in‚0$ proc.memory.availableMB‚1030.875 network.sent‚ 1.3616075e+07$ disk.dm-0.out‚112451.20751953125' disk.nvme0n1.out‚112457.58251953125# proc.memory.rssMB‚ 2870.36328125 disk.rbd3.in‚0 disk.sda.out‚ 608.8984375Ò‰@tŸÒ‚(:” ðéÒÎê¨÷ gpu.0.gpu‚48.0 gpu.0.memory‚15, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚38.0 gpu.0.powerWatts‚110.415' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.083000000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0™•öµ Ó‚(:‚ ðéÒüª½ù disk.rbd0.out‚0 disk.sda2.out‚0( proc.memory.availableMB‚ 1030.6171875# proc.memory.rssMB‚ 2870.36328125 cpu‚25.53441136392241 network.sent‚ 1.3678504e+07 disk.sda.out‚ 609.48046875 disk.dm-1.out‚ 609.48046875 disk.dm-0.in‚27271.12890625 disk.sda1.in‚0 disk.rbd3.in‚0& disk.nvme0n1.out‚113401.9033203125 disk.sda2.in‚0 disk.sda.in‚0 disk.rbd3.out‚0) disk./.usagePercent‚37.30828133376566# disk.dm-0.out‚113401.9033203125 disk.drbd2165.out‚0$ disk.nvme0c0n1.in‚27271.12890625# memory_percent‚93.7096118927002 network.recv‚ 2.967046e+06 disk.rbd0.in‚0* proc.memory.percent‚17.519307136535645( disk.nvme0c0n1.out‚113401.9033203125" disk.nvme0n1.in‚27271.12890625 disk.rbd1.in‚ 0.47314453125 disk.dm-1.in‚0 disk.sda3.out‚ 609.48046875 proc.cpu.threads‚17 disk.sda1.out‚0 disk.sda3.in‚0 disk.drbd2165.in‚0# disk./.usageGB‚555.920352935791 disk.rbd1.out‚ 31.435546875pHúdŸÔ‚(:” ÿéÒÈàù gpu.0.gpu‚90.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚273.925' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚54.785000000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0£-T}$Õ‚(:™ ÿéÒÁ³Ùû disk.sda.out‚ 610.06640625 disk.dm-1.out‚ 610.06640625 disk.sda3.out‚ 610.06640625 disk.drbd2165.in‚0 disk.drbd2165.out‚0$ disk./.usageGB‚558.6949043273926 disk.rbd1.in‚ 0.47314453125 disk.rbd1.out‚ 31.435546875 disk.rbd0.out‚0 disk.dm-0.in‚ 27327.9921875 disk.rbd3.in‚0$ memory_percent‚93.70908737182617 cpu‚25.53418998678412 network.sent‚ 1.3684119e+07 disk.dm-1.in‚ 0.05078125' proc.memory.availableMB‚ 1030.703125 proc.cpu.threads‚17% disk.nvme0n1.out‚113447.009765625 disk.sda.in‚ 0.05078125" disk.dm-0.out‚113447.009765625# disk.nvme0c0n1.in‚ 27327.9921875 network.recv‚ 2.970412e+06 disk.sda1.in‚0 disk.sda1.out‚0* disk./.usagePercent‚37.494483805659726 disk.sda2.out‚0 disk.sda3.in‚ 0.05078125 disk.rbd3.out‚0# proc.memory.rssMB‚ 2870.36328125* proc.memory.percent‚17.519307136535645! disk.nvme0n1.in‚ 27327.9921875 disk.rbd0.in‚0' disk.nvme0c0n1.out‚113447.009765625 disk.sda2.in‚0 ƒŽÍŸÖ‚(:” Ž‘éÒêýŒ÷ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚142.643' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚28.528599999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0i3\Ë%ׂ(:š Ž‘éÒ§Ôù disk.rbd1.in‚ 0.47314453125 disk.dm-1.out‚ 612.9765625 disk.rbd3.out‚0$ memory_percent‚93.71075630187988# proc.memory.rssMB‚ 2870.36328125 cpu‚25.534030230446497 proc.cpu.threads‚17$ disk./.usageGB‚558.8118858337402 disk.rbd0.out‚0 disk.sda.out‚ 612.9765625 network.recv‚ 2.973695e+06$ disk.nvme0c0n1.in‚27330.01953125 disk.dm-1.in‚ 0.05859375( proc.memory.availableMB‚ 1030.4296875* proc.memory.percent‚17.519307136535645 network.sent‚ 1.3688607e+07) disk./.usagePercent‚37.50233453270474 disk.drbd2165.in‚0 disk.rbd3.in‚0% disk.nvme0n1.out‚113929.228515625 disk.sda.in‚ 0.05859375 disk.dm-0.in‚27330.01953125" disk.dm-0.out‚113929.228515625 disk.sda1.out‚0 disk.sda3.in‚ 0.05859375 disk.sda3.out‚ 612.9765625 disk.rbd1.out‚ 31.591796875 disk.rbd0.in‚0 disk.drbd2165.out‚0' disk.nvme0c0n1.out‚113929.228515625" disk.nvme0n1.in‚27330.01953125 disk.sda2.in‚0 disk.sda1.in‚0 disk.sda2.out‚0Þ “ŸØ‚(:” ‘éÒ­¿÷ø gpu.0.gpu‚94.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚295.399' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚59.079800000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÈjV‚#Ù‚(:˜ ‘éÒ’Áöú disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.rbd3.out‚0 disk.nvme0n1.in‚ 27334.109375 disk.rbd1.out‚ 31.591796875 disk.dm-1.out‚ 615.140625 disk.rbd0.in‚0 disk.sda1.out‚0 disk.sda.out‚ 615.140625) proc.memory.availableMB‚ 1030.67578125* proc.memory.percent‚17.519307136535645 disk.sda3.out‚ 615.140625 network.sent‚ 1.3694349e+07 network.recv‚ 2.977405e+06* disk./.usagePercent‚37.586144743219116 disk.sda1.in‚0# proc.memory.rssMB‚ 2870.36328125# disk./.usageGB‚560.060718536377 disk.rbd3.in‚0& disk.nvme0n1.out‚118158.3544921875 disk.rbd1.in‚ 0.47314453125 disk.sda.in‚ 44.93359375 disk.rbd0.out‚0$ memory_percent‚93.70925426483154" disk.nvme0c0n1.in‚ 27334.109375( disk.nvme0c0n1.out‚118158.3544921875 proc.cpu.threads‚17 disk.sda2.out‚0 disk.dm-0.in‚ 27334.109375 disk.sda2.in‚0 disk.dm-1.in‚ 44.93359375# disk.dm-0.out‚118158.3544921875 disk.sda3.in‚ 44.93359375 cpu‚25.533814250413364 DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚257 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6768113556992 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 trainer/global_step‚7199 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚62307317 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20420120 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2073537902387712 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 123648963 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8732065332736 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2066768443441152 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2066768417267712 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6757878901850 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6769458946560 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚262 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚38885525 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8732030796288 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 % _timestamp‚1.7843017421744773e+09 _runtime‚11926.391872369 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚61331667 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2073536530824704 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6768113556992 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 trainer/global_step‚7199 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚62307317 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20420120 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2073537902387712 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 123648963 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8732065332736 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2066768443441152 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2066768417267712 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6757878901850 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6769458946560 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚262 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚38885525 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8732030796288 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 % _timestamp‚1.7843017421744773e+09 _runtime‚11926.391872369 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚61331667 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2073536530824704 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚257 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 JDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 123655668 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.alloƒ¼ËC¤ cated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2067046417997824 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2073815456590336 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 123665661 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚61339873 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 $ _timestamp‚1.784301743560413e+09 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2072912452699856 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2067046444171264 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚38890810 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚248 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚558989281259520 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚61339879 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚18467948ÁŽ¤¨'äŒN H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 123665661 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚61339873 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2067046417997824 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2073815456590336 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 123665661 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚61339873 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 $ _timestamp‚1.784301743560413e+09 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2072912452699856 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2067046444171264 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚38890810 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚248 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚558989281259520 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚61339879 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚18467948 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 123655668 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 _runtime‚11927.777853262 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2066153624903680 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6758802368208 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚18467953 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 123655668 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 F&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5246156454086304 6 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.248324915766716 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.13354231417179108 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.7445244789123535 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 6 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.638978123664856 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.13778187334537506 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.2371358871459961 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 jet/accuracy_5_of_5‚0 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.47542253136634827 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6281780004501343 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.5954414010047913 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2598365247249603 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5566509962081909 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8710305094718933 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8727134466171265 8"Purity/1FRt0SRqqt0FBt/event_purity‚0.248324915766716 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.0555555559694767 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6371612548828125 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 /particle/accuracy_4_of_4‚0.9467383027076721 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8796674013137817 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 +jet/accuracy_3_of_4‚0.10253318399190903 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3072916865348816 particle/accuracy_4_of_5‚1 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6149489283561707 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.3248113989830017 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9994879364967346 *jet/accuracy_1_of_3‚0.9125086665153503 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.9375 epoch‚17 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.23256894946098328 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5470083951950073 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.2347654402256012 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3299300968647003 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.32375454902648926 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9888684153556824 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9960675239562988 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7976568341255188 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 *jet/accuracy_1_of_2‚0.7705140113830566 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 particle/accuracy_2_of_3‚1 (particle/accuracy‚0.8455092906951904 6validation_average_jet_accuracy‚0.3527488708496094 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7527272701263428 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9015350341796875 8"Purity/*FRt0SRqqt2FBt/event_purity‚0.949863612651825 ( Purity/2FRt1SRqqt1FBt/FRt_purity‚0.5 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.044750623404979706 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2893780767917633 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.2395339012145996 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9814814329147339 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.18723024427890778 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5694260001182556 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.14474140107631683 6 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.889191210269928 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9992997050285339 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.5475642085075378 , Purity/0FRt2SRqqt1FBt/FBt_purity‚0.96875 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2498835176229477 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6347247958183289 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9988892674446106 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.13583333790302277 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 +particle/sensitivity‚0.6193710565567017 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.03846153989434242 jet/accuracy_4_of_5‚0 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.6602426171302795 *jet/accuracy_2_of_4‚0.7716466784477234 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8009259104728699 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 particle/accuracy_1_of_5‚1 particle/accuracy_5_of_5‚1 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.23790451884269714 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5849172472953796 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.13712409138679504 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.6641327738761902 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.23240673542022705 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6377664804458618 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.5547417402267456 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 & Purity/2FRt0SRqqt2FBt/FRt_purity‚1 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9780850410461426 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.5866312980651855 +jet/accuracy_4_of_4‚0.01908157207071781 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 6 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.998015820980072 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.6822515726089478 *jet/accuracy_2_of_3‚0.5201858878135681 particle/accuracy_1_of_3‚1 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9958112835884094 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9890425801277161 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.30544349551200867 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.062438271939754486 *jet/accuracy_1_of_4‚0.9706571698188782 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.6336446404457092 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5505571961402893 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.043828368186950684 % _timestamp‚1.7843017697022638e+09 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6774798631668091 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9987789392471313 8"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.637751579284668 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6431251168251038 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03303461894392967 +jet/accuracy_3_of_3‚0.07638553529977798 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.0555555559694767 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2601209282875061 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.13354231417179108 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.2472737580537796 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.05000000074505806 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.5912485718727112 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 +jet/accuracy_2_of_2‚0.14000259339809418 particle/accuracy_2_of_2‚1 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.22529233992099762 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.5937654376029968 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8875840306282043 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.13728097081184387 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9952563047409058 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6302660703659058 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 +jet/accuracy_1_of_1‚0.30843740701675415 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 particle/accuracy_1_of_2‚1 'particle/f_score‚0.6326031684875488 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5811187624931335 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7156789898872375 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6395623087882996 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.4968721270561218 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9807692170143127 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6352474689483643 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9814814329147339 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8748109936714172 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 particle/accuracy_1_of_4‚1 8 Purity/*FŠˆWlÙRt1SRqqt*FBt/FRt_purity‚0.24506217241287231 , Purity/0FRt2SRqqt*FBt/FBt_purity‚0.96875 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.25593677163124084 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4613332152366638 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.48683157563209534 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.6851230263710022 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6276041269302368 particle/accuracy_3_of_5‚1 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 /particle/accuracy_3_of_4‚0.9965986609458923 jet/accuracy_3_of_5‚1 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9964557886123657 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7421191930770874 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.5811999440193176 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2729833722114563 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.30742940306663513 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.5893455147743225 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9205806255340576 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 _step‚593 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 6 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.996676504611969 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.19789722561836243 particle/accuracy_2_of_4‚1 jet/accuracy_1_of_5‚1 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5470083951950073 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8748109936714172 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.23856723308563232 particle/accuracy_2_of_5‚1 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9722222089767456 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9992997050285339 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.03846153989434242 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.24983017146587372 particle/accuracy_3_of_3‚1 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5676901340484619 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3077884316444397 6 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.998429000377655 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4296875 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.032207801938056946 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8009259104728699 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 +particle/specificity‚0.9073574542999268 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.3343014419078827 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.05000000074505806 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5616666078567505 ("Purity/2FRt0SRqqt2FBt/event_purity‚1 trainer/global_step‚7199 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.2689407467842102 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9964489936828613 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.13664504885673523 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.24942442774772644 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.875 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9904571771621704 9"Purity/*FRt2SRqqt0FBt/event_purity‚0.1474658101797104 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6263697743415833 _runtime‚11953.920198842 (loss/val_total_loss‚7.89100147855135 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.2599668502807617 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 particle/accuracy_1_of_1‚1 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 jet/accuracy_2_of_5‚1 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6512223482131958 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.7445244789123535 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837{èƳ=ì­{ :"Purity/*FRt*SRqqt*FBt/event_purity‚0.30742940306663513 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.5893455147743225 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9205806255340576 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 _step‚593 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 6 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.996676504611969 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.19789722561836243 particle/accuracy_2_of_4‚1 jet/accuracy_1_of_5‚1 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5470083951950073 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8748109936714172 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.23856723308563232 particle/accuracy_2_of_5‚1 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9722222089767456 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9992997050285339 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.03846153989434242 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.24983017146587372 particle/accuracy_3_of_3‚1 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5676901340484619 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3077884316444397 6 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.998429000377655 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4296875 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.032207801938056946 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8009259104728699 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 +particle/specificity‚0.9073574542999268 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.3343014419078827 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.05000000074505806 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5616666078567505 ("Purity/2FRt0SRqqt2FBt/event_purity‚1 trainer/global_step‚7199 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.2689407467842102 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9964489936828613 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.13664504885673523 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.24942442774772644 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.875 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9904571771621704 9"Purity/*FRt2SRqqt0FBt/event_purity‚0.1474658101797104 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6263697743415833 _runtime‚11953.920198842 (loss/val_total_loss‚7.89100147855135 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.2599668502807617 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 particle/accuracy_1_of_1‚1 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 jet/accuracy_2_of_5‚1 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6512223482131958 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.7445244789123535 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9891248345375061 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.42129629850387573 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.02278815023601055 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5246156454086304 6 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.248324915766716 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.13354231417179108 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.7445244789123535 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 6 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.638978123664856 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.13778187334537506 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.2371358871459961 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 jet/accuracy_5_of_5‚0 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.47542253136634827 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6281780004501343 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.5954414010047913 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2598365247249603 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5566509962081909 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8710305094718933 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8727134466171265 8"Purity/1FRt0SRqqt0FBt/event_purity‚0.248324915766716 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.0555555559694767 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6371612548828125 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 /particle/accuracy_4_of_4‚0.9467383027076721 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8796674013137817 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 +jet/accuracy_3_of_4‚0.10253318399190903 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3072916865348816 particle/accuracy_4_of_5‚1 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6149489283561707 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.3248113989830017 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9994879364967346 *jet/accuracy_1_of_3‚0.9125086665153503 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.9375 epoch‚17 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.23256894946098328 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5470083951950073 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.2347654402256012 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3299300968647003 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.32375454902648926 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9888684153556824 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9960675239562988 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7976568341255188 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 *jet/accuracy_1_of_2‚0.7705140113830566 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 particle/accuracy_2_of_3‚1 (particle/accuracy‚0.8455092906951904 6validation_average_jet_accuracy‚0.3527488708496094 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7527272701263428 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9015350341796875 8"Purity/*FRt0SRqqt2FBt/event_purity‚0.949863612651825 ( Purity/2FRt1SRqqt1FBt/FRt_purity‚0.5 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.044750623404979706 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2893780767917633 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.2395339012145996 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9814814329147339 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.18723024427890778 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5694260001182556 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.14474140107631683 6 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.889191210269928 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9992997050285339 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.5475642085075378 , Purity/0FRt2SRqqt1FBt/FBt_purity‚0.96875 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2498835176229477 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6347247958183289 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9988892674446106 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.13583333790302277 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 +particle/sensitivity‚0.6193710565567017 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.03846153989434242 jet/accuracy_4_of_5‚0 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.6602426171302795 *jet/accuracy_2_of_4‚0.7716466784477234 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8009259104728699 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 particle/accuracy_1_of_5‚1 particle/accuracy_5_of_5‚1 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.23790451884269714 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5849172472953796 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.13712409138679504 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.6641327738761902 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.23240673542022705 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6377664804458618 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.5547417402267456 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 & Purity/2FRt0SRqqt2FBt/FRt_purity‚1 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9780850410461426 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.5866312980651855 +jet/accuracy_4_of_4‚0.01908157207071781 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 6 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.998015820980072 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.6822515726089478 *jet/accuracy_2_of_3‚0.5201858878135681 particle/accuracy_1_of_3‚1 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9958112835884094 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9890425801277161 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.30544349551200867 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.062438271939754486 *jet/accuracy_1_of_4‚0.9706571698188782 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.6336446404457092 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5505571961402893 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.043828368186950684 % _timestamp‚1.7843017697022638e+09 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6774798631668091 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9987789392471313 8"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.637751579284668 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6431251168251038 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03303461894392967 +jet/accuracy_3_of_3‚0.07638553529977798 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.0555555559694767 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2601209282875061 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.13354231417179108 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.2472737580537796 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.05000000074505806 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.5912485718727112 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 +jet/accuracy_2_of_2‚0.14000259339809418 particle/accuracy_2_of_2‚1 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.22529233992099762 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.5937654376029968 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8875840306282043 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.13728097081184387 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9952563047409058 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6302660703659058 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 +jet/accuracy_1_of_1‚0.30843740701675415 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 particle/accuracy_1_of_2‚1 'particle/f_score‚0.6326031684875488 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5811187624931335 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7156789898872375 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6395623087882996 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.4968721270561218 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9807692170143127 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6352474689483643 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9814814329147339 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8748109936714172 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 particle/accuracy_1_of_4‚1 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.24506217241287231 , Purity/0FRt2SRqqt*FBt/FBt_purity‚0.96875 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.25593677163124084 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4613332152366638 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.48683157563209534 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.6851230263710022 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6276041269302368 particle/accuracy_3_of_5‚1 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 /particle/accuracy_3_of_4‚0.9965986609458923 jet/accuracy_3_of_5‚1 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9964557886123657 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7421191930770874 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.5811999440193176 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2729833722114563Ñ£gÞÆ”í‚(:‰ Ê‘éÒ‘¢àö gpu.0.gpu‚92.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚259.878' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚51.9756 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0q`þ(î‚(: Ê‘éÒ…¨ú network.sent‚ 1.3767558e+07 disk.rbd3.in‚0# disk.nvme0c0n1.in‚ 27428.8046875! disk.nvme0n1.in‚ 27428.8046875' disk.nvme0n1.out‚118195.43505859375* disk./.usagePercent‚37.583724703997945# disk./.usageGB‚560.024658203125 disk.rbd0.in‚0 disk.sda3.in‚ 45.046875 disk.rbd1.out‚ 31.794921875 disk.drbd2165.out‚0 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.rbd3.out‚0* proc.memory.percent‚17.519307136535645 disk.sda2.out‚0 disk.rbd1.in‚ 0.47314453125$ disk.dm-0.out‚118195.43505859375 disk.sda3.out‚ 618.16796875) disk.nvme0c0n1.out‚118195.43505859375 disk.sda2.in‚0$ memory_percent‚93.70694160461426 proc.cpu.threads‚17 disk.rbd0.out‚0 disk.dm-0.in‚ 27428.8046875 disk.sda1.in‚0( proc.memory.availableMB‚ 1031.0546875# proc.memory.rssMB‚ 2870.36328125 network.recv‚ 2.988998e+06 disk.sda.in‚ 45.046875 disk.sda.out‚ 618.16796875 disk.dm-1.in‚ 45.046875 disk.dm-1.out‚ 618.16796875 cpu‚25.539116806864982þ´k;Ÿï‚(:” Ù‘éÒÊûÜö gpu.0.gpu‚90.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚110.286' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.057199999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ep»\!ð‚(:– Ù‘éÒ†Òéø disk.rbd0.out‚0 disk.drbd2165.in‚0 disk.dm-1.out‚ 620.76171875$ memory_percent‚93.70698928833008 disk.dm-0.in‚27430.12109375" disk.nvme0n1.in‚27430.12109375 disk.dm-1.in‚ 61.19140625* proc.memory.percent‚17.519307136535645 disk.dm-0.out‚ 118221.484375 disk.rbd3.out‚0$ disk.nvme0c0n1.in‚27430.12109375 network.sent‚ 1.3824981e+07 disk.drbd2165.out‚0$ disk.nvme0c0n1.out‚ 118221.484375 disk.rbd1.out‚ 31.888671875" disk.nvme0n1.out‚ 118221.484375 disk.rbd1.in‚ 0.47314453125' proc.memory.availableMB‚ 1031.046875 disk.sda1.in‚0# proc.memory.rssMB‚ 2870.36328125 cpu‚25.538928184356994) disk./.usagePercent‚37.58290778395618 disk.sda3.in‚ 61.19140625 disk.sda3.out‚ 620.76171875 disk.rbd3.in‚0 disk.sda2.in‚0 disk.sda.in‚ 61.19140625 network.recv‚ 2.993658e+06$ disk./.usageGB‚560.0124855041504 disk.rbd0.in‚0 disk.sda1.out‚0 disk.sda2.out‚0 disk.sda.out‚ 620.76171875 proc.cpu.threads‚17§Þº”ñ‚(:‰ è‘éÒ¬åƒ÷ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚265.843' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.1686 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0·Nv‰)ò‚(:ž è‘éÒØèÍù disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd1.out‚ 32.029296875 disk.rbd0.out‚0 disk.sda.out‚ 622.09375$ memory_percent‚93.70596408843994) proc.memory.availableMB‚ 1031.21484375) disk./.usagePercent‚37.48364291061478 disk.drbd2165.out‚0$ disk.nvme0c0n1.in‚27430.32421875" disk.nvme0n1.in‚27430.32421875 disk.sda3.out‚ 622.09375 disk.drbd2165.in‚0# proc.memory.rssMB‚ 2870.36328125 network.sent‚ 1.3830042e+07 disk.dm-1.in‚ 61.19140625 cpu‚25.538709524640545) disk.nvme0c0n1.out‚118234.76806640625' disk.nvme0n1.out‚118234.76806640625* proc.memory.percent‚17.519307136535645 proc.cpu.threads‚17 disk.rbd1.in‚ 0.47314453125 network.recv‚ 2.996951e+06$ disk./.usageGB‚558.5333671569824 disk.dm-1.out‚ 622.09375 disk.rbd3.out‚0 disk.sda1.in‚0 disk.rbd3.in‚0 disk.sda.in‚ 61.19140625$ disk.dm-0.out‚118234.76806640625 disk.sda1.out‚0 disk.sda3.in‚ 61.19140625 disk.rbd0.in‚0 disk.dm-0.in‚27430.324218754[  ”ó‚(:‰ ÷‘éÒÔ¿Óö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚46.0 gpu.0.powerWatts‚331.878' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚66.3756 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 )T)ô‚(:ž ÷‘éÒôõÔø disk.rbd3.out‚0! disk.nvme0n1.in‚ 27481.9140625 disk.rbd1.out‚ 32.029296875 disk.sda3.out‚ 622.703125* disk./.usagePercent‚37.483889957885324$ memory_percent‚93.70465278625488 disk.sda3.in‚ 61.19140625 disk.rbd3.in‚0 disk.rbd1.in‚ 0.47314453125 disk.sda.in‚ 61.19140625 cpu‚25.538561479962265# disk.nvme0c0n1.in‚ 27481.9140625 disk.dm-1.out‚ 622.703125( proc.memory.availableMB‚ 1031.4296875# proc.memory.rssMB‚ 2870.36328125$ disk./.usageGB‚558.5370483398438 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-1.in‚ 61.19140625 disk.drbd2165.in‚0 network.sent‚ 1.3835125e+07 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda1.in‚0 disk.sda1.out‚0* proc.memory.percent‚17.519307136535645 proc.cpu.threads‚17$ disk.dm-0.out‚118253.74169921875) disk.nvme0c0n1.out‚118253.74169921875' disk.nvme0n1.out‚118253.74169921875 disk.sda.out‚ 622.703125 disk.drbd2165.out‚0 network.recv‚ 3.000594e+06 disk.dm-0.in‚ 27481.9140625š—€(Ÿõ‚(:” †’éÒ—ãƒ÷ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚277.903' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚55.580600000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ìEjK,ö‚(:¡ †’éÒšÝñù( disk./.usagePercent‚37.4346446046064# disk.nvme0c0n1.in‚ 27482.1328125 disk.sda.in‚ 61.19140625 disk.sda1.in‚0 disk.sda3.in‚ 61.19140625 cpu‚25.538321074556748 proc.cpu.threads‚17 disk.sda2.in‚0 disk.drbd2165.in‚0 disk.rbd3.out‚0$ disk./.usageGB‚557.8032569885254' disk.nvme0n1.out‚118266.17236328125 disk.rbd1.in‚ 0.47314453125 disk.rbd1.out‚ 32.091796875 disk.rbd3.in‚0 disk.sda.out‚ 625.08203125# proc.memory.rssMB‚ 2870.36328125) disk.nvme0c0n1.out‚118266.17236328125 disk.rbd0.in‚0 disk.sda1.out‚0 disk.sda3.out‚ 625.08203125 network.sent‚ 1.3840219e+07 disk.rbd0.out‚0 disk.drbd2165.out‚0# memory_percent‚93.7049388885498* proc.memory.percent‚17.519307136535645 disk.dm-1.in‚ 61.19140625 disk.dm-1.out‚ 625.08203125 disk.dm-0.in‚ 27482.1328125$ disk.dm-0.out‚118266.17236328125 network.recv‚ 3.004148e+06! disk.nvme0n1.in‚ 27482.1328125 disk.sda2.out‚0( proc.memory.availableMB‚ 1031.3828125Ñ&Ͳž÷‚(:“ •’éÒØšñö gpu.0.gpu‚92.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚46.0 gpu.0.powerWatts‚345.878' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚69.17559999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0lwÍ)ø‚(:ž •’éÒèäèù network.recv‚ 3.007678e+06 disk.rbd0.out‚0 disk.dm-0.in‚ 27483.7890625 disk.sda3.in‚ 74.4609375 disk.rbd1.out‚ 32.091796875 disk.sda.in‚ 74.4609375 disk.dm-1.in‚ 74.4609375# disk.dm-0.out‚118278.8447265625# proc.memory.rssMB‚ 2870.36328125 cpu‚25.538170062657063# disk.nvme0c0n1.in‚ 27483.7890625& disk.nvme0n1.out‚118278.8447265625 disk.sda2.out‚0 disk.dm-1.out‚ 625.55859375 disk.drbd2165.in‚0 disk.drbd2165.out‚0* proc.memory.percent‚17.519307136535645$ disk./.usageGB‚557.7566223144531 disk.rbd0.in‚0$ memory_percent‚93.70605945587158 proc.cpu.threads‚17) disk./.usagePercent‚37.43151491250031 disk.sda2.in‚0 disk.sda1.out‚0 disk.rbd3.in‚0! disk.nvme0n1.in‚ 27483.7890625 disk.rbd1.in‚ 0.47314453125) proc.memory.availableMB‚ 1031.19921875 disk.rbd3.out‚0( disk.nvme0c0n1.out‚118278.8447265625 disk.sda.out‚ 625.55859375 disk.sda1.in‚0 disk.sda3.out‚ 625.55859375 network.sent‚ 1.3845379e+07ÁC/™4ù/ _wandbruntime‚12031  _runtime‚12031ÆØO Èú % _timestamp‚1.7843018471930416e+09 _runtime‚12031.410227376 _step‚594 & lr-AdamW/pg1‚0.0005035137296964123 & lr-AdamW/pg2‚0.0005035137296964123 trainer/global_step‚7249dÎ7ÍûÇ _step‚594 & lr-AdamW/pg1‚0.0005035137296964123 & lr-AdamW/pg2‚0.0005035137296964123 trainer/global_step‚7249 % _timestamp‚1.7843018471930416e+09 _runtime‚12031.410227376ÒœrÍ4ü/ _wandbruntime‚12031  _runtime‚12031æ̧j ýýO \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚32439296 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2090405711541275 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6820272099328 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚237 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚571633847382528 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚562834227482624 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 124676003 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2091318078870528 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6809945146395 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2084496435208192 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6808600546073 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 400530432 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4614144¤´âÊ™8192 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚571634247912960 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 trainer/global_step‚7249 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚62802024 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2084496435208192 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8799652339200 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2091316707307520 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6821617488896 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6820272099328 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚62792051 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2091316707307520 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2084496461381632 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2084496461381632 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20587711 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 _runtime‚12031.414944342 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 _step‚595 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚61873979 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2091318078870528 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6821617488896 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚61873973 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2090404341512985 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2083595766394880 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚62792051 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚18601478 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚242 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚562834595573760 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2091316707307520 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6821617488896 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6820272099328 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚62792051 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2091316707307520 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2084496461381632 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2084496461381632 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20587711 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 _runtime‚12031.414944342 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 _step‚595 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚61873979 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2091318078870528 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6821617488896 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚61873973 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2090404341512985 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2083595766394880 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚62792051 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚18601478 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚242 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚562834595573760 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6808600546073 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 400530432 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚571634247912960 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322Ó1³-ª4ÿ/ _wandbruntime‚12032  _runtime‚12032ý·g˜ '€‡N LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚62800529 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚32432128 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚39194463 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 trainer/global_step‚7249 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚61882179 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚39194234 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6810868612753 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6809524012319 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚61882179 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚20590218 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 124692701 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚62810516 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚18604021 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚571712550894080 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2091597004643328 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2090684516785809 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2084774435938304 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 124692701 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚62810516 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2084774462111744 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚562912081981440 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2083873648173056 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 _runtime‚12032.798850321 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2083873622745088 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚224 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚596 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚61882185 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2090683146757407 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚571712951417344 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 400523264 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 $ _timestamp‚1.784301848581419e+09 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2091597004643328 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6822542531584 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚62800529 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6821197134848 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚61882185 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚20590442 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8800869435904 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2091595633073152FµBûŒN J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚62800529 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6821197134848 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚61882185 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚20590442 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8800869435904 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2091595633073152 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚62800529 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚32432128 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚39194463 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 trainer/global_step‚7249 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚61882179 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚39194234 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6810868612753 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6809524012319 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚61882179 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚20590218 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 124692701 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚62810516 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚18604021 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚571712550894080 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2091597004643328 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2090684516785809 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2084774435938304 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 124692701 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚62810516 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2084774462111744 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚562912081981440 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2083873648173056 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 _runtime‚12032.798850321 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2083873622745088 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚224 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚596 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚61882185 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2090683146757407 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚571712951417344 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 400523264 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 $ _timestamp‚1.784301848581419e+09 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2091597004643328 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6822542531584 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175Ô¡µô4‚/ _wandbruntime‚12032  _runtime‚12032_×® ƒ„ 1loss/SRqqt1/detection_loss‚0.8243710518666795 /loss/FBt1/detection_loss‚0.5226446720900841 /loss/FBt2/detection_loss‚0.4555554552944606 %loss/total_loss‚7.493510471162018 _step‚597 0loss/FRt1/assignment_loss‚1.5213735403024335 0loss/FRt2/assignment_loss‚0.6260804228148381 2loss/SRqqt2/assignment_loss‚0.4222277201079344 epoch‚18 _runtime‚12032.882578808 2loss/SRqqt1/assignment_loss‚0.4853895071420445 1loss/FBt1/assignment_loss‚0.05151540676307123 1loss/SRqqt2/detection_loss‚0.6062980537772974 2loss/FBt2/assignment_loss‚0.011871625630493274 /loss/FRt1/detection_loss‚0.5127333295260782 trainer/global_step‚7249 % _timestamp‚1.7843018486656153e+09 /loss/FRt2/detection_loss‚1.4534496858466035•„§„‰ /loss/FRt2/detection_loss‚1.4534496858466035 1loss/SRqqt1/detection_loss‚0.8243710518666795 /loss/FBt1/detection_loss‚0.5226446720900841 /loss/FBt2/detection_loss‚0.4555554552944606 %loss/total_loss‚7.493510471162018 _step‚597 0loss/FRt1/assignment_loss‚1.5213735403024335 0loss/FRt2/assignment_loss‚0.6260804228148381 2loss/SRqqt2/assignment_loss‚0.4222277201079344 epoch‚18 _runtime‚12032.882578808 2loss/SRqqt1/assignment_loss‚0.4853895071420445 1loss/FBt1/assignment_loss‚0.05151540676307123 1loss/SRqqt2/detection_loss‚0.6062980537772974 2loss/FBt2/assignment_loss‚0.011871625630493274 /loss/FRt1/detection_loss‚0.5127333295260782 trainer/global_step‚7249 % _timestamp‚1.7843018486656153e+09ÕšMýp”…‚(:‰ ¤’éÒØËãö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚291.971' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚58.3942 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0VÅ &†‚(:› ¤’éÒ°¤ù disk.sda.in‚ 74.4609375 disk.dm-1.in‚ 74.4609375 disk.drbd2165.out‚0) disk.nvme0c0n1.out‚118293.06103515625 proc.cpu.threads‚17# disk.nvme0c0n1.in‚ 27483.9921875 disk.sda2.in‚0$ disk.dm-0.out‚118293.06103515625 disk.dm-1.out‚ 626.328125 disk.rbd0.in‚0) disk./.usagePercent‚37.43104027453183 disk.rbd3.out‚0' disk.nvme0n1.out‚118293.06103515625# proc.memory.rssMB‚ 2870.36328125 cpu‚25.537959197876624 network.recv‚ 3.010838e+06 disk.rbd1.in‚ 0.47314453125 disk.sda1.out‚0 disk.sda3.in‚ 74.4609375 disk.dm-0.in‚ 27483.9921875! disk.nvme0n1.in‚ 27483.9921875 disk.rbd0.out‚0 network.sent‚ 1.3850488e+07 disk.rbd1.out‚ 32.169921875 disk.sda1.in‚0 disk.sda3.out‚ 626.328125) proc.memory.availableMB‚ 1030.60546875 disk.drbd2165.in‚0$ memory_percent‚93.70968341827393$ disk./.usageGB‚557.7495498657227 disk.sda.out‚ 626.328125 disk.rbd3.in‚0 disk.sda2.out‚0* proc.memory.percent‚17.519307136535645mÇ””‡‚(:‰ ³’éÒíÇâö gpu.0.gpu‚93.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚46.0 gpu.0.powerWatts‚337.528' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚67.5056 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0g´x,ˆ‚(:“ ³’éÒ®ò„ù disk.drbd2165.out‚0 disk.rbd3.out‚0) disk.nvme0c0n1.out‚118310.31982421875 cpu‚25.537768344731962 disk.dm-0.in‚ 27550.4375 disk.sda3.in‚ 74.4609375 disk.sda3.out‚ 628.3359375 disk.drbd2165.in‚0# memory_percent‚93.7126636505127 disk.sda2.out‚0 network.sent‚ 1.3913621e+07) disk./.usagePercent‚37.43022028239966 disk.sda.out‚ 628.3359375 disk.nvme0n1.in‚ 27550.4375 disk.sda2.in‚0# proc.memory.rssMB‚ 2870.36328125$ disk.dm-0.out‚118310.31982421875 disk.rbd1.out‚ 32.169921875 disk.dm-1.in‚ 74.4609375 disk.rbd0.in‚0' disk.nvme0n1.out‚118310.31982421875( proc.memory.availableMB‚ 1030.1171875* proc.memory.percent‚17.519307136535645$ disk./.usageGB‚557.7373313903809 disk.nvme0c0n1.in‚ 27550.4375 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd0.out‚0 disk.sda.in‚ 74.4609375 disk.rbd3.in‚0 disk.rbd1.in‚ 0.47314453125 proc.cpu.threads‚17 network.recv‚ 3.015623e+06 disk.dm-1.out‚ 628.3359375n! ŽŸ‰‚(:” Â’éÒ™„èö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚295.399' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚59.079800000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¸jô5"Š‚(:— Â’éÒ ¬²ù disk.drbd2165.in‚0 disk.rbd3.in‚0 disk.rbd0.out‚0 disk.sda3.out‚ 628.9453125# proc.memory.rssMB‚ 2870.36328125$ disk./.usageGB‚557.7440605163574" disk.nvme0c0n1.in‚ 27550.859375 disk.nvme0n1.in‚ 27550.859375 disk.rbd1.in‚ 0.47314453125 cpu‚25.53765792974599) disk./.usagePercent‚37.43067187969006 disk.drbd2165.out‚0$ memory_percent‚93.71135234832764 disk.sda3.in‚ 74.4609375 network.recv‚ 3.018329e+06 disk.rbd3.out‚0( disk.nvme0c0n1.out‚118322.6943359375 disk.sda2.in‚0 disk.dm-1.in‚ 74.4609375 disk.dm-1.out‚ 628.9453125 disk.rbd1.out‚ 32.232421875 disk.sda1.in‚0* proc.memory.percent‚17.519307136535645 proc.cpu.threads‚17 network.sent‚ 1.3918604e+07 disk.sda.in‚ 74.4609375# disk.dm-0.out‚118322.6943359375& disk.nvme0n1.out‚118322.6943359375 disk.sda2.out‚0 disk.rbd0.in‚0 disk.dm-0.in‚ 27550.859375 disk.sda.out‚ 628.9453125 disk.sda1.out‚0) proc.memory.availableMB‚ 1030.33203125Ê{¬ù‹‚(:’ Ñ’éÒÆè÷ gpu.0.gpu‚1.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚38.0 gpu.0.powerWatts‚110.286' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.057199999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ä=88*Œ‚(:Ÿ Ñ’éÒþÑÇù network.recv‚ 3.022014e+06 disk.dm-1.out‚ 629.6015625 disk.sda3.in‚ 74.8359375# disk.nvme0c0n1.in‚ 27551.0703125$ memory_percent‚93.71130466461182) proc.memory.availableMB‚ 1030.33984375 cpu‚25.537920138970993! disk.nvme0n1.in‚ 27551.0703125 disk.rbd1.in‚ 0.47314453125 disk.sda.in‚ 74.8359375 disk.rbd0.out‚0 disk.rbd3.out‚0 network.sent‚ 1.3922915e+07' disk.nvme0n1.out‚118334.27685546875 disk.dm-1.in‚ 74.8359375 disk.rbd0.in‚0 disk.rbd3.in‚0# proc.memory.rssMB‚ 2870.36328125$ disk./.usageGB‚557.7507438659668 disk.rbd1.out‚ 32.232421875 disk.sda.out‚ 629.6015625 disk.sda3.out‚ 629.6015625 disk.sda2.out‚0 disk.drbd2165.out‚0) disk.nvme0c0n1.out‚118334.27685546875 proc.cpu.threads‚17 disk.dm-0.in‚ 27551.0703125$ disk.dm-0.out‚118334.27685546875* proc.memory.percent‚17.519307136535645 disk.sda1.out‚0 disk.drbd2165.in‚0* disk./.usagePercent‚37.431120404890045 disk.sda2.in‚0 disk.sda1.in‚0Oâò~”‚(:‰ à’éÒÛÁ÷ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚295.976' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚59.1952 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0c“{(Ž‚(: à’éÒ±þÒù$ disk.dm-0.out‚118342.87744140625 disk.sda1.in‚0 disk.sda3.out‚ 630.50390625 disk.sda1.out‚0 disk.sda3.in‚ 74.8359375! disk.nvme0n1.in‚ 27551.4921875 disk.sda.out‚ 630.50390625' proc.memory.availableMB‚ 1030.296875 proc.cpu.threads‚17 network.sent‚ 1.3927975e+07) disk./.usagePercent‚37.43149904003319 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.drbd2165.in‚0) disk.nvme0c0n1.out‚118342.87744140625 disk.rbd1.in‚ 0.47314453125# proc.memory.rssMB‚ 2870.36328125$ disk./.usageGB‚557.7563858032227 disk.rbd3.out‚0' disk.nvme0n1.out‚118342.87744140625 disk.sda2.in‚0 disk.dm-1.out‚ 630.50390625$ memory_percent‚93.71156692504883* proc.memory.percent‚17.519307136535645 cpu‚25.5381614331597 disk.dm-0.in‚ 27551.4921875 disk.drbd2165.out‚0 disk.sda.in‚ 74.8359375 disk.dm-1.in‚ 74.8359375 network.recv‚ 3.024984e+06 disk.rbd3.in‚0# disk.nvme0c0n1.in‚ 27551.4921875 disk.sda2.out‚0 disk.rbd1.out‚ 32.310546875‘äÍË4/ _wandbruntime‚12110  _runtime‚12110[ŽÈ & lr-AdamW/pg1‚0.0005010407549048984 & lr-AdamW/pg2‚0.0005010407549048984 trainer/global_step‚7299 % _timestamp‚1.7843019261027799e+09 _runtime‚12110.319964877 _step‚598G 3nÍ‘Ç & lr-AdamW/pg2‚0.0005010407549048984 trainer/global_step‚7299 % _timestamp‚1.7843019261027799e+09 _runtime‚12110.319964877 _step‚598 & lr-AdamW/pg1‚0.0005010407549048984ÖêW\m4’/ _wandbruntime‚12110  _runtime‚1211092̪(“ýO WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2105264431306752 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚62284273 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2097489829875712 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚575568932195840 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2098396497887232 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20724575 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚32439296 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚63227776 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 400530432 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6856182259503 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6854837659181 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6866588029952 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2104344667534893 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚246 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚63227776 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2104346037563183 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2098396497887232 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚62284279 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 125502076 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6867933419520 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚63217803 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2105263059743744 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2098396471713792 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20724334 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6867933419520 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚39453203 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2105264431306752 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 125512055 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚18728623 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚566708915957760 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚241 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚575568531665408 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚62284279 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8860016238080 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0.C¦v`”‚P VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 trainer/global_step‚7299 _runtime‚12110.321404775 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2098396471713792 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚18728628 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚62284273 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2097489829875712 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚575568932195840 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2098396497887232 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20724575 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚32439296 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚63227776 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 400530432 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6856182259503 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6854837659181 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6866588029952 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2104344667534893 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚246 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚63227776 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2104346037563183 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2098396497887232 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚62284279 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 125502076 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6867933419520 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚63217803 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2105263059743744 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2098396471713792 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20724334 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6867933419520 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚39453203 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2105264431306752 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 125512055 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚18728623 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚566708915957760 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚241 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚575568531665408 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.GI(|¨all.peak‚ 47499471119 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚62284279 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8860016238080 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2105264431306752 J8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚63236268 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6867513065472 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 125528753 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2097767737081856 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2105543357079552 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2098674472443904 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6868858462208 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚566786034274304 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚39458244 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2098674498617344 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚575647622642176 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚566786402365440 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚62292479 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚239 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2104624842807717j<°'—N C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚62292479 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚239 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2104624842807717 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 125528753 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2098674472443904 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚62292485 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8861187844608 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚63236268 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6867513065472 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 125528753 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2097767737081856 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2105543357079552 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2098674472443904 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6868858462208 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚566786034274304 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚39458244 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2098674498617344 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚575647622642176 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚566786402365440ØX†Ï-4˜/ _wandbruntime‚12111  _runtime‚12111FD:™ˆ /loss/FRt1/assignment_loss‚1.523256280926909 0loss/FRt2/assignment_loss‚0.6074181475366155 /loss/FRt2/detection_loss‚1.5306598303201706 /loss/FBt2/detection_loss‚0.5253317794250324 _runtime‚12111.793283409 _step‚601 epoch‚18 2loss/SRqqt2/assignment_loss‚0.4381008685271086 3loss/FBt2/assignment_loss‚0.0007717847019262081 1loss/SRqqt1/detection_loss‚0.7683806925165382 1loss/SRqqt2/detection_loss‚0.5688347824110782 0loss/FBt1/detection_loss‚0.45750630668523473 trainer/global_step‚7299 % _timestamp‚1.7843019275763552e+09 3loss/SRqqt1/assignment_loss‚0.44773347440641925 2loss/FBt1/assignment_loss‚0.004230423223591148 0loss/FRt1/detection_loss‚0.48233382340377234 %loss/total_loss‚7.354558194084396´}2¦š 1loss/SRqqt1/detection_loss‚0.7683806925165382 1loss/SRqqt2/detection_loss‚0.5688347824110782 0loss/FBt1/detection_loss‚0.45750630668523473 trainer/global_step‚7299 % _timestamp‚1.7843019275763552e+09 3loss/SRqqt1/assignment_loss‚0.44773347440641925 2loss/FBt1/assignment_loss‚0.004230423223591148 0loss/FRt1/detection_loss‚0.48233382340377234 %loss/total_loss‚7.354558194084396 /loss/FRt1/assignment_loss‚1.523256280926909 0loss/FRt2/assignment_loss‚0.6074181475366155 /loss/FRt2/detection_loss‚1.5306598303201706 /loss/FBt2/detection_loss‚0.5253317794250324 _runtime‚12111.793283409 _step‚601 epoch‚18 2loss/SRqqt2/assignment_loss‚0.4381008685271086 3loss/FBt2/assignment_loss‚0.0007717847019262081Ùya" Ÿ›‚(:” ï’éÒ¯µˆ÷ gpu.0.gpu‚78.0 gpu.0.memory‚24, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚39.0 gpu.0.powerWatts‚110.286' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.057199999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0F˜á'œ‚(:œ ï’éÒ¾ôÐù disk.sda3.out‚ 630.765625 disk.drbd2165.in‚0$ disk./.usageGB‚557.7608985900879 disk.sda.in‚ 74.8359375 disk.sda.out‚ 630.765625 disk.sda1.in‚0) disk./.usagePercent‚37.43180189694619) disk.nvme0c0n1.out‚118359.14599609375" disk.nvme0n1.in‚27611.64453125# proc.memory.rssMB‚ 2870.36328125 network.sent‚ 1.3933073e+07 disk.sda2.in‚0* proc.memory.percent‚17.519307136535645 proc.cpu.threads‚17 disk.rbd1.in‚ 0.47314453125 disk.dm-1.out‚ 630.765625 disk.rbd0.in‚0$ disk.dm-0.out‚118359.14599609375 cpu‚25.538443767932772 disk.drbd2165.out‚0 disk.rbd3.in‚0$ disk.nvme0c0n1.in‚27611.64453125' proc.memory.availableMB‚ 1030.296875 network.recv‚ 3.028593e+06 disk.rbd3.out‚0$ memory_percent‚93.71156692504883 disk.dm-1.in‚ 74.8359375 disk.rbd0.out‚0' disk.nvme0n1.out‚118359.14599609375 disk.sda2.out‚0 disk.rbd1.out‚ 32.310546875 disk.dm-0.in‚27611.64453125 disk.sda1.out‚0 disk.sda3.in‚ 74.8359375q¯Ä)Ÿ‚(:” þ’éÒÏמ÷ gpu.0.gpu‚92.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚274.423' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚54.884600000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0LYÞ.ž‚(:£ þ’éÒ†«¨ù$ disk.dm-0.out‚118368.35302734375' disk.nvme0n1.out‚118368.35302734375 cpu‚25.538623925505703 disk.drbd2165.in‚0 disk.rbd3.in‚0 disk.rbd1.in‚ 0.47314453125 disk.sda.in‚ 74.8359375$ memory_percent‚93.71194839477539' proc.memory.availableMB‚ 1030.234375 network.recv‚ 3.033058e+06 disk.sda3.out‚ 631.08203125" disk.nvme0n1.in‚27611.84765625 disk.rbd0.out‚0 disk.sda1.in‚0 disk.sda3.in‚ 74.8359375 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd1.out‚ 32.373046875 disk.dm-1.in‚ 74.8359375# proc.memory.rssMB‚ 2870.36328125 proc.cpu.threads‚17$ disk./.usageGB‚557.7676086425781 disk.dm-1.out‚ 631.08203125* disk./.usagePercent‚37.432252214198925 disk.rbd0.in‚0 disk.sda1.out‚0) disk.nvme0c0n1.out‚118368.35302734375 disk.sda.out‚ 631.08203125 network.sent‚ 1.3996185e+07 disk.dm-0.in‚27611.84765625 disk.drbd2165.out‚0 disk.rbd3.out‚0$ disk.nvme0c0n1.in‚27611.84765625* proc.memory.percent‚17.519307136535645bïOÚŸŸ‚(:” “éÒè†üø gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚299.991' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚59.998200000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÅN, ‚(:¡ “éÒÜÊÿú network.sent‚ 1.4001352e+07 disk.rbd0.out‚0 disk.dm-0.in‚ 27613.1015625$ disk.dm-0.out‚118385.24658203125 disk.drbd2165.in‚0 disk.rbd3.out‚0! disk.nvme0n1.in‚ 27613.1015625 cpu‚25.538897592370276 disk.sda2.out‚0 network.recv‚ 3.036615e+06 disk.rbd1.out‚ 32.373046875 disk.sda.in‚ 74.8359375 disk.sda3.out‚ 631.67578125 proc.cpu.threads‚17 disk.dm-1.out‚ 631.67578125* proc.memory.percent‚17.519307136535645$ disk./.usageGB‚557.7745666503906 disk.sda1.out‚0$ memory_percent‚93.71063709259033 disk.rbd1.in‚ 0.47314453125 disk.dm-1.in‚ 74.8359375 disk.rbd3.in‚0) proc.memory.availableMB‚ 1030.44921875 disk.sda2.in‚0) disk./.usagePercent‚37.43271917194137) disk.nvme0c0n1.out‚118385.24658203125# proc.memory.rssMB‚ 2870.36328125 disk.sda.out‚ 631.67578125 disk.rbd0.in‚0 disk.sda1.in‚0 disk.sda3.in‚ 74.8359375 disk.drbd2165.out‚0# disk.nvme0c0n1.in‚ 27613.1015625' disk.nvme0n1.out‚118385.24658203125#ÇTž¡‚(:“ œ“éÒŒÙãö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚291.095' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚58.21900000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Øå)¢‚(:ž œ“éÒ‚¬©ù) disk.nvme0c0n1.out‚118393.98583984375 cpu‚25.539082623073 disk.dm-1.in‚ 74.8359375 disk.rbd0.out‚0 disk.sda1.out‚0 disk.sda3.out‚ 631.8984375( proc.memory.availableMB‚ 1030.3515625 network.recv‚ 3.039896e+06* disk./.usagePercent‚37.432986443807124$ disk.dm-0.out‚118393.98583984375 disk.rbd3.in‚0' disk.nvme0n1.out‚118393.98583984375 disk.sda2.out‚0* proc.memory.percent‚17.519307136535645 disk.sda2.in‚0$ memory_percent‚93.71123313903809 disk.rbd1.in‚ 0.47314453125 network.sent‚ 1.4005925e+07 disk.sda.in‚ 74.8359375 disk.dm-1.out‚ 631.8984375 disk.rbd0.in‚0# proc.memory.rssMB‚ 2870.36328125$ disk./.usageGB‚557.7785491943359 disk.sda.out‚ 631.8984375 disk.sda1.in‚0 disk.sda3.in‚ 74.8359375 disk.drbd2165.in‚0 disk.drbd2165.out‚0" disk.nvme0n1.in‚27613.32421875 disk.rbd1.out‚ 32.451171875 proc.cpu.threads‚17 disk.dm-0.in‚27613.32421875 disk.rbd3.out‚0$ disk.nvme0c0n1.in‚27613.32421875D:Œ£‚(:… «“éÒšé˜ù gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚293.8' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚58.76 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ŒãW«¤‚(:‘ «“éÒ¤ç…û disk.dm-1.out‚ 632.04296875' disk.nvme0c0n1.out‚118404.462890625 disk.sda2.out‚0 disk.sda.in‚ 74.8359375 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.nvme0c0n1.in‚ 27674.9375 disk.sda2.in‚0 disk.rbd1.in‚ 0.47314453125 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda1.out‚0 disk.drbd2165.out‚0 disk.nvme0n1.in‚ 27674.9375$ memory_percent‚93.71206760406494 network.sent‚ 1.4011538e+07# disk./.usageGB‚557.787784576416" disk.dm-0.out‚118404.462890625 disk.drbd2165.in‚0 network.recv‚ 3.043476e+06 disk.dm-1.in‚ 74.8359375% disk.nvme0n1.out‚118404.462890625# proc.memory.rssMB‚ 2870.36328125 disk.sda3.in‚ 74.8359375 disk.rbd1.out‚ 32.451171875) proc.memory.availableMB‚ 1030.21484375 disk.sda.out‚ 632.04296875 disk.sda1.in‚0* proc.memory.percent‚17.519307136535645 cpu‚25.539334344551474) disk./.usagePercent‚37.43360623804752 disk.dm-0.in‚ 27674.9375 disk.sda3.out‚ 632.04296875N6ô4¥/ _wandbruntime‚12189  _runtime‚12189y’ûȦ & lr-AdamW/pg1‚0.0004985578172685878 & lr-AdamW/pg2‚0.0004985578172685878 trainer/global_step‚7349 % _timestamp‚1.7843020054187508e+09 _runtime‚12189.635950476 _step‚602~l´/꤂ % _timestamp‚1.7843020054187508e+09 _runtime‚12189.635950476 _step‚602 & lr-AdamW/pg1‚0.0004985578172685878 & lr-AdamW/pg2‚0.0004985578172685878 trainer/global_step‚7349Ú˜üSG4¨/ _wandbruntime‚12189  _runtime‚12189°8‹F©þO MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚63643555 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 126338128 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 OÆCl¾$@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚570582868250624 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2112296534392832 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6912903960576 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6901074772289 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2112296508219392 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2119209412179968 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚63653528 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2119210783742976 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚62694573 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20860520 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2119209412179968 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚570583236341760 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚62694573 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚245 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8920168660480 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚570582868250624 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2112296534392832 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6912903960576 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6901074772289 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2112296508219392 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2119209412179968 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚63653528 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2119210783742976 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚62694573 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20860520 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2119209412179968 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚570583236341760 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚62694573 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚245 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8920168660480 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 JDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚39721340 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 _runtime‚12191.022549345 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚62702779 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2112574508949504 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚62702785 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 % _timestamp‚1.7843020068052042e+09 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚20863270 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚570660722749440 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚246 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2118563798801223 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚62702785 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8921324849664 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚63662020 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 126364805 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 trainer/global_step‚7349 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8921359378944 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2119489709515776 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2112574508949504 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6915174392832 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6901998238535 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 _step‚604 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚63652033 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2119489709515776 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚62702779 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2112574535122944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 126364805 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚579582082128384 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚570660354658304 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6915174392832 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2118565168829625 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2111661825990656 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚62702785 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8921324849664 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚63662020 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 12636¨á%È?4805 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 trainer/global_step‚7349 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8921359378944 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2119489709515776 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2112574508949504 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6915174392832 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6901998238535 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 _step‚604 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚63652033 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2119489709515776 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚62702779 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2112574535122944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 126364805 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚579582082128384 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚570660354658304 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6915174392832 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2118565168829625 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2111661825990656 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚39721340 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648Üô \³4®/ _wandbruntime‚12191  _runtime‚12191ëÎR ¯† 2loss/FBt1/assignment_loss‚0.001541656279984035 /loss/FRt2/detection_loss‚1.3981490693012868 1loss/SRqqt1/detection_loss‚0.7852303498262501 0loss/FBt2/detection_loss‚0.40242431661889155 trainer/global_step‚7349 _runtime‚12191.10881813 _step‚605 3loss/SRqqt1/assignment_loss‚0.46012284326747876 2loss/FBt2/assignment_loss‚0.005938450259293643 /loss/FRt1/detection_loss‚0.5032752326221472 epoch‚18 0loss/FRt2/assignment_loss‚0.5448744191931237 2loss/SRqqt2/assignment_loss‚0.4444372333546262 1loss/SRqqt2/detection_loss‚0.6293267730706268 %loss/total_loss‚7.135843476050202 /loss/FBt1/detection_loss‚0.5773291887449367 % _timestamp‚1.7843020068918433e+09 0loss/FRt1/assignment_loss‚1.3831939435115568îA^°‹ 0loss/FRt2/assignment_loss‚0.5448744191931237 2loss/SRqqt2/assignment_loss‚0.4444372333546262 1loss/SRqqt2/detection_loss‚0.6293267730706268 %loss/total_loss‚7.135843476050202 /loss/FBt1/detection_loss‚0.5773291887449367 % _timestamp‚1.7843020068918433e+09 0loss/FRt1/assignment_loss‚1.3831939435115568 2loss/FBt1/assignment_loss‚0.001541656279984035 /loss/FRt2/detection_loss‚1.3981490693012868 1loss/SRqqt1/detection_loss‚0.7852303498262501 0loss/FBt2/detection_loss‚0.40242431661889155 trainer/global_step‚7349 _runtime‚12191.10881813 _step‚605 3loss/SRqqt1/assignment_loss‚0.46012284326747876 2loss/FBt2/assignment_loss‚0.005938450259293643 /loss/FRt1/detection_loss‚0.5032752326221472 epoch‚18ÝBûPÙ”±‚(:‰ º“éÒõ÷Þö gpu.0.gpu‚90.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚206.579' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚41.3158 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0qÆ)²‚(:ž º“é҈έù) disk.nvme0c0n1.out‚118417.49560546875 disk.sda2.out‚0 disk.rbd1.out‚ 32.513671875) proc.memory.availableMB‚ 1029.47265625* proc.memory.percent‚17.519307136535645 proc.cpu.threads‚17 network.sent‚ 1.4016064e+07 disk.rbd0.in‚0 disk.sda1.out‚0 disk.sda3.in‚ 74.8359375 disk.sda2.in‚0 disk.dm-1.in‚ 74.8359375 disk.dm-1.out‚ 632.6484375$ disk.dm-0.out‚118417.49560546875 disk.sda.in‚ 74.8359375# proc.memory.rssMB‚ 2870.36328125 disk.rbd0.out‚0# disk.nvme0c0n1.in‚ 27729.6328125$ memory_percent‚93.71659755706787 disk.rbd3.out‚0' disk.nvme0n1.out‚118417.49560546875 cpu‚25.539538984480636$ disk./.usageGB‚557.7922286987305 disk.dm-0.in‚ 27729.6328125 disk.sda.out‚ 632.6484375 network.recv‚ 3.046755e+06 disk.sda1.in‚0 disk.sda3.out‚ 632.6484375 disk.drbd2165.in‚0 disk.rbd3.in‚0! disk.nvme0n1.in‚ 27729.6328125 disk.rbd1.in‚ 0.47314453125) disk./.usagePercent‚37.43390448682491 disk.drbd2165.out‚0+KÎÁ”³‚(:‰ É“éÒÃþãö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚295.976' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚59.1952 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚01Sït%´‚(:š É“éÒÃœù) disk./.usagePercent‚37.43435762016051 disk.rbd3.in‚0 disk.sda2.in‚0 disk.sda.in‚ 74.8359375 disk.rbd0.in‚0 disk.sda3.in‚ 74.8359375$ disk./.usageGB‚557.7989807128906 disk.rbd3.out‚0 disk.sda1.out‚0( proc.memory.availableMB‚ 1029.9453125! disk.nvme0n1.in‚ 27729.8515625 disk.rbd1.out‚ 32.513671875 disk.dm-1.out‚ 633.0546875 cpu‚25.539736319269437 network.sent‚ 1.4079857e+07 disk.rbd1.in‚ 0.47314453125 disk.dm-1.in‚ 74.8359375 disk.drbd2165.in‚0 proc.cpu.threads‚17 network.recv‚ 3.051721e+06& disk.nvme0n1.out‚118428.2333984375 disk.sda2.out‚0 disk.sda.out‚ 633.0546875 disk.sda3.out‚ 633.0546875( disk.nvme0c0n1.out‚118428.2333984375 disk.dm-0.in‚ 27729.8515625# proc.memory.rssMB‚ 2870.36328125# disk.nvme0c0n1.in‚ 27729.8515625 disk.rbd0.out‚0 disk.drbd2165.out‚0$ memory_percent‚93.71371269226074* proc.memory.percent‚17.519307136535645# disk.dm-0.out‚118428.2333984375 disk.sda1.in‚0öØŠÜžµ‚(:“ Ø“éÒ’Åëö gpu.0.gpu‚99.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚336.465' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚67.29299999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¹‹%¶‚(:š Ø“éÒ£²¡ù disk.rbd0.in‚0 network.sent‚ 1.4085047e+07 network.recv‚ 3.055376e+06 disk.sda1.in‚0 disk.rbd3.in‚0 disk.sda2.in‚0 disk.rbd1.out‚ 32.599609375 proc.cpu.threads‚17" disk.dm-0.out‚118441.064453125 disk.drbd2165.out‚0# disk.nvme0c0n1.in‚ 27730.2734375' disk.nvme0c0n1.out‚118441.064453125 disk.sda3.in‚ 74.8359375% disk.nvme0n1.out‚118441.064453125( proc.memory.availableMB‚ 1029.6640625 disk.drbd2165.in‚0 disk.rbd3.out‚0 disk.sda.in‚ 74.8359375 disk.sda.out‚ 633.24609375# proc.memory.rssMB‚ 2870.36328125 disk.rbd0.out‚0 disk.sda3.out‚ 633.24609375 disk.dm-1.out‚ 633.24609375* proc.memory.percent‚17.519307136535645 cpu‚25.53997325038631 disk.dm-0.in‚ 27730.2734375! disk.nvme0n1.in‚ 27730.2734375$ memory_percent‚93.71542930603027 disk.sda1.out‚0 disk.sda2.out‚0 disk.rbd1.in‚ 0.47314453125 disk.dm-1.in‚ 74.8359375* disk./.usagePercent‚37.434826113948176$ disk./.usageGB‚557.8059616088867ü´*kž·‚(:“ ç“éÒǶëö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚283.847' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚56.76939999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 YùD*¸‚(:Ÿ ç“éÒ½þ«ù! disk.nvme0n1.in‚ 27730.4765625 disk.dm-1.in‚ 74.8359375 disk.rbd0.out‚0 cpu‚25.540127840678245$ disk./.usageGB‚557.8101577758789 disk.sda3.in‚ 74.8359375 disk.dm-1.out‚ 633.66796875 disk.rbd1.in‚ 0.47314453125$ disk.dm-0.out‚118451.99951171875 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-0.in‚ 27730.4765625 network.sent‚ 1.4090105e+07 network.recv‚ 3.05896e+06 disk.rbd3.in‚0 disk.sda.in‚ 74.8359375( proc.memory.availableMB‚ 1029.6640625# proc.memory.rssMB‚ 2870.36328125* proc.memory.percent‚17.519307136535645) disk./.usagePercent‚37.43510772223584 disk.drbd2165.out‚0# disk.nvme0c0n1.in‚ 27730.4765625) disk.nvme0c0n1.out‚118451.99951171875 disk.rbd1.out‚ 32.599609375 disk.sda.out‚ 633.66796875 disk.rbd0.in‚0 disk.sda1.in‚0 disk.drbd2165.in‚0 disk.rbd3.out‚0 disk.sda1.out‚0$ memory_percent‚93.71542930603027 proc.cpu.threads‚17' disk.nvme0n1.out‚118451.99951171875 disk.sda3.out‚ 633.66796875§À™=Ÿ¹‚(:” ö“éÒõïóö gpu.0.gpu‚100.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚339.941' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚67.98819999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‘¬’°º‚(: ö“éүǩù network.recv‚ 3.062289e+06 disk.nvme0n1.in‚ 27836.40625 disk.sda.out‚ 633.96875* proc.memory.percent‚17.519307136535645 cpu‚25.540465766374002 proc.cpu.threads‚17 disk.rbd3.out‚0 disk.sda.in‚ 74.8359375 disk.sda1.in‚0 disk.rbd1.in‚ 0.47314453125 disk.dm-0.in‚ 27836.40625 network.sent‚ 1.4095263e+07 disk.sda3.in‚ 74.8359375 disk.sda3.out‚ 633.96875 disk.drbd2165.in‚0 disk.dm-1.in‚ 74.8359375 disk.dm-1.out‚ 633.96875' proc.memory.availableMB‚ 1029.671875! disk.nvme0c0n1.in‚ 27836.40625 disk.sda2.out‚0 disk.rbd0.in‚0 disk.rbd0.out‚0# disk.dm-0.out‚118460.3779296875$ memory_percent‚93.71538162231445$ disk./.usageGB‚557.8198661804199 disk.rbd3.in‚0( disk.nvme0c0n1.out‚118460.3779296875 disk.sda2.in‚0 disk.rbd1.out‚ 32.662109375 disk.sda1.out‚0& disk.nvme0n1.out‚118460.3779296875# proc.memory.rssMB‚ 2870.36328125) disk./.usagePercent‚37.43575926141048 disk.drbd2165.out‚0õŠÚ4»/ _wandbruntime‚12268  _runtime‚12268]EEȼ & lr-AdamW/pg1‚0.0004960650762619257 & lr-AdamW/pg2‚0.0004960650762619257 trainer/global_step‚7399 % _timestamp‚1.7843020846107576e+09 _runtime‚12268.827968984 _step‚606é°ÁŸÍ½Ç % _timestamp‚1.7843020846107576e+09 _runtime‚12268.827968984 _step‚606 & lr-AdamW/pg1‚0.0004960650762619257 & lr-AdamW/pg2‚0.0004960650762619257 trainer/global_step‚7399Þ¨»ýJ4¾/ _wandbruntime‚12268  _runtime‚122688@Ú (¿þO VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2133155764616192 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚18982923 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚64069307 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚64079280 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2133157136179200 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20996211 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2125278007693312 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 trainer/global_step‚7399 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚263 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2125278033121280 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2133155764616192 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6960565280768 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 _runtime‚12268.829183878 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2133157136179200 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 % _timestamp‚1.7843020846118112e+09 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6948656485719 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8979754581504 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚583437345843712 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2132226689606999 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 127174180 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2126196570898432 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8979789117952 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚258 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2126196570898432 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6947311885397 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚63104879 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚574457556725760 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚64079280 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚63104873 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6960565280768 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 127184159 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚39979139 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚583436943216128 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6959219891200 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2132225319578709 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648;cïÍ+ÀƒP L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚64069307 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚64079280 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2133157136179200 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20996211 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2125278007693312 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 trainer/global_step‚7399 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚263 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2125278033121280 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2133155764616192 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6960565280768 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 _runtime‚12268.829183878 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2133157136179200 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 % _timestamp‚1.7843020846118112e+09 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6948656485719 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8979754581504 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚583437345843712 =7DeviceStatsMonitor.on_train_¼ø:eÞbatch_start/num_device_free‚0 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2132226689606999 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 127174180 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2126196570898432 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8979789117952 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚258 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2126196570898432 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6947311885397 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚63104879 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚574457556725760 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚64079280 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚63104873 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6960565280768 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 127184159 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚39979139 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚583436943216128 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6959219891200 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2132225319578709 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2133155764616192 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚18982923ßÐ ùžÁ‚(:“ …”éÒ¥æïö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚264.23' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚52.846000000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0߯¹#‚(:˜ …”éÒ×·£ù disk.sda1.in‚0 disk.sda1.out‚0# proc.memory.rssMB‚ 2870.36328125$ disk.dm-0.out‚118475.37548828125 disk.sda3.in‚ 74.91015625 disk.drbd2165.out‚0' disk.nvme0n1.out‚118475.37548828125 disk.rbd1.out‚ 32.662109375 disk.dm-1.in‚ 74.91015625 proc.cpu.threads‚17 disk.nvme0c0n1.in‚ 27836.625 disk.nvme0n1.in‚ 27836.625 disk.sda2.in‚0) proc.memory.availableMB‚ 1029.67578125* proc.memory.percent‚17.519307136535645 network.recv‚ 3.065492e+06 disk.sda3.out‚ 634.32421875 disk.drbd2165.in‚0 disk.dm-0.in‚ 27836.625) disk./.usagePercent‚37.43606032627075 disk.rbd3.out‚0 disk.rbd0.out‚0 disk.sda2.out‚0 disk.rbd1.in‚ 0.47314453125 disk.sda.in‚ 74.91015625 disk.sda.out‚ 634.32421875$ memory_percent‚93.71535778045654 network.sent‚ 1.4100368e+07$ disk./.usageGB‚557.8243522644043 disk.rbd3.in‚0 disk.rbd0.in‚0) disk.nvme0c0n1.out‚118475.37548828125 disk.dm-1.out‚ 634.32421875 cpu‚25.540660625493224 ØNµ4Ã/ _wandbruntime‚12270  _runtime‚122706Ä8b'ĈN 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2126474571628544 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2126474545455104 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚18985471 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8980945946112 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚256 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 % _timestamp‚1.7843020859978156e+09 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚64087772 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚64077785 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2125555889471488 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6961490323456 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚39984167 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6960144926720 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2132505494851533 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8980980475392 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2133436061952000 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 127200857 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚18985466 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚64077785 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚39984423 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚20998952 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚63113085 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚574535043133440 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚63113079 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6949579952077 _step‚608 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 127190864 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚63113085 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚583516023608832 trainer/global_step‚7399 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6961490323456 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2125555914899456 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 127200857 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚251 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2126474545455104 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚583515620988416 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 127190864Ÿd‹†'ÅN @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚63113079 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6949579952077 _step‚608 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 127190864 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚63113085 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚583516023608832 trainer/global_step‚7399 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6961490323456 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2125555914899456 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 127200857 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚251 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2126474545455104 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚583515620988416 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 127190864 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2126474571628544 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2126474545455104 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚18985471 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8980945946112 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚256 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 % _timestamp‚1.7843020859978156e+09 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚64087772 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚64077785 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2125555889471488 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6961490323456 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚39984167 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6960144926720 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2132505494851533 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8980980475392 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2133436061952000 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 127200857 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚18985466 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚64077785 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚39984423 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚20998952 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚63113085 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚574535043133440àQéiá4Æ/ _wandbruntime‚12270  _runtime‚12270?†äç Ç… 2loss/FBt1/assignment_loss‚0.005622618910111285 trainer/global_step‚7399 _runtime‚12270.300517643 _step‚609 0loss/FRt2/assignment_loss‚0.5991209816576829 .loss/FRt1/detection_loss‚0.518016220327962 /loss/FRt2/detection_loss‚1.4663275373712719 1loss/SRqqt1/detection_loss‚0.7625063989456888 0loss/FBt1/detection_loss‚0.43461268821711396 0loss/FBt2/detection_loss‚0.41523388931201416 %loss/total_loss‚7.299636216704283 1loss/FBt2/assignment_loss‚0.01278630785715329 1loss/SRqqt2/detection_loss‚0.6010276505179671 epoch‚18 % _timestamp‚1.7843020860836253e+09 0loss/FRt1/assignment_loss‚1.4574274212921259 2loss/SRqqt1/assignment_loss‚0.5234340598179023 2loss/SRqqt2/assignment_loss‚0.5035204424772899"™”ÈŠ 1loss/FBt2/assignment_loss‚0.01278630785715329 1loss/SRqqt2/detection_loss‚0.6010276505179671 epoch‚18 % _timestamp‚1.7843020860836253e+09 0loss/FRt1/assignment_loss‚1.4574274212921259 2loss/SRqqt1/assignment_loss‚0.5234340598179023 2loss/SRqqt2/assignment_loss‚0.5035204424772899 2loss/FBt1/assignment_loss‚0.005622618910111285 trainer/global_step‚7399 _runtime‚12270.300517643 _step‚609 0loss/FRt2/assignment_loss‚0.5991209816576829 .loss/FRt1/detection_loss‚0.518016220327962 /loss/FRt2/detection_loss‚1.4663275373712719 1loss/SRqqt1/detection_loss‚0.7625063989456888 0loss/FBt1/detection_loss‚0.43461268821711396 0loss/FBt2/detection_loss‚0.41523388931201416 %loss/total_loss‚7.299636216704283á áùŸÉ‚(:‚ ””éÒÕÙñø gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚337.0' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚67.4 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0&ñòÊ‚(:’ ””éÒñª‚û" disk.nvme0c0n1.in‚ 27836.828125 disk.nvme0n1.in‚ 27836.828125* disk./.usagePercent‚37.436504755350185 disk.sda.in‚ 74.91015625 disk.dm-1.out‚ 634.59375 disk.rbd0.in‚0 disk.sda1.out‚0$ memory_percent‚93.71445178985596* proc.memory.percent‚17.519307136535645 disk.sda.out‚ 634.59375 disk.dm-1.in‚ 74.91015625" disk.dm-0.out‚118483.369140625 disk.sda3.in‚ 74.91015625' disk.nvme0c0n1.out‚118483.369140625% disk.nvme0n1.out‚118483.369140625 cpu‚25.54094883969704 network.sent‚ 1.4152708e+07 disk.sda1.in‚0 disk.sda2.in‚0 proc.cpu.threads‚17 network.recv‚ 3.069951e+06$ disk./.usageGB‚557.8309745788574 disk.drbd2165.in‚0 disk.dm-0.in‚ 27836.828125 disk.drbd2165.out‚0 disk.sda2.out‚0 disk.rbd3.in‚0# proc.memory.rssMB‚ 2870.36328125 disk.rbd1.in‚ 0.47314453125 disk.rbd0.out‚0) proc.memory.availableMB‚ 1029.82421875 disk.rbd1.out‚ 32.833984375 disk.sda3.out‚ 634.59375 disk.rbd3.out‚0˦,”Ë‚(:‰ £”éÒþØÞö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚289.016' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚57.8032 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0õöIUÌ‚(:ˆ £”éÒØú# disk./.usageGB‚557.837818145752 disk.sda.in‚ 74.91015625 disk.sda1.out‚0 disk.nvme0c0n1.in‚27837.25' disk.nvme0c0n1.out‚118498.537109375 network.sent‚ 1.4204572e+07 disk.sda3.out‚ 634.82421875 disk.drbd2165.in‚0 disk.rbd3.in‚0% disk.nvme0n1.out‚118498.537109375 disk.rbd1.in‚ 0.47314453125$ memory_percent‚93.71414184570312# proc.memory.rssMB‚ 2870.36328125 disk.sda.out‚ 634.82421875 disk.dm-1.out‚ 634.82421875 disk.rbd0.out‚0 disk.sda2.out‚0 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.nvme0n1.in‚27837.25) disk./.usagePercent‚37.43696403286661 cpu‚25.54110571031053 network.recv‚ 3.074137e+06 disk.rbd3.out‚0$ proc.memory.availableMB‚1029.875 disk.dm-1.in‚ 74.91015625" disk.dm-0.out‚118498.537109375* proc.memory.percent‚17.519307136535645 disk.dm-0.in‚27837.25 disk.sda1.in‚0 disk.sda3.in‚ 74.91015625 disk.drbd2165.out‚0 disk.sda2.in‚0 disk.rbd1.out‚ 32.833984375=/ÅDÍ‚(:’ ²”éÒ¿ú÷ö gpu.0.gpu‚30.0 gpu.0.memory‚9, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚41.0 gpu.0.powerWatts‚328.257' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚65.65140000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¾Óùr)΂(:ž ²”éÒûì¦ù disk.rbd1.in‚ 0.47314453125 disk.rbd0.in‚0& disk.nvme0n1.out‚118506.2314453125 disk.sda2.out‚0 cpu‚25.541343053643764 disk.sda.in‚ 74.91015625 disk.drbd2165.out‚0( disk.nvme0c0n1.out‚118506.2314453125# proc.memory.rssMB‚ 2870.36328125* proc.memory.percent‚17.519307136535645) disk./.usagePercent‚37.43742433441318$ disk./.usageGB‚557.8446769714355# disk.dm-0.out‚118506.2314453125 disk.rbd3.in‚0) proc.memory.availableMB‚ 1029.62890625 network.recv‚ 3.077489e+06 disk.sda1.in‚0 disk.drbd2165.in‚0 disk.rbd3.out‚0" disk.nvme0n1.in‚27901.98828125 disk.rbd1.out‚ 32.896484375 disk.sda.out‚ 635.015625 disk.dm-0.in‚27901.98828125$ disk.nvme0c0n1.in‚27901.98828125 proc.cpu.threads‚17 network.sent‚ 1.4210129e+07 disk.rbd0.out‚0 disk.sda1.out‚0 disk.sda3.in‚ 74.91015625 disk.dm-1.in‚ 74.91015625 disk.dm-1.out‚ 635.015625 disk.sda3.out‚ 635.015625 disk.sda2.in‚0$ memory_percent‚93.71564388275146ª C’Ï‚(:‡ Á”éÒØŠ÷ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚260.99' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.198 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0gÝxßЂ(:¤ Á”éÒ„×ù disk.rbd0.in‚0 disk.sda1.out‚0 disk.sda3.out‚ 635.40234375( disk.nvme0c0n1.out‚118517.0107421875 disk.sda.in‚ 74.91015625 disk.sda.out‚ 635.40234375 disk.sda1.in‚0 disk.sda3.in‚ 74.91015625 disk.drbd2165.in‚0 cpu‚25.541533982241535 network.sent‚ 1.4215209e+07 disk.dm-0.in‚27902.41015625 disk.drbd2165.out‚0 network.recv‚ 3.âØ:’081257e+06 disk.sda2.in‚0 disk.dm-1.in‚ 74.91015625 disk.rbd0.out‚0* proc.memory.percent‚17.519307136535645$ disk./.usageGB‚557.8518333435059& disk.nvme0n1.out‚118517.0107421875 disk.rbd1.in‚ 0.47314453125# proc.memory.rssMB‚ 2870.36328125 disk.rbd1.out‚ 32.896484375) disk./.usagePercent‚37.43790460454742$ disk.nvme0c0n1.in‚27902.41015625# disk.dm-0.out‚118517.0107421875$ memory_percent‚93.71716976165771 disk.dm-1.out‚ 635.40234375) proc.memory.availableMB‚ 1029.37890625 disk.sda2.out‚0 disk.rbd3.in‚0 disk.rbd3.out‚0" disk.nvme0n1.in‚27902.41015625 proc.cpu.threads‚17Æz»”Ñ‚(:‰ ДéÒ¶†€÷ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚299.456' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚59.8912 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0÷7$,Ò‚(:¡ ДéÒßÈ«ù) disk./.usagePercent‚37.43820157328713 disk.sda3.out‚ 635.5546875 disk.sda2.in‚0$ disk.dm-0.out‚118530.81591796875 disk.drbd2165.in‚0) disk.nvme0c0n1.out‚118530.81591796875# proc.memory.rssMB‚ 2870.36328125 disk.sda2.out‚0 disk.rbd1.in‚ 0.47314453125 disk.sda.out‚ 635.5546875$ memory_percent‚93.71581077575684* proc.memory.percent‚17.519307136535645 cpu‚25.54174785254508 network.sent‚ 1.4220254e+07 disk.sda1.in‚0 disk.rbd3.out‚0$ disk.nvme0c0n1.in‚27902.61328125 disk.dm-1.in‚ 74.91015625# disk./.usageGB‚557.856258392334 disk.rbd3.in‚0 disk.rbd1.out‚ 32.974609375 network.recv‚ 3.084877e+06 disk.dm-0.in‚27902.61328125 disk.sda3.in‚ 74.91015625' disk.nvme0n1.out‚118530.81591796875 disk.sda.in‚ 74.91015625 disk.sda1.out‚0" disk.nvme0n1.in‚27902.61328125( proc.memory.availableMB‚ 1029.6015625 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.drbd2165.out‚0 disk.dm-1.out‚ 635.5546875Ó&w4Ó/ _wandbruntime‚12348  _runtime‚12348*|ä_ÇÔÁ trainer/global_step‚7449 $ _timestamp‚1.784302163865743e+09 _runtime‚12348.082927808 _step‚610 & lr-AdamW/pg1‚0.0004935626919890085 & lr-AdamW/pg2‚0.0004935626919890085…¿Ë0ÌÕÆ trainer/global_step‚7449 $ _timestamp‚1.784302163865743e+09 _runtime‚12348.082927808 _step‚610 & lr-AdamW/pg1‚0.0004935626919890085 & lr-AdamW/pg2‚0.0004935626919890085âŽPW4Ö/ _wandbruntime‚12348  _runtime‚12348 üà (×þO 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 _step‚611 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2139172122030080 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚587371266606592 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚63515173 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9039389496832 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _runtime‚12348.084140056 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6993548998505 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚64505032 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 128020211 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21132545 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚587370863979008 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚40242880 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚21132802 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2146167015628907 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚19110078 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 trainer/global_step‚7449 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2140096607404032 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2147103488615424 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7005535821824 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚262 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚19110073 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7006881211392 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2147103488615424 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2147102117052416 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7006881211392 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚578331877109760 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2139172096602112 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚40242618 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚578331509018624 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7005535821824 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2146165645600617 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚63515179 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚64505032 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚64495059 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚63515173 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2147102117052416 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973KzFÈ (؃P PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚64505032 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 128020211 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21132545 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚587370863979008 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚40242880 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚21132802 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2146167015628907 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚19110078 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 trainer/global_step‚7449 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2140096607404032 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2147103488615424 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7005535821824 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚262 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚19110073 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7006881211392 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2147103488615424 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2147102117052416 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7006881211392 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚578331877109760 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2139172096602112 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚40242618 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚578331509018624 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7005535821824 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2146165645600617 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚63515179 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚64505032 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚64495059 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚63515173 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2147102117052416 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 _step‚611 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2139172122030080 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚587371266606592 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚63515173 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9039389496832 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _runtime‚12348.084140056 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6993548998505 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1ãa÷äC4Ù/ _wandbruntime‚12349  _runtime‚12349Ugÿˆ%%ÚˆN JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 128036909 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚63523385 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚63523379 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2140374608134144 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2140374581960704 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 trainer/global_step‚7449 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2147382414388224 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2139449978380288 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚612 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6994472464751 % _timestamp‚1.7843021652528517e+09 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7006460857344 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2147382414388224 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21135192 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7007806254080 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚243 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚578409363517440 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6995817065185 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7007806254080 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚64513524 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2139450003808256 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚19112616 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚578408995426304 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2147381042818048 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2146445820873441 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚587449940390400 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚63523379 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚40247808 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚587449537769984 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚248 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 128026916 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2140374581960704 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚64503537 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7006460857344 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚19112621 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚21135435 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 128036909 _runtime‚12349.470203671 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚63523385^dMd'ÛN VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚19112621 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚21135435 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 128036909 _runtime‚12349.470203671 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚63523385 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 128036909 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚63523385 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚63523379 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2140374608134144 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2140374581960704 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 trainer/global_step‚7449 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2147382414388224 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2139449978380288 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚612 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6994472464751 % _timestamp‚1.7843021652528517e+09 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7006460857344 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2147382414388224 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21135192 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7007806254080 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚243 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚578409363517440 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6995817065185 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7007806254080 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚64513524 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2139450003808256 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚19112616 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚578408995426304 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2147381042818048 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2146445820873441 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚587449940390400 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚63523379 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚40247808 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚587449537769984 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚248 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 128026916 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2140374581960704 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚64503537 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7006460857344 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820ä<ÆÃ4Ü/ _wandbruntime‚12349  _runtime‚12349àÆ Ý‚ 1loss/SRqqt2/detection_loss‚0.6724332184998206 &loss/total_loss‚7.7562383478895285 _runtime‚12349.556974452 /loss/FRt1/detection_loss‚0.4809800174744024 trainer/global_step‚7449 /loss/FRt1/assignment_loss‚1.491542794256591 /loss/FBt2/detection_loss‚0.4826390622146127 epoch‚18 % _timestamp‚1.7843021653400698e+09 0loss/FRt2/assignment_loss‚0.6147672963019815 2loss/SRqqt1/assignment_loss‚0.5260596617456952 3loss/SRqqt2/assignment_loss‚0.41206915144869877 1loss/FBt1/assignment_loss‚0.07070158972577269 1loss/SRqqt1/detection_loss‚0.8482802739166894 .loss/FBt1/detection_loss‚0.579776210930302 _step‚613 1loss/FBt2/assignment_loss‚0.03608246010639346 .loss/FRt2/detection_loss‚1.540906611268569Zc«÷ Þ‡ .loss/FRt2/detection_loss‚1.540906611268569 1loss/SRqqt2/detection_loss‚0.6724332184998206 &loss/total_loss‚7.7562383478895285 _runtime‚12349.556974452 /loss/FRt1/detection_loss‚0.4809800174744024 trainer/global_step‚7449 /loss/FRt1/assignment_loss‚1.491542794256591 /loss/FBt2/detection_loss‚0.4826390622146127 epoch‚18 % _timestamp‚1.7843021653400698e+09 0loss/FRt2/assignment_loss‚0.6147672963019815 2loss/SRqqt1/assignment_loss‚0.5260596617456952 3loss/SRqqt2/assignment_loss‚0.41206915144869877 1loss/FBt1/assignment_loss‚0.07070158972577269 1loss/SRqqt1/detection_loss‚0.8482802739166894 .loss/FBt1/detection_loss‚0.579776210930302 _step‚613 1loss/FBt2/assignment_loss‚0.03608246010639346åÛßv#Ÿß‚(:” ß”éÒþßçö gpu.0.gpu‚90.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚284.382' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚56.876400000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0èÍžÏ2à‚(:§ ß”éÒí¶¨ù# proc.memory.rssMB‚ 2870.36328125 disk.dm-1.out‚ 635.80078125 disk.rbd0.in‚0 disk.sda1.out‚0$ disk.dm-0.out‚118541.88525390625 disk.sda2.in‚0 disk.rbd1.in‚ 0.47314453125 disk.rbd1.out‚ 32.974609375 disk.sda.in‚ 74.91015625 proc.cpu.threads‚17 disk.drbd2165.in‚0$ disk.nvme0c0n1.in‚27902.83203125' disk.nvme0n1.out‚118541.88525390625 disk.sda1.in‚0* proc.memory.percent‚17.519307136535645 cpu‚25.541959843139136 disk.drbd2165.out‚0 disk.rbd3.out‚0 disk.sda3.out‚ 635.80078125 network.recv‚ 3.088478e+06 disk.rbd0.out‚0 disk.dm-0.in‚27902.83203125 disk.sda3.in‚ 74.91015625 network.sent‚ 1.4225607e+07# disk./.usageGB‚557.853099822998) disk.nvme0c0n1.out‚118541.88525390625" disk.nvme0n1.in‚27902.83203125 disk.sda2.out‚0 disk.sda.out‚ 635.80078125 disk.dm-1.in‚ 74.91015625$ memory_percent‚93.71745586395264* disk./.usagePercent‚37.437989599048784 disk.rbd3.in‚0) proc.memory.availableMB‚ 1029.33203125gíÒR”á‚(:‰ î”éÒŒûö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚290.031' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚58.0062 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0EXJô&â‚(:› î”éÒŽ¤óù disk.dm-1.out‚ 635.97265625 disk.dm-0.in‚27997.32421875 disk.dm-0.out‚118554.4453125 disk.drbd2165.out‚0" disk.nvme0n1.in‚27997.32421875 disk.sda.in‚ 74.91015625 network.recv‚ 3.092765e+06 disk.rbd0.out‚0 disk.sda1.out‚0# disk.nvme0n1.out‚118554.4453125 disk.rbd1.out‚ 33.037109375$ disk./.usageGB‚557.8598403930664% disk.nvme0c0n1.out‚118554.4453125 proc.cpu.threads‚17 disk.sda1.in‚0 disk.rbd3.in‚0$ memory_percent‚93.71495246887207* disk./.usagePercent‚37.438441964361786 disk.rbd1.in‚ 0.47314453125 network.sent‚ 1.4288746e+07 disk.rbd3.out‚0$ disk.nvme0c0n1.in‚27997.32421875( proc.memory.availableMB‚ 1029.7421875* proc.memory.percent‚17.519307136535645 cpu‚25.542168702237134 disk.rbd0.in‚0 disk.sda3.in‚ 74.91015625 disk.sda3.out‚ 635.97265625 disk.drbd2165.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-1.in‚ 74.91015625 disk.sda.out‚ 635.97265625# proc.memory.rssMB‚ 2870.36328125ÙrFŸã‚(:” ý”éÒüõïö gpu.0.gpu‚90.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚265.314' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚53.062799999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ¤è1ä‚(:¦ ý”éÒ©‘ù disk.sda1.in‚0 disk.rbd3.out‚0 disk.sda2.in‚0( proc.memory.availableMB‚ 1029.9765625 cpu‚25.542443276664414 network.sent‚ 1.4292987e+07 disk.rbd1.out‚ 33.037109375 disk.sda.out‚ 636.33203125 disk.dm-0.in‚27997.52734375 disk.drbd2165.out‚0 disk.sda2.out‚0 network.recv‚ 3.095746e+06 disk.dm-1.in‚ 74.91015625$ disk.dm-0.out‚118564.73974609375 disk.rbd3.in‚0$ disk.nvme0c0n1.in‚27997.52734375' disk.nvme0n1.out‚118564.73974609375# proc.memory.rssMB‚ 2870.36328125 disk.sda.in‚ 74.91015625 disk.dm-1.out‚ 636.33203125 disk.rbd0.in‚0 disk.drbd2165.in‚0 disk.rbd0.out‚0 disk.sda1.out‚0 disk.sda3.out‚ 636.33203125) disk./.usagePercent‚37.43889125758438 disk.rbd1.in‚ 0.47314453125 disk.sda3.in‚ 74.91015625" disk.nvme0n1.in‚27997.52734375 proc.cpu.threads‚17) disk.nvme0c0n1.out‚118564.73974609375$ disk./.usageGB‚557.8665351867676$ memory_percent‚93.71352195739746* proc.memory.percent‚17.519307136535645[ß9X”å‚(:‰ Œ•éÒšÇûø gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚293.271' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚58.6542 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÃYél+æ‚(:  Œ•éÒ•íû disk.drbd2165.in‚0 disk.rbd3.in‚0$ disk.nvme0c0n1.in‚27997.94921875 disk.sda.in‚ 74.91015625 disk.sda2.out‚0 disk.dm-1.out‚ 636.55859375 disk.rbd0.out‚0 disk.sda3.in‚ 74.91015625# memory_percent‚93.7150239944458* proc.memory.percent‚17.519307136535645 proc.cpu.threads‚17) disk./.usagePercent‚37.43937203973368 disk.sda.out‚ 636.55859375 disk.sda3.out‚ 636.55859375' disk.nvme0c0n1.out‚118574.013671875) proc.memory.availableMB‚ 1029.73046875 disk.rbd1.in‚ 0.47314453125 disk.sda1.in‚0 disk.drbd2165.out‚0 disk.rbd3.out‚0 disk.rbd1.out‚ 33.099609375 disk.dm-1.in‚ 74.91015625" disk.dm-0.out‚118574.013671875 disk.sda1.out‚0 disk.sda2.in‚0% disk.nvme0n1.out‚118574.013671875 network.sent‚ 1.4298046e+07 network.recv‚ 3.099091e+06$ disk./.usageGB‚557.8736991882324 disk.rbd0.in‚0" disk.nvme0n1.in‚27997.94921875# proc.memory.rssMB‚ 2870.36328125 cpu‚25.542584732437053 disk.dm-0.in‚27997.94921875ù„‡8“ç‚(:ˆ ›•éÒðôö gpu.0.gpu‚100.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚338.21' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚67.642 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0â$V1è‚(:¦ ›•éÒ´­¡ù network.sent‚ 1.4302616e+07$ disk.dm-0.out‚118607.84228515625 disk.sda1.in‚0$ disk.nvme0c0n1.in‚27998.16796875) disk./.usagePercent‚37.44790349081224 disk.rbd1.in‚ 0.47314453125 disk.sda1.out‚0# memory_percent‚93.7150239944458 cpu‚25.542862784686708) disk.nvme0c0n1.out‚118607.84228515625 disk.rbd3.in‚0) proc.memory.availableMB‚ 1029.73046875* proc.memory.percent‚17.519307136535645" disk.nvme0n1.in‚27998.16796875 disk.sda2.out‚0 disk.dm-1.out‚ 640.53515625 disk.sda3.in‚ 74.91015625# proc.memory.rssMB‚ 2870.36328125 proc.cpu.threads‚17 disk.sda2.in‚0 disk.rbd1.out‚ 33.099609375 disk.sda.in‚ 74.91015625 disk.rbd0.in‚0 disk.drbd2165.out‚0 disk.rbd3.out‚0' disk.nvme0n1.out‚118607.84228515625 network.recv‚ 3.102345e+06$ disk./.usageGB‚558.0008239746094 disk.dm-1.in‚ 74.91015625 disk.rbd0.out‚0 disk.sda3.out‚ 640.53515625 disk.drbd2165.in‚0 disk.sda.out‚ 640.53515625 disk.dm-0.in‚27998.16796875ö§Ò4é/ _wandbruntime‚12427  _runtime‚12427kÞ@Èê % _timestamp‚1.7843022432197478e+09 _runtime‚12427.436940256 _step‚614 & lr-AdamW/pg1‚0.0004910508251733023 & lr-AdamW/pg2‚0.0004910508251733023 trainer/global_step‚7499Ì» ’ÍëÇ _step‚614 & lr-AdamW/pg1‚0.0004910508251733023 & lr-AdamW/pg2‚0.0004910508251733023 trainer/global_step‚7499 % _timestamp‚1.7843022432197478e+09 _runtime‚12427.436940256æ«–õJ4ì/ _wandbruntime‚12427  _runtime‚12427bGÅÑ(íýO $ _timestamp‚1.784302243220866e+09 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 128846284 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2161048469488640 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2153996617736192 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 trainer/global_step‚7499 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚64920811 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚63925473 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚261 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚591304880753152 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚64930784 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7039786111613 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚266 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚19237223 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7051851752448 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2160105971622525 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2153066185510912 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7053197142016 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚63925473 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 128856263 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚64920811 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚21269182 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9099085886976 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2161048469488640 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2153996617736192 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 trainer/global_step‚7499 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚64920811 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚63925473 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚261 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚591304880753152 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚64930784 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7039786111613 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚266 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚19237223 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7051851752448 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2160105971622525 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2153066185510912 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7053197142016 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚63925473 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 128856263 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚64920811 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚21269182 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9099085886976 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 JDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 128862968 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚63933679 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2160384776866947 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚258 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚591383556951552 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚582283683901440 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2153344067289088 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7042054178293 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 128872961 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚64939276 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚40511329 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚253 trainer/global_step‚7499 _runtime‚12428.823151872 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2154274618466304 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 128862968 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚63933679 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2154274618466304 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚63933685 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7040709577859 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚64929289 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2161327395254272 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2154274644639744 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21271563 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9100275670528 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175Ô÷Ê›'ñN ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚258 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚591383556951552 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚582283683901440 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2153344067289088 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7042054178293 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 128872961 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚64939276 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚40511329 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚253 trainer/global_step‚7499 _runtime‚12428.823151872 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2154274618466304 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 128862968 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚63933679 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2154274618466304 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚63933685 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7040709577859 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚64929289 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2161327395254272 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2154274644639744 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21271563 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9100275670528 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚591383959571968 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9100241141248 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 128862968 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚63933679 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2160384776866947 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0蛥üµ4ò/ _wandbruntime‚12428  _runtime‚12428BÈ ó† /loss/FRt2/detection_loss‚1.5094395877735676 /loss/FBt2/detection_loss‚0.4782138718106911 3loss/FBt1/assignment_loss‚0.0017666477923115043 $ _timestamp‚1.784302244692846e+09 _step‚617 epoch‚18 0loss/FRt2/assignment_loss‚0.5604544870305123 /loss/FRt1/detection_loss‚0.5217436004204208 0loss/SRqqt1/detection_loss‚0.731702393314306 1loss/SRqqt2/detection_loss‚0.6262312229260214 trainer/global_step‚7499 3loss/SRqqt2/assignment_loss‚0.45583134986538626 3loss/FBt2/assignment_loss‚3.599590745112681e-05 /loss/FBt1/detection_loss‚0.8016898768728424 _runtime‚12428.90983587 %loss/total_loss‚7.684167634921954 0loss/FRt1/assignment_loss‚1.5054910265925256 3loss/SRqqt1/assignment_loss‚0.49156757461591777nqY³ô‹ 3loss/FBt1/assignment_loss‚0.0017666477923115043 $ _timestamp‚1.784302244692846e+09 _step‚617 epoch‚18 0loss/FRt2/assignment_loss‚0.5604544870305123 /loss/FRt1/detection_loss‚0.5217436004204208 0loss/SRqqt1/detection_loss‚0.731702393314306 1loss/SRqqt2/detection_loss‚0.6262312229260214 trainer/global_step‚7499 3loss/SRqqt2/assignment_loss‚0.45583134986538626 3loss/FBt2/assignment_loss‚3.599590745112681e-05 /loss/FBt1/detection_loss‚0.8016898768728424 _runtime‚12428.90983587 %loss/total_loss‚7.684167634921954 0loss/FRt1/assignment_loss‚1.5054910265925256 3loss/SRqqt1/assignment_loss‚0.49156757461591777 /loss/FRt2/detection_loss‚1.5094395877735676 /loss/FBt2/detection_loss‚0.4782138718106911éîì$ÌŸõ‚(:” ª•éÒÀ ¹ö gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚274.958' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚54.991600000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ŸÈžb)ö‚(:ž ª•éÒÿåÝù' disk.nvme0n1.out‚119081.74560546875 disk.rbd1.in‚ 0.47314453125 disk.sda.in‚ 74.91015625 network.sent‚ 1.4308201e+07 disk.drbd2165.in‚0! disk.nvme0c0n1.in‚ 28073.46875) disk.nvme0c0n1.out‚119081.74560546875* proc.memory.percent‚17.519307136535645 network.recv‚ 3.105344e+06 disk.sda1.out‚0 disk.rbd3.in‚0 disk.rbd3.out‚0$ memory_percent‚93.71495246887207( proc.memory.availableMB‚ 1029.7421875* disk./.usagePercent‚37.716196570613185$ disk./.usageGB‚561.9985847473145 disk.sda3.in‚ 74.91015625 disk.rbd1.out‚ 33.224609375 disk.rbd0.in‚0 disk.dm-0.in‚ 28073.46875 disk.sda2.in‚0 disk.sda.out‚ 641.05078125# proc.memory.rssMB‚ 2870.36328125 cpu‚25.543029964633597 disk.sda1.in‚0 disk.sda3.out‚ 641.05078125 disk.drbd2165.out‚0 disk.nvme0n1.in‚ 28073.46875 disk.sda2.out‚0 disk.dm-1.out‚ 641.05078125 proc.cpu.threads‚17 disk.rbd0.out‚0 disk.dm-1.in‚ 74.91015625$ disk.dm-0.out‚119037.37060546875¸š¶â’÷‚(:‡ ¹•éÒ ªèö gpu.0.gpu‚59.0 gpu.0.memory‚19, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚338.59' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚67.718 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 à€)ø‚(:ž ¹•éÒ驹ù disk.nvme0n1.in‚ 28073.890625& disk.nvme0n1.out‚126138.6708984375 disk.sda2.out‚0 disk.rbd1.in‚ 0.47314453125 network.recv‚ 3.109949e+06 disk.sda.in‚ 74.91015625 disk.drbd2165.in‚0 disk.sda3.in‚ 74.91015625 disk.dm-1.out‚ 641.62109375$ memory_percent‚93.71495246887207 network.sent‚ 1.4371115e+07$ disk./.usageGB‚565.2337112426758 disk.sda1.in‚0" disk.nvme0c0n1.in‚ 28073.890625# disk.dm-0.out‚126138.6708984375# proc.memory.rssMB‚ 2870.36328125* proc.memory.percent‚17.519307136535645 proc.cpu.threads‚17 disk.rbd3.out‚0( disk.nvme0c0n1.out‚126138.6708984375 disk.sda2.in‚0 disk.rbd0.out‚0 cpu‚25.543340121278796* disk./.usagePercent‚37.933308624168816 disk.sda1.out‚0 disk.drbd2165.out‚0 disk.rbd3.in‚0 disk.dm-1.in‚ 74.91015625( proc.memory.availableMB‚ 1029.7421875 disk.rbd1.out‚ 33.224609375 disk.sda.out‚ 641.62109375 disk.rbd0.in‚0 disk.dm-0.in‚ 28073.890625 disk.sda3.out‚ 641.62109375ö¶F£Åù‚(:‰ È•éÒ›ñö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts˜mΘÏ‚281.671' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚56.3342 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0OjzÆ"ú‚(:— È•éÒò˜Žù disk.rbd1.out‚ 33.302734375 disk.dm-0.in‚ 28074.109375 disk.sda3.out‚ 642.0078125& disk.nvme0n1.out‚126234.0068359375 disk.sda1.out‚0 disk.drbd2165.out‚0 disk.rbd3.out‚0$ memory_percent‚93.71395111083984# proc.memory.rssMB‚ 2870.36328125 disk.sda.out‚ 642.0078125 network.sent‚ 1.4376203e+07 disk.sda2.out‚0 disk.rbd0.in‚0* proc.memory.percent‚17.519307136535645) disk./.usagePercent‚37.92668622127309( disk.nvme0c0n1.out‚126234.0068359375" disk.nvme0c0n1.in‚ 28074.109375 cpu‚25.543529734008754 disk.dm-1.in‚ 74.91015625 disk.dm-1.out‚ 642.0078125 disk.rbd0.out‚0# disk.dm-0.out‚126234.0068359375 disk.sda1.in‚0 disk.rbd3.in‚0& proc.memory.availableMB‚ 1029.90625 network.recv‚ 3.11331e+06 disk.nvme0n1.in‚ 28074.109375 disk.rbd1.in‚ 0.47314453125 disk.sda3.in‚ 74.91015625 disk.drbd2165.in‚0 proc.cpu.threads‚17 disk.sda.in‚ 74.91015625$ disk./.usageGB‚565.1350326538086 disk.sda2.in‚0@ÐÅ•û‚(:Š וéÒÍîðö gpu.0.gpu‚100.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚334.052' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚66.8104 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0™åþ),ü‚(:¡ וéÒÆ›©ù! disk.nvme0n1.in‚ 28074.3828125 disk.rbd0.out‚0 network.sent‚ 1.4381352e+07 network.recv‚ 3.117105e+06 disk.rbd1.in‚ 0.47314453125 disk.dm-1.in‚ 74.91015625 disk.dm-0.in‚ 28074.3828125 disk.drbd2165.in‚0 disk.rbd3.out‚0$ memory_percent‚93.71616840362549* proc.memory.percent‚17.519307136535645$ disk./.usageGB‚565.1386642456055 disk.rbd1.out‚ 33.302734375 disk.sda.out‚ 644.08984375 disk.sda.in‚ 74.91015625# disk.dm-0.out‚126249.7001953125 disk.sda3.out‚ 644.08984375# disk.nvme0c0n1.in‚ 28074.3828125& disk.nvme0n1.out‚126249.7001953125 cpu‚25.543756543406676 proc.cpu.threads‚17 disk.sda3.in‚ 74.91015625 disk.sda2.in‚0 disk.dm-1.out‚ 644.08984375 disk.rbd0.in‚0) disk./.usagePercent‚37.92692994044569 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd2165.out‚0 disk.rbd3.in‚0# proc.memory.rssMB‚ 2870.36328125 disk.sda2.out‚0( disk.nvme0c0n1.out‚126249.7001953125) proc.memory.availableMB‚ 1029.54296875Û[½Ð”ý‚(:‰ æ•éÒš´úö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚266.646' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.3292 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0?†.#þ‚(:˜ æ•éÒ…ƒþù disk.sda2.in‚0 disk.rbd1.out‚ 33.365234375 disk.sda1.in‚0# disk.nvme0c0n1.in‚ 28254.0703125$ memory_percent‚93.71612071990967 network.sent‚ 1.4386396e+07* disk./.usagePercent‚37.927587879808684 disk.sda.in‚ 74.91015625 disk.dm-1.in‚ 74.91015625 disk.drbd2165.out‚0 disk.rbd1.in‚ 0.47314453125 disk.dm-0.in‚ 28254.0703125 disk.dm-0.out‚126263.1328125 disk.sda1.out‚0 disk.sda3.out‚ 645.66796875 disk.rbd3.in‚0 disk.rbd3.out‚0% disk.nvme0c0n1.out‚126263.1328125 proc.cpu.threads‚17 disk.sda.out‚ 645.66796875# proc.memory.rssMB‚ 2870.36328125* proc.memory.percent‚17.519307136535645 cpu‚25.54394096181056 disk.rbd0.out‚0 disk.sda3.in‚ 74.91015625# disk.nvme0n1.out‚126263.1328125 disk.sda2.out‚0) proc.memory.availableMB‚ 1029.55078125$ disk./.usageGB‚565.1484680175781 disk.dm-1.out‚ 645.66796875 disk.rbd0.in‚0 disk.drbd2165.in‚0 network.recv‚ 3.120429e+06! disk.nvme0n1.in‚ 28254.0703125M9´4ÿ/ _wandbruntime‚12506  _runtime‚12506ðv€Ç€ Á & lr-AdamW/pg2‚0.0004885296371473184 trainer/global_step‚7549 $ _timestamp‚1.784302322504772e+09 _runtime‚12506.721984741 _step‚618 & lr-AdamW/pg1‚0.0004885296371473184²^ãÌ Æ _runtime‚12506.721984741 _step‚618 & lr-AdamW/pg1‚0.0004885296371473184 & lr-AdamW/pg2‚0.0004885296371473184 trainer/global_step‚7549 $ _timestamp‚1.784302322504772e+09êç>Ì4‚ / _wandbruntime‚12506  _runtime‚12506Ç_(ƒ þO [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚64335773 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚40770446 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7098167683072 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2174996193487872 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2167896654241792 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚252 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2174994821924864 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2174996193487872 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2174994821924864 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚64335773 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2167896680415232 % _timestamp‚1.7843023225059369e+09 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2167896680415232 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21406073 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7099513072640 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7098167683072 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚65346563 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚21406325 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2166960274419712 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚65346563 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2166960299847680 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7087367825043 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚40770703 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚595238820689408 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚65356536 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2167896654241792 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7086023224721 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9158670902784 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995hü¢Ì („ ƒP GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 129692315 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚586080517877760 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 129682336 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚64335773 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚40770446 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7098167683072 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2174996193487872 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2167896654241792 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚252 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2174994821924864 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2174996193487872 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2174994821924864 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚64335773 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2167896680415232 % _timestamp‚1.7843023225059369e+09 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2167896680415232 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21406073 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7099513072640 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7098167683072 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚65346563 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚21406325 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2166960274419712 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚65346563 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2166960299847680 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7087367825043 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚40770703 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚595238820689408 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚65356536 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2167896654241792 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7086023224721 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9158670902784 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 129709013 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚64343985 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚32432128 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7086946690967 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚65365028 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚586158004285440 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7088291291401 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2175275119260672 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚65355041 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2168174654971904 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2168174654971904 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2167238156197888 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚40775673 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚21409001 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 129709013 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2174326472917257 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 M=DeviceStatsMonitor.on_train_batch_end/reserved_byt0k es.all.peak‚ 49071259648 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2168174681145344 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚19366921 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2174325102888855 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚12508.107698015 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 129699020 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚64343979 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21408757 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚595317903056384 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚249 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 129699020 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚64343979 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7100438115328 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2167238181625856ž75o'‡ N VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2168174681145344 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚19366921 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2174325102888855 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚12508.107698015 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 129699020 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚64343979 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21408757 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚595317903056384 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚249 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 129699020 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚64343979 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7100438115328 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2167238181625856 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2175275119260672 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚40775922 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚595317502533120 % _timestamp‚1.7843023238903615e+09 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9159866338816 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚64343985 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚244 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 129709013 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚64343985 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚32432128 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7086946690967 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚65365028 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚586158004285440 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7088291291401 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2175275119260672 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚65355041 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2168174654971904 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2168174654971904 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2167238156197888 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚40775673 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚21409001 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 129709013 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2174326472917257ì¤P4ˆ / _wandbruntime‚12508  _runtime‚12508HÃNO ‰ ƒ /loss/FRt1/assignment_loss‚1.434765321947985 0loss/FRt2/assignment_loss‚0.5919290248351761 2loss/SRqqt1/assignment_loss‚0.4293213613222102 1loss/SRqqt1/detection_loss‚0.9045351497671026 epoch‚18 trainer/global_step‚7549 _runtime‚12508.193508388 2loss/SRqqt2/assignment_loss‚0.4363020281739715 3loss/FBt2/assignment_loss‚0.0014924420309596724 /loss/FBt1/detection_loss‚1.4088703980503072 /loss/FBt2/detection_loss‚0.5320413854829307 1loss/FBt1/assignment_loss‚0.00445929970601736 /loss/FRt1/detection_loss‚0.5103993557859503 % _timestamp‚1.7843023239765885e+09 .loss/FRt2/detection_loss‚1.413618382549833 1loss/SRqqt2/detection_loss‚0.6064924461927741 %loss/total_loss‚8.274226595845217 _step‚621. ωŠ ˆ %loss/total_loss‚8.274226595845217 _step‚621 /loss/FRt1/assignment_loss‚1.434765321947985 0loss/FRt2/assignment_loss‚0.5919290248351761 2loss/SRqqt1/assignment_loss‚0.4293213613222102 1loss/SRqqt1/detection_loss‚0.9045351497671026 epoch‚18 trainer/global_step‚7549 _runtime‚12508.193508388 2loss/SRqqt2/assignment_loss‚0.4363020281739715 3loss/FBt2/assignment_loss‚0.0014924420309596724 /loss/FBt1/detection_loss‚1.4088703980503072 /loss/FBt2/detection_loss‚0.5320413854829307 1loss/FBt1/assignment_loss‚0.00445929970601736 /loss/FRt1/detection_loss‚0.5103993557859503 % _timestamp‚1.7843023239765885e+09 .loss/FRt2/detection_loss‚1.413618382549833 1loss/SRqqt2/detection_loss‚0.6064924461927741íW‰Ÿ‹ ‚(:” õ•éÒ¢þ¡ù gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚182.554' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚36.510799999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0–Ò1#Œ ‚(:˜ õ•éÒ¦†êú disk.rbd0.in‚0# proc.memory.rssMB‚ 2870.36328125* proc.memory.percent‚17.519307136535645 disk.sda.out‚647.6875 network.recv‚ 3.124039e+06# disk.nvme0c0n1.in‚ 28254.2890625 disk.dm-0.in‚ 28254.2890625$ disk.dm-0.out‚126284.73486328125 disk.drbd2165.in‚0 disk.drbd2165.out‚0 network.sent‚ 1.4391493e+07$ disk./.usageGB‚565.1544151306152 disk.rbd1.out‚ 33.365234375 disk.dm-1.in‚ 74.91015625 disk.dm-1.out‚647.6875 cpu‚25.544203872637354) disk.nvme0c0n1.out‚126284.73486328125) proc.memory.availableMB‚ 1029.31640625 disk.rbd1.in‚ 0.47314453125) disk./.usagePercent‚37.92798699555456 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda3.out‚647.6875 disk.rbd3.in‚0 disk.rbd3.out‚0! disk.nvme0n1.in‚ 28254.2890625 disk.rbd0.out‚0' disk.nvme0n1.out‚126284.73486328125 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda.in‚ 74.91015625$ memory_percent‚93.71755123138428 disk.sda3.in‚ 74.91015625 proc.cpu.threads‚17ŧ¥ž ‚(:“ „–éÒÅñªø gpu.0.gpu‚90.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚272.302' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚54.46040000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0|‹YÄ&Ž ‚(:› „–éÒÙ‡“ù disk.rbd1.in‚ 0.47314453125 disk.dm-1.out‚ 650.24609375" disk.dm-0.out‚126298.033203125 disk.sda.out‚ 650.24609375 disk.dm-1.in‚ 74.91015625 disk.dm-0.in‚ 28254.7109375 disk.drbd2165.out‚0# disk.nvme0c0n1.in‚ 28254.7109375 disk.sda3.in‚ 74.91015625 disk.sda3.out‚ 650.24609375# memory_percent‚93.7190055847168 disk.rbd3.out‚0 disk.sda1.in‚0 network.recv‚ 3.128412e+06) disk./.usagePercent‚37.92832031736415 disk.rbd1.out‚ 33.427734375 disk.rbd0.in‚0# proc.memory.rssMB‚ 2870.36328125 disk.sda1.out‚0$ disk./.usageGB‚565.1593818664551 disk.sda.in‚ 74.91015625 disk.rbd0.out‚0 cpu‚25.544417239966638 proc.cpu.threads‚17* proc.memory.percent‚17.519307136535645% disk.nvme0n1.out‚126298.033203125 disk.sda2.in‚0' proc.memory.availableMB‚ 1029.078125 disk.drbd2165.in‚0 disk.rbd3.in‚0 network.sent‚ 1.4453843e+07' disk.nvme0c0n1.out‚126298.033203125! disk.nvme0n1.in‚ 28254.7109375 disk.sda2.out‚0"æó Ÿ ‚(:” “–éÒçÍ»ø gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚305.928' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚61.185599999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ü æ' ‚(:œ “–éÒÔœ¿ú disk.sda3.in‚ 74.91015625 disk.rbd0.out‚0 disk.sda2.in‚0 disk.sda.in‚ 74.91015625 cpu‚25.5446370457617 disk.rbd3.out‚0' disk.nvme0c0n1.out‚126315.443359375! disk.nvme0n1.in‚ 28254.9140625% disk.nvme0n1.out‚126315.443359375 disk.rbd1.out‚ 33.427734375 disk.rbd0.in‚0 disk.dm-0.in‚ 28254.9140625* proc.memory.percent‚17.519307136535645 network.recv‚ 3.132123e+06* disk./.usagePercent‚37.928635206631256 disk.sda1.in‚0 disk.rbd3.in‚0( proc.memory.availableMB‚ 1029.2890625 proc.cpu.threads‚17 disk.sda3.out‚ 650.93359375 disk.drbd2165.out‚0# disk.nvme0c0n1.in‚ 28254.9140625 disk.sda2.out‚0 disk.dm-1.in‚ 74.91015625 disk.dm-1.out‚ 650.93359375$ memory_percent‚93.71771812438965$ disk./.usageGB‚565.1640739440918 disk.sda1.out‚0 disk.drbd2165.in‚0# proc.memory.rssMB‚ 2870.36328125 network.sent‚ 1.4458899e+07 disk.rbd1.in‚ 0.47314453125" disk.dm-0.out‚126315.443359375 disk.sda.out‚ 650.93359375ÐÓF?”‘ ‚(:‰ ¢–éÒéÅçö gpu.0.gpu‚90.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚278.726' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.7452 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0´Ó!Ÿ2’ ‚(:§ ¢–éҫʼnù disk.drbd2165.in‚0 disk.drbd2165.out‚0' disk.nvme0n1.out‚126323.70458984375 disk.rbd1.in‚ 0.47314453125# proc.memory.rssMB‚ 2870.36328125 disk.dm-0.in‚28464.71484375 disk.sda1.in‚0" disk.nvme0n1.in‚28464.71484375 disk.rbd1.out‚ 33.490234375 proc.cpu.threads‚17 network.recv‚ 3.135608e+06 disk.sda3.out‚ 651.12109375 disk.rbd3.in‚0$ disk.nvme0c0n1.in‚28464.71484375 disk.sda2.in‚0 disk.rbd0.out‚0) disk.nvme0c0n1.out‚126323.70458984375 disk.sda2.out‚0* proc.memory.percent‚17.519307136535645 cpu‚25.544848234598426$ disk./.usageGB‚565.1714134216309 disk.dm-1.out‚ 651.12109375* disk./.usagePercent‚37.929127765127134$ disk.dm-0.out‚126323.70458984375 disk.dm-1.in‚ 74.91015625( proc.memory.availableMB‚ 1028.9453125 disk.sda.in‚ 74.91015625 disk.sda.out‚ 651.12109375 disk.rbd0.in‚0 disk.sda1.out‚0 disk.rbd3.out‚0$ memory_percent‚93.71981620788574 network.sent‚ 1.4464644e+07 disk.sda3.in‚ 74.910156252H¤d”“ ‚(:‰ ±–éÒÍ°âö gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚283.649' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚56.7298 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 &µt0” ‚(:¥ ±–éÒ罈ù disk.sda2.in‚0 disk.rbd1.out‚ 33.490234375 disk.sda.out‚ 653.76953125 disk.rbd0.out‚0 cpu‚25.545033073186037 disk.dm-1.out‚ 653.76953125 network.recv‚ 3.139468e+06) disk.nvme0c0n1.out‚126344.77197265625" disk.nvme0n1.in‚28465.13671875' disk.nvme0n1.out‚126344.77197265625 disk.rbd0.in‚0 disk.dm-0.in‚28465.13671875$ memory_percent‚93.72143745422363 disk.dm-1.in‚ 74.91015625 disk.sda3.out‚ 653.76953125 disk.drbd2165.out‚0* proc.memory.percent‚17.519307136535645$ disk.nvme0c0n1.in‚28465.13671875 disk.sda1.in‚0 disk.sda3.in‚ 74.91015625( proc.memory.availableMB‚ 1028.6796875 proc.cpu.threads‚17 disk.drbd2165.in‚0) disk./.usagePercent‚37.92989117959424 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.sda2.out‚0 disk.sda.in‚ 74.91015625$ disk.dm-0.out‚126344.77197265625 disk.sda1.out‚0 network.sent‚ 1.4469791e+07# disk./.usageGB‚565.182788848877 disk.rbd1.in‚ 0.47314453125# proc.memory.rssMB‚ 2870.36328125¹\甕 ‚(:‰ À–éÒå©ñö gpu.0.gpu‚99.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚340.764' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚68.1528 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚02QïQ0– ‚(:¥ À–éÒï–¹ù disk.drbd2165.out‚0) disk.nvme0c0n1.out‚126365.45458984375' proc.memory.availableMB‚ 1029.140625 network.sent‚ 1.4474848e+07 disk.sda2.out‚0 cpu‚25.545295147766304 disk.rbd1.in‚ 0.47314453125 network.recv‚ 3.142623e+06 disk.rbd3.out‚0$ disk.nvme0c0n1.in‚28465.33984375 disk.sda1.in‚0 disk.sda3.in‚ 74.91015625$ memory_percent‚93.71862411499023# proc.memory.rssMB‚ 2870.36328125 disk.sda.in‚ 74.91015625 disk.rbd0.out‚0 disk.rbd1.out‚ 33.615234375 disk.sda2.in‚0 disk.dm-1.out‚ 656.15234375 disk.sda1.out‚0* proc.memory.percent‚17.519307136535645* disk./.usagePercent‚37.930300791649024 disk.dm-1.in‚ 74.91015625 disk.sda3.out‚ 656.15234375 proc.cpu.threads‚17 disk.drbd2165.in‚0# disk./.usageGB‚565.188892364502 disk.rbd3.in‚0" disk.nvme0n1.in‚28465.33984375$ disk.dm-0.out‚126365.45458984375' disk.nvme0n1.out‚126365.45458984375 disk.sda.out‚ 656.15234375 disk.rbd0.in‚0 disk.dm-0.in‚28465.33984375>{wü4— / _wandbruntime‚12586  _runtime‚12586q9ãȘ  _step‚622 & lr-AdamW/pg1‚0.0004859992898422531 & lr-AdamW/pg2‚0.0004859992898422531 trainer/global_step‚7599 % _timestamp‚1.7843024019568927e+09 _runtime‚12586.174026597V<±Í™ Ç & lr-AdamW/pg1‚0.0004859992898422531 & lr-AdamW/pg2‚0.0004859992898422531 trainer/global_step‚7599 % _timestamp‚1.7843024019568927e+09 _runtime‚12586.174026597 _step‚622î¾ÉO4š / _wandbruntime‚12586  _runtime‚12586mì\;\#› ýO GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚64746079 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21542547 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚21542785 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9218533524992 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2181796716920832 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚589954838261760 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7144483613696 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚41034313 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2180854388756480 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 130518388 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚19491523 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 400530432 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7145829003264 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚599172971256320 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2188941174361088 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9218501085696 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚65782288 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚65772315 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚243 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7132260337829 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 trainer/global_step‚7599 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7145829003264 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2187987993694631 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚19491528 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2188941174361088 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚589954470170624 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2188942545924096 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚64746079 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚238 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚599173371786752 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2187986623666341 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2181796716920832 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 _runtime‚12586.175369461 _step‚623 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚64746073 $ _timestamp‚1.784302401958051e+09 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2180854363328512 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚64746073 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2188942545924096 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 130528367 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WEDeviceStatsMong•7”§itor.on_train_batch_start/active_bytes.small_pool.freed‚ 7144483613696 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2181796690747392 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚32439296 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚65772315 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚41034070 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚65782288 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2181796690747392 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2187987993694631 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚19491528 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2188941174361088 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚589954470170624 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2188942545924096 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚64746079 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚238 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚599173371786752 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2187986623666341 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2181796716920832 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 _runtime‚12586.175369461 _step‚623 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚64746073 $ _timestamp‚1.784302401958051e+09 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2180854363328512 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚64746073 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2188942545924096 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 130528367 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7144483613696 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2181796690747392 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚32439296 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚65772315 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚41034070 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚65782288 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2181796690747392 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚41034313 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2180854388756480 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 130518388 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚19491523 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 400530432 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7145829003264 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚599172971256320 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2188941174361088 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9218501085696 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚65782288ä¼4 / _wandbruntime‚12587  _runtime‚12587 v¤_'ž ˆN REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚64754279 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚21545522 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚590032324669440 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7134528404509 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚41039360 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2188265428910763 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 % _timestamp‚1.7843024033563745e+09 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚64754285 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 _runtime‚12587.573744019 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚64754285 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2182074717650944 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2182074691477504 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2182074717650944 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚65790780 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚19494066 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚599251656966656 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2189221471696896 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚32432128 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9219700388352 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2181132245106688 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 trainer/global_step‚7599 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7146754045952 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2182074691477504 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚233 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2188266798939165 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 130535072 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚65780793 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2189221471696896 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚590031956578304 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 130545065 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚228 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21545294 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9219732820480 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚19494071 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚599252057489920 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚65780793 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7145408649216 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 130535072 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572VÜ1ž'Ÿ N FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 130545065 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚228 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21545294 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9219732820480 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚19494071 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚599252057489920 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚65780793 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7145408649216 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 130535072 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚64754279 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚21545522 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚590032324669440 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7134528404509 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚41039360 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2188265428910763 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 % _timestamp‚1.7843024033563745e+09 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚64754285 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 _runtime‚12587.573744019 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚64754285 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2182074717650944 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2182074691477504 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2182074717650944 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚65790780 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚19494066 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚599251656966656 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2189221471696896 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚32432128 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9219700388352 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2181132245106688 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 trainer/global_step‚7599 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7146754045952 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2182074691477504 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚233 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2188266798939165 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 130535072 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚65780793 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2189221471696896 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚590031956578304 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995ð14  / _wandbruntime‚12587  _runtime‚12587'¿1N ¡ † /loss/FBt1/detection_loss‚0.5079910053625235 /loss/FRt1/assignment_loss‚1.452212315548298 3loss/SRqqt1/assignment_loss‚0.47940444537858246 2loss/FBt2/assignment_loss‚0.006620904913571264 /loss/FRt2/detection_loss‚1.4564316880115151 % _timestamp‚1.7843024034343278e+09 _runtime‚12587.651520601 0loss/FBt2/detection_loss‚0.43751126208349667 &loss/total_loss‚7.3452634786696915 2loss/SRqqt2/assignment_loss‚0.4812882698057139 2loss/FBt1/assignment_loss‚0.006075958739653134 1loss/SRqqt1/detection_loss‚0.7653916973325282 1loss/SRqqt2/detection_loss‚0.6003850762360621 /loss/FRt2/assignment_loss‚0.646500107220241 /loss/FRt1/detection_loss‚0.5054507480375062 epoch‚18 trainer/global_step‚7599 _step‚625¼½„¢ ‹ /loss/FRt2/assignment_loss‚0.646500107220241 /loss/FRt1/detection_loss‚0.5054507480375062 epoch‚18 trainer/global_step‚7599 _step‚625 /loss/FBt1/detection_loss‚0.5079910053625235 /loss/FRt1/assignment_loss‚1.452212315548298 3loss/SRqqt1/assignment_loss‚0.47940444537858246 2loss/FBt2/assignment_loss‚0.006620904913571264 /loss/FRt2/detectio6ùƒìn_loss‚1.4564316880115151 % _timestamp‚1.7843024034343278e+09 _runtime‚12587.651520601 0loss/FBt2/detection_loss‚0.43751126208349667 &loss/total_loss‚7.3452634786696915 2loss/SRqqt2/assignment_loss‚0.4812882698057139 2loss/FBt1/assignment_loss‚0.006075958739653134 1loss/SRqqt1/detection_loss‚0.7653916973325282 1loss/SRqqt2/detection_loss‚0.6003850762360621ñ¹~Ÿ£ ‚(:” Ï–éÒÏàö gpu.0.gpu‚100.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚334.587' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚66.91739999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0SJ÷$¤ ‚(:™ Ï–éÒ½¬Îø disk.sda3.in‚ 74.91015625 disk.sda3.out‚ 657.796875 disk.drbd2165.out‚0( disk.nvme0c0n1.out‚126384.1064453125 disk.sda2.out‚0& proc.memory.availableMB‚ 1028.65625 network.sent‚ 1.4479945e+07 network.recv‚ 3.145954e+06" disk.nvme0n1.in‚28465.55859375 disk.sda.out‚ 657.796875 disk.dm-0.in‚28465.55859375 disk.sda1.in‚0& disk.nvme0n1.out‚126384.1064453125 proc.cpu.threads‚17$ disk./.usageGB‚557.9704971313477 disk.dm-1.out‚ 657.796875 disk.rbd0.in‚0 disk.rbd0.out‚0# disk.dm-0.out‚126384.1064453125 disk.drbd2165.in‚0 disk.sda2.in‚0 disk.sda.in‚ 74.91015625# memory_percent‚93.7215805053711 cpu‚25.54774027806773 disk.sda1.out‚0$ disk.nvme0c0n1.in‚28465.55859375 disk.rbd1.out‚ 33.615234375" proc.memory.rssMB‚ 2870.6953125* proc.memory.percent‚17.521333694458008 disk.rbd3.out‚0* disk./.usagePercent‚37.445868230915025 disk.rbd3.in‚0 disk.rbd1.in‚ 0.47314453125 disk.dm-1.in‚ 74.91015625膧Œ4¥ / _wandbruntime‚12614  _runtime‚12614&æD‡í=¦ ç{ :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.07407407462596893 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9893755316734314 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.19396711885929108 6 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.256561815738678 *jet/accuracy_2_of_3‚0.5344117879867554 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.6966203451156616 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.22172124683856964 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 jet/accuracy_1_of_5‚1 8"Purity/*FRt*SRqqt0FBt/event_purity‚0.280536025762558 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.24804016947746277 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9993476867675781 /particle/accuracy_3_of_4‚0.9965986609458923 particle/accuracy_4_of_5‚1 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9629628658294678 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6683636903762817 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7325661778450012 +particle/sensitivity‚0.6199108362197876 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5708444714546204 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7607132792472839 9"Purity/*FRt2SRqqt*FBt/event_purity‚0.1593015193939209 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6239995956420898 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7316704988479614 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.27013781666755676 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.07828729599714279 % _timestamp‚1.7843024304871035e+09 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9988515973091125 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5483332872390747 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6226220726966858 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9615384340286255 jet/accuracy_3_of_5‚1 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6233590841293335 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9953210949897766 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.5177043080329895 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8067271709442139 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 +"Purity/2FRt*SRqqt2FBt/event_purity‚0.75 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.24532122910022736 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5802963972091675 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 -"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4375 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.26495179533958435 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6832152009010315 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5708444714546204 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6104557514190674 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 trainer/global_step‚7599 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9211474657058716 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9981684684753418 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 +jet/accuracy_2_of_2‚0.14675603806972504 6 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.988112211227417 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6598780751228333 particle/accuracy_2_of_4‚1 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.07692307978868484 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 8"Purity/*FRt0SRqqt*FBt/event_purity‚0.315530389547348 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6109339594841003 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9814814329147339 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8009259104728699 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 8"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.573768138885498 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.3249010443687439 particle/accuracy_1_of_1‚1 particle/accuracy_2_of_3‚1 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 *loss/val_total_loss‚7.9292645502768595 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8156840801239014 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6623148918151855 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5495725870132446 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.03073298931121826 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9285714030265808 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.09288384020328522 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 , Purity/0FRt2SRqqt1FBt/FBt_purity‚0.96875 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2462700456380844 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6573730707168579 'particle/f_score‚0.6379730105400085 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9073538780212402 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6676201224327087 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6683384776115417 +jet/accuracy_4_of_4‚0.01628045178949833 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4937571585178375 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9953050017356873 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 , Purity/0FRt2SRqqt*FBt/FBt_purity‚0.96875 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6566101908683777 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9343931078910828 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8745091557502747 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9876638054847717 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.47632530331611633 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9947041273117065 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4987938404083252 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7010595798492432 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6710101366043091 +particle/specificity‚0.9115456938743591 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6641576886177063 +jet/accuracy_3_of_3‚0.07446277141571045 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9814814329147339 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 +jet/accuracy_1_of_1‚0.31905561685562134 /particle/accuracy_4_of_4‚0.9417513012886047 7 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.0928938016295433 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.23445038497447968 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7146207094192505 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9984291791915894 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7402340769767761 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.07692307978868484 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 8"Purity/2FRt*SRqqt1FBt/event_purity‚0.148333340883255 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.25635334849357605 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.07828729599714279 particle/accuracy_1_of_3‚1 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6157273650169373 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 jet/accuracy_2_of_5‚1 jet/accuracy_4_of_5‚0 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8017138242721558 (particle/accuracy‚0.8489117622375488 jet/accuracy_5_of_5‚0 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.8571428656578064 _step‚626 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3697916865348816 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9960570335388184 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3388458788394928 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.16235534846782684 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.20468538999557495 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.03980899974703789 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.24449820816516876 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.23302073776721954 8"Purity/1FRt0SRqqt0FBt/event_purity‚0.256561815738678 *jet/accuracy_1_of_4‚0.9733967781066895 particle/accuracy_1_of_2‚1 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.6686177253723145 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9960944652557373 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9787510633468628 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4745370149612427 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8017138242721558 particle/accuracy_5_of_5‚0 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9904571771621704 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.36456114053726196 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6550249457359314 +jet/accuracy_3_of_4‚0.09955698996782303 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2380513846874237 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.07781771570444107 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.31845372915267944 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8060469031333923 particle/accuracy_2_of_2‚1 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5968766212463379 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6082075238227844 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.6733895540237427 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6102458834648132 6 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.584516704082489 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8230537176132202 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.42824074625968933 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.3542350232601166 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6536458730697632 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.19903244078159332 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 particle/accuracy_2_of_5‚1 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9994588494300842 6 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.995456874370575 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9993476867675781 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.048920631408691406 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.09802428632974625 * Purity/2FRt*SRqqt2FBt/FRt_purity‚0.875 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3449241518974304 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.5707840919494629 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.07407407462596893 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.18919825553894043 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 _runtime‚12614.704967931 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8725374937057495 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.26803743839263916 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2683295011520386 epoch‚18 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5949066877365112 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8060469031333923 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.060512010008096695 particle/accuracy_3_of_5‚1 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.26807132363319397 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.09814433753490448 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.21345174312591553 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_4‚1 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 )jet/accuracy_1_of_2‚0.775239109992981 *jet/accuracy_1_of_3‚0.9212849140167236 *jet/accuracy_2_of_4‚0.7909814715385437 /particle/accuracy_3_of_3‚0.9994120597839355 6validation_average_jet_accuracy‚0.3620518743991852 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.2555597126483917 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.5790383815765381 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 8"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.731351375579834 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8009259104728699 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9966031312942505 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9990078806877136 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.20353372395038605 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6493993997573853 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03659316524863243 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 particle/accuracy_1_of_5‚1¿‹LOe:§ ì{ 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9211474657058716 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9981684684753418 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 +jet/accuracy_2_of_2‚0.14675603806972504 6 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.988112211227417 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6598780751228333 particle/accuracy_2_of_4‚1 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.07692307978868484 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 8"Purity/*FRt0SRqqt*FBt/event_purity‚0.315530389547348 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6109339594841003 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9814814329147339 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8009259104728699 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 8"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.573768138885498 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.3249010443687439 particle/accuracy_1_of_1‚1 particle/accuracy_2_of_3‚1 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 *loss/val_total_loss‚7.9292645502768595 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8156840801239014 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6623148918151855 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5495725870132446 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.03073298931121826 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9285714030265808 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.09288384020328522 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 , Purity/0FRt2SRqqt1FBt/FBt_purity‚0.96875 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2462700456380844 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6573730707168579 'particle/f_score‚0.6379730105400085 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9073538780212402 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6676201224327087 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6683384776115417 +jet/accuracy_4_of_4‚0.01628045178949833 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4937571585178375 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9953050017356873 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 , Purity/0FRt2SRqqt*FBt/FBt_purity‚0.96875 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6566101908683777 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9343931078910828 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8745091557502747 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9876638054847717 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.47632530331611633 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9947041273117065 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4987938404083252 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7010595798492432 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6710101366043091 +particle/specificity‚0.9115456938743591 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6641576886177063 +jet/accuracy_3_of_3‚0.07446277141571045 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9814814329147339 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 +jet/accuracy_1_of_1‚0.31905561685562134 /particle/accuracy_4_of_4‚0.9417513012886047 7 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.0928938016295433 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.23445038497447968 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7146207094192505 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9984291791915894 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7402340769767761 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.07692307978868484 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 8"Purity/2FRt*SRqqt1FBt/event_purity‚0.148333340883255 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.25635334849357605 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.07828729599714279 particle/accuracy_1_of_3‚1 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6157273650169373 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 jet/accuracy_2_of_5‚1 jet/accuracy_4_of_5‚0 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8017138242721558 (particle/accuracy‚0.8489117622375488 jet/accuracy_5_of_5‚0 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.8571428656578064 _step‚626 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3697916865348816 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9960570335388184 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3388458788394928 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.16235534846782684 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.20468538999557495 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.03980899974703789 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.24449820816516876 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.23302073776721954 8"Purity/1FRt0SRqqt0FBt/event_purity‚0.256561815738678 *jet/accuracy_1_of_4‚0.9733967781066895 particle/accuracy_1_of_2‚1 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.6686177253723145 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9960944652557373 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9787510633468628 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4745370149612427 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8017138242721558 particle/accuracy_5_of_5‚0 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9904571771621704 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.36456114053726196 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6550249457359314 +jet/accuracy_3_of_4‚0.09955698996782303 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.2380513846874237 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.07781771570444107 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.31845372915267944 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8060469031333923 particle/accuracy_2_of_2‚1 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5968766212463379 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6082075238227844 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.6733895540237427 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6102458834648132 6 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.584516704082489 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8230537176132202 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.42824074625968933 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.3542350232601166 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6536458730697632 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.19903244078159332 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 particle/accuracy_2_of_5‚1 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9994588494300842 6 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.995456874370575 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9993476867675781 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.048920631408691406 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.09802428632974625 * Purity/2FRt*SRqqt2FBt/FRt_purity‚0.875 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3449241518974304 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.5707840919494629 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.07407407462596893 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.18919825553894043 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 _runtime‚12614.704967931 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8725374937057495 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.26803743839263916 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2683295011520386 epoch‚18 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5949066877365112 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8060469031333923 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.060512010008096695 particle/accuracy_3_of_5‚1 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.26807132363319397 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.09814433753490448 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.21345174312591553 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_4‚1 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 )jet/accuracy_1_of_2‚0.775239109992981 *jet/accuracy_1_of_3‚0.9212849140167236 *jet/accuracy_2_of_4‚0.7909814715385437 /particle/accuracy_3_of_3‚0.9994120597839355 6validation_average_jet_accuracy‚0.3620518743991852 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.2555597126483917 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.5790383815765381 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 8"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.731351375579834 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8009259104728699 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9966031312942505 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9990078806877136 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.20353372395038605 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6493993997573853 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03659316524863243 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 particle/accuracy_1_of_5‚1 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.07407407462596893 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9893755316734314 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.19396711885929108 6 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.256561815738678 *jet/accuracy_2_of_3‚0.5344117879867554 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.6966203451156616 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.22172124683856964 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 jet/accuracy_1_of_5‚1 8"Purity/*FRt*SRqqt0FBt/event_purity‚0.280536025762558 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.24804016947746277 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9993476867675781 /particle/accuracy_3_of_4‚0.9965986609458923 particle/accuracy_4_of_5‚1 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9629628658294678 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6683636903762817 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7325661778450012 +particle/sensitivity‚0.6199108362197876 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5708444714546204 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7607132792472839 9"Purity/*FRt2SRqqt*FBt/event_purity‚0.1593015193939209 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6239995956420898 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7316704988479614 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.27013781666755676 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.07828729599714279 % _timestamp‚1.7843024304871035e+09 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9988515973091125 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5483332872390747 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6226220726966858 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9615384340286255 jet/accuracy_3_of_5‚1 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6233590841293335 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9953210949897766 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.5177043080329895 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8067271709442139 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 +"Purity/2FRt*SRqqt2FBt/event_purity‚0.75 @&Purity/2FRtA G*SRqqt1FBt/event_proportion‚0.0016043527284637094 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.24532122910022736 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5802963972091675 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 -"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4375 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.26495179533958435 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6832152009010315 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5708444714546204 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6104557514190674 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 trainer/global_step‚7599 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828òÈÙÿ¨ ‚(:’ Þ–éÒ¾‚âö gpu.0.gpu‚3.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚35.0 gpu.0.powerWatts‚109.058' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.811600000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ƒQê $© ‚(:™ Þ–éÒ–Šƒù) proc.memory.availableMB‚ 1096.65234375 disk.rbd1.out‚ 33.677734375 disk.dm-1.out‚ 659.2890625 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.sda1.out‚0 disk.sda3.out‚ 659.2890625 disk.sda1.in‚0 disk.drbd2165.in‚0 network.recv‚ 3.150383e+06! disk.nvme0n1.in‚ 28534.3828125 disk.sda2.in‚0 disk.rbd1.in‚ 0.47314453125) proc.memory.percent‚17.10531711578369% disk.nvme0n1.out‚126401.384765625 disk.drbd2165.out‚0# proc.memory.rssMB‚ 2802.53515625' disk.nvme0c0n1.out‚126401.384765625 disk.rbd0.in‚0 disk.rbd0.out‚0" disk.dm-0.out‚126401.384765625 disk.sda3.in‚ 74.91015625 cpu‚25.550945746544013# disk.nvme0c0n1.in‚ 28534.3828125 disk.dm-0.in‚ 28534.3828125 proc.cpu.threads‚17 disk.sda.out‚ 659.2890625 disk.dm-1.in‚ 74.91015625* disk./.usagePercent‚37.446321876265706$ disk./.usageGB‚557.9772567749023 disk.sda2.out‚0 disk.sda.in‚ 74.91015625 network.sent‚ 1.4542857e+07" memory_percent‚93.306565284729›(BÇŸª ‚(:” í–éÒÜö gpu.0.gpu‚99.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚302.407' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚60.481399999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0K1N¬(« ‚(: í–éÒ“ƒú network.sent‚ 1.4600309e+07 disk.sda3.in‚ 74.91015625 disk.rbd3.out‚0 disk.sda2.in‚0) proc.memory.availableMB‚ 1096.58203125 proc.cpu.threads‚17 disk.dm-0.in‚ 28671.046875 disk.sda2.out‚0# proc.memory.rssMB‚ 2802.53515625) proc.memory.percent‚17.10531711578369 disk.drbd2165.in‚0" disk.nvme0c0n1.in‚ 28671.046875 disk.sda.in‚ 74.91015625 disk.sda.out‚ 659.8671875 cpu‚25.55119684239217 network.recv‚ 3.154857e+06 disk.drbd2165.out‚0 disk.rbd1.in‚ 0.47314453125 disk.rbd1.out‚ 33.677734375* disk./.usagePercent‚37.446748128810206 disk.rbd0.out‚0 disk.sda1.out‚0 disk.dm-1.out‚ 659.8671875 disk.rbd0.in‚0 disk.sda1.in‚0 disk.sda3.out‚ 659.8671875 disk.rbd3.in‚0' disk.nvme0n1.out‚126414.09912109375 disk.dm-1.in‚ 74.91015625$ memory_percent‚93.30699443817139$ disk./.usageGB‚557.9836082458496$ disk.dm-0.out‚126414.09912109375) disk.nvme0c0n1.out‚126414.09912109375 disk.nvme0n1.in‚ 28671.046875•â­”¬ ‚(:‰ ü–éÒ±üö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚39.0 gpu.0.powerWatts‚110.527' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.1054 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Nþœ­ ‚(:“ ü–éÒ™ÐÁù disk.sda2.in‚0 disk.dm-0.in‚ 28671.46875 disk.sda3.out‚ 663.203125 disk.drbd2165.in‚0 disk.rbd1.out‚ 33.740234375 disk.sda.in‚ 74.91015625! disk.nvme0c0n1.in‚ 28671.46875( disk.nvme0c0n1.out‚126431.6884765625 disk.dm-1.in‚ 74.91015625 disk.sda1.out‚0& disk.nvme0n1.out‚126431.6884765625 disk.sda.out‚ 663.203125$ memory_percent‚93.22190284729004# proc.memory.rssMB‚ 2804.85546875 cpu‚25.55135585795997$ disk./.usageGB‚557.9868202209473 disk.dm-1.out‚ 663.203125 proc.cpu.threads‚17 disk.sda1.in‚0 disk.sda3.in‚ 74.91015625 disk.rbd3.out‚0 disk.sda2.out‚0* proc.memory.percent‚17.119479179382324 network.sent‚ 1.4605405e+07 disk.rbd0.out‚0# disk.dm-0.out‚126431.6884765625 disk.nvme0n1.in‚ 28671.46875( proc.memory.availableMB‚ 1110.5234375) disk./.usagePercent‚37.44696368715404 disk.rbd0.in‚0 disk.rbd1.in‚ 0.47314453125 network.recv‚ 3.158042e+06 disk.drbd2165.out‚0 disk.rbd3.in‚0úæÅP’® ‚(:‡ ‹—éÒÆÜèö gpu.0.gpu‚90.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚259.94' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚51.988 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¶ é¯ ‚(:” ‹—éÒ¬¸›ù" disk.dm-0.out‚126449.853515625 disk.sda1.in‚0 disk.sda3.out‚ 663.73828125% disk.nvme0n1.out‚126449.853515625 disk.rbd1.out‚ 33.740234375 network.sent‚ 1.4610451e+07) disk./.usagePercent‚37.44726142391636 disk.dm-1.out‚ 663.73828125 disk.rbd0.out‚0 disk.sda1.out‚0 disk.nvme0n1.in‚ 28671.6875 disk.drbd2165.in‚0' disk.nvme0c0n1.out‚126449.853515625 disk.sda2.out‚0 proc.cpu.threads‚17 disk.dm-1.in‚ 74.91015625* proc.memory.percent‚17.119479179382324 cpu‚25.551572369218267 disk.dm-0.in‚ 28671.6875 disk.rbd1.in‚ 0.47314453125$ memory_percent‚93.22190284729004 network.recv‚ 3.161637e+06$ disk./.usageGB‚557.9912567138672 disk.sda.out‚ 663.73828125 disk.rbd0.in‚0# proc.memory.rssMB‚ 2804.85546875 disk.sda3.in‚ 74.91015625 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.nvme0c0n1.in‚ 28671.6875 disk.sda2.in‚0 disk.sda.in‚ 74.91015625 disk.drbd2165.out‚0( proc.memory.availableMB‚ 1110.5234375¨pŸ° ‚(:” š—éÒ»Ðèö gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚300.279' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚60.055800000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0hÁ-$± ‚(:™ š—éÒ€¯Ÿù" disk.nvme0c0n1.in‚ 28671.890625 disk.sda2.out‚0 disk.rbd1.in‚ 0.47314453125( proc.memory.availableMB‚ 1110.7734375 disk.dm-0.in‚ 28671.890625 disk.sda1.in‚0' disk.nvme0n1.out‚126457.64306640625 disk.sda2.in‚0 disk.rbd1.out‚ 33.802734375$ memory_percent‚93.22037696838379# proc.memory.rssMB‚ 2804.85546875 disk.rbd0.in‚0 disk.sda3.in‚ 74.91015625 disk.sda3.out‚ 664.140625 disk.rbd3.in‚0 cpu‚25.551708177536455# disk./.usageGB‚557.997932434082$ disk.dm-0.out‚126457.64306640625 disk.drbd2165.in‚0) disk.nvme0c0n1.out‚126457.64306640625 network.recv‚ 3.164854e+06) disk./.usagePercent‚37.44770943710128 disk.sda.out‚ 664.140625 disk.dm-1.in‚ 74.91015625 disk.dm-1.out‚ 664.140625 network.sent‚ 1.4615485e+07 disk.drbd2165.out‚0 disk.nvme0n1.in‚ 28671.890625* proc.memory.percent‚17.119479179382324 disk.sda.in‚ 74.91015625 disk.sda1.out‚0 disk.rbd3.out‚0 disk.rbd0.out‚0 proc.cpu.threads‚17ÙhqC”² ‚(:‰ ©—éÒå÷äö gpu.0.gpu‚90.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚295.976' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚59.1952 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0û,¡®-³ ‚(:¢ ©—éÒ¹¦ù network.sent‚ 1.4619717e+07 disk.dm-1.in‚ 74.91015625 cpu‚25.55198204138793 network.recv‚ 3.16769e+06* disk./.usagePercent‚37.447590393597864 disk.sda.out‚ 664.61328125$ disk.dm-0.out‚126472.88232421875# disk.nvme0c0n1.in‚ 28937.9609375) disk.nvme0c0n1.out‚126472.88232421875 disk.dm-1.out‚ 664.61328125 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda3.out‚ 664.61328125 disk.rbd3.in‚0 disk.drbd2165.out‚0' disk.nvme0n1.out‚126472.88232421875 proc.cpu.threads‚17 disk.dm-0.in‚ 28937.9609375 disk.rbd3.out‚0 disk.rbd1.out‚ 33.802734375* proc.memory.percent‚17.119479179382324 disk.sda.in‚ 74.91015625 disk.rbd0.in‚0 disk.rbd0.out‚0! disk.nvme0n1.in‚ 28937.9609375$ disk./.usageGB‚557.9961585998535 disk.sda3.in‚ 74.91015625 disk.drbd2165.in‚0 disk.sda2.out‚0# memory_percent‚93.2204008102417) proc.memory.availableMB‚ 1110.76953125# proc.memory.rssMB‚ 2804.85546875 disk.sda2.in‚0 disk.rbd1.in‚ 0.47314453125{x©=4´ / _wandbruntime‚12692  _runtime‚12692S߭ǵ Á & lr-AdamW/pg1‚0.0004834599457775849 & lr-AdamW/pg2‚0.0004834599457775849 trainer/global_step‚7649 $ _timestamp‚1.784302508322741e+09 _runtime‚12692.540980441 _step‚627LYÜë̶ Æ _runtime‚12692.540980441 _step‚627 & lr-AdamW/pg1‚0.0004834599457775849 & lr-AdamW/pg2‚0.0004834599457775849 trainer/global_step‚7649 $ _timestamp‚1.784302508322741e+09ój›¸4· / _wandbruntime‚12692  _runtime‚12692’Ø+V(¸ þO WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 131545428 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7184326582374 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7196642156032 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚65288379 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 131555407 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚65288385 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2199524708687872 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2199524734861312 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2205758687752294 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚41342334 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21714496 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚66267022 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7196642156032 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚242 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7185671182696 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2199524708687872 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚41342087 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7197987545600 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚65288379 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚603163235908608 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2205760057780584 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚32439296 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2206721350843904 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7197987545600 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚19627596 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _runtime‚12692.541422625 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2199524734861312 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚21714738 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9285932576768 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2206722722406912 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚19627591 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚603162835378176 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚247 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9285965016064 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚7649 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2206721350843904 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚65288385 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 400530432 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚593877270892544 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚66257049 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2206722722406912 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2198574361169920 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0”± G (¹ ƒP ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2199524708687872 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2199524734861312 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2205758687752294 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚41342334 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21714496 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚66267022 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7196642156032 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚242 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7185671182696 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2199524708687872 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚41342087 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7197987545600 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚65288379 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚603163235908608 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2205760057780584 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚32439296 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2206721350843904 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7197987545600 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚19627596 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _runtime‚12692.541422625 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2199524734861312 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚21714738 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9285932576768 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2206722722406912 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚19627591 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚603162835378176 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚247 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9285965016064 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚7649 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2206721350843904 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚65288385 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 400530432 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚593877270892544 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚66257049 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2206722722406912 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2198574361169920 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 131545428 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7184326582374 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7196642156032 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚65288379 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 131555407 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚65288385 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640ôk#ã4º / _wandbruntime‚12693  _runtime‚12693Ÿ Ë5» †N JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2207001648179712 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚66265527 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _step‚629 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2207000276609536 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚65296591 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 131572105 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚65296585 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚237 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21717251 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 LDDevi( ×ceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2206038863025118 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2198852242948096 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7198912588288 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚603241525159936 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7186594649054 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9287170480128 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2207001648179712 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚66275514 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚593954757300224 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7185250048620 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7198912588288 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2199802709417984 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚41347385 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2199802709417984 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2198852268376064 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚66265527 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7197567191552 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 131572105 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 131562112 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 trainer/global_step‚7649 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2199802735591424 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9287135950848 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚603241927780352 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚19630134 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2206037492996716 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚41347627 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 131562112 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚242 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚65296591 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7197567191552 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚65296585 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 _runtime‚ 12693.9269141 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚593954389209088 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚19630139 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚66275514 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚21717488 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7198912588288 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2199802709417984 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚41347385 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2199802709417984 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2198852268376064 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚66265527 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7197567191552 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 131572105 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 131562112 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 trainer/global_step‚7649 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2199802735591424 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9287135950848 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚603241927780352 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚19630134 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2206037492996716 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚41347627 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 131562112 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚242 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚65296591 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7197567191552 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚65296585 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 _runtime‚ 12693.9269141 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚593954389209088 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚19630139 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚66275514 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚21717488 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚65296585 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚237 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21717251 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2206038863025118 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2198852242948096 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7198912588288 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚603241525159936 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7186594649054 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9287170480128 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2207001648179712 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚66275514 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚593954757300224 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7185250048620 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926õƨ‹4½ / _wandbruntime‚12694  _runtime‚12694ù)µJ ¾ ‡ epoch‚19 % _timestamp‚1.7843025097958684e+09 0loss/FRt2/assignment_loss‚0.5109683443738527 1loss/SRqqt2/detection_loss‚0.5877326804055966 0loss/FBt2/detection_loss‚0.34224491741042584 %loss/total_loss‚7.093515650866488 trainer/global_step‚7649 _step‚630 0loss/FRt1/assignment_loss‚1.4297038970800338 .loss/FRt2/detection_loss‚1.462169872886984 2loss/SRqqt1/assignment_loss‚0.4348734278800425 2loss/FBt1/assignment_loss‚0.013848902886378802 3loss/FBt2/assignment_loss‚0.0038928748141896473 /loss/FRt1/detection_loss‚0.5097612084001826 1loss/SRqqt1/detection_loss‚0.7830288229908253 /loss/FBt1/detection_loss‚0.5935653526699398 _runtime‚12694.012808843 3loss/SRqqt2/assignment_loss‚0.42172534906803666×Sg_¿ Œ %loss/total_loss‚7.093515650866488 trainer/global_step‚7649 _step‚630 0loss/FRt1/assignment_loss‚1.4297038970800338 .loss/FRt2/detection_loss‚1.462169872886984 2loss/SRqqt1/assignment_loss‚0.4348734278800425 2loss/FBt1/assignment_loss‚0.013848902886378802 3loss/FBt2/assignment_loss‚0.0038928748141896473 /loss/FRt1/detection_loss‚0.5097612084001826 1loss/SRqqt1/detection_loss‚0.7830288229908253 /loss/FBt1/detection_loss‚0.5935653526699398 _runtime‚12694.012808843 3loss/SRqqt2/assignment_loss‚0.42172534906803666 epoch‚19 % _timestamp‚1.7843025097958684e+09 0loss/FRt2/assignment_loss‚0.5109683443738527 1loss/SRqqt2/detection_loss‚0.5877326804055966 0loss/FBt2/detection_loss‚0.34224491741042584öµÜräžÀ ‚(:“ ¸—éÒ°Íâö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚264.23' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚52.846000000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Iþo\.Á ‚(:£ ¸—éÒÍܳù disk.sda.out‚ 665.96484375$ disk.dm-0.out‚126486.38330078125( proc.memory.availableMB‚ 1110.7578125# proc.memory.rssMB‚ 2804.85546875 proc.cpu.threads‚17) disk./.usagePercent‚37.44830030249031# disk.nvme0c0n1.in‚ 28938.1796875 disk.dm-1.in‚ 74.91015625 disk.rbd3.out‚0$ memory_percent‚93.22047233581543 disk.rbd1.out‚ 33.865234375 disk.rbd0.out‚0 disk.sda1.out‚0) disk.nvme0c0n1.out‚126486.38330078125! disk.nvme0n1.in‚ 28938.1796875 disk.rbd0.in‚0 disk.sda1.in‚0 disk.rbd3.in‚0 cpu‚25.552115934369983 disk.sda.in‚ 74.91015625 disk.dm-0.in‚ 28938.1796875 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.dm-1.out‚ 665.96484375 disk.rbd1.in‚ 0.47314453125 disk.sda3.in‚ 74.91015625 disk.sda3.out‚ 665.96484375 network.sent‚ 1.4624806e+07$ disk./.usageGB‚558.0067367553711 disk.sda2.in‚0* proc.memory.percent‚17.119479179382324 network.recv‚ 3.171363e+06' disk.nvme0n1.out‚126486.38330078125 disk.sda2.out‚0y¾Ë• ‚(:Š Ç—éÒÊüõö gpu.0.gpu‚100.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚337.146' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚67.4292 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ºmj©+à ‚(:  Ç—éÒÀÂœù disk.sda1.out‚0 disk.sda3.out‚ 669.41015625! disk.nvme0n1.in‚ 28938.6015625 network.recv‚ 3.176424e+06 disk.rbd1.in‚ 0.47314453125 cpu‚25.552369238880086 disk.dm-1.in‚ 74.91015625 disk.dm-0.in‚ 28938.6015625 disk.rbd3.in‚0* proc.memory.percent‚17.119479179382324 disk.rbd0.in‚0 disk.drbd2165.in‚0 disk.rbd3.out‚0 disk.rbd1.out‚ 33.865234375 disk.drbd2165.out‚0# disk.nvme0c0n1.in‚ 28938.6015625 disk.sda2.out‚0 network.sent‚ 1.4688275e+07) disk./.usagePercent‚37.44961413315603$ disk./.usageGB‚558.0263137817383 disk.sda.out‚ 669.41015625 disk.sda1.in‚0& disk.nvme0n1.out‚126516.3916015625 disk.sda2.in‚0 disk.sda.in‚ 74.91015625 disk.dm-1.out‚ 669.41015625 disk.sda3.in‚ 74.91015625( disk.nvme0c0n1.out‚126516.3916015625( proc.memory.availableMB‚ 1110.7578125 disk.rbd0.out‚0# disk.dm-0.out‚126516.3916015625$ memory_percent‚93.22047233581543# proc.memory.rssMB‚ 2804.85546875 proc.cpu.threads‚17ö62ÞÄ ‚(:“ Ö—éÒ´”åö gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚272.542' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚54.50839999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0{÷…@&Å ‚(:› Ö—éҼȃù network.sent‚ 1.4693329e+07 disk.sda1.out‚0 disk.sda3.out‚ 672.3671875* proc.memory.percent‚17.119479179382324) disk./.usagePercent‚37.44991340596356 disk.rbd1.in‚ 0.47314453125 disk.dm-0.in‚ 28938.8046875) proc.memory.availableMB‚ 1110.94140625 cpu‚25.552484840302423 proc.cpu.threads‚17 network.recv‚ 3.179793e+06% disk.nvme0n1.out‚126541.013671875 disk.sda2.out‚0 disk.rbd1.out‚ 33.927734375 disk.sda.in‚ 74.91015625$ disk./.usageGB‚558.0307731628418 disk.sda2.in‚0 disk.dm-1.in‚ 74.91015625 disk.rbd0.in‚0 disk.sda3.in‚ 74.91015625# disk.nvme0c0n1.in‚ 28938.8046875! disk.nvme0n1.in‚ 28938.8046875 disk.sda.out‚ 672.3671875" disk.dm-0.out‚126541.013671875$ memory_percent‚93.21935176849365 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.rbd3.in‚0 disk.rbd3.out‚0# proc.memory.rssMB‚ 2804.85546875 disk.sda1.in‚0 disk.dm-1.out‚ 672.3671875 disk.rbd0.out‚0' disk.nvme0c0n1.out‚126541.013671875sãN&ŸÆ ‚(:” å—éÒшýö gpu.0.gpu‚100.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚333.524' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚66.70479999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0z€V/Ç ‚(:¤ å—éÒñé¤ù disk.dm-1.out‚ 674.01171875 disk.rbd0.out‚0 disk.sda1.in‚0 disk.sda1.out‚0" disk.nvme0n1.in‚29053.43359375 disk.sda2.out‚0# disk./.usageGB‚558.035961151123 disk.dm-1.in‚ 74.91015625 disk.rbd0.in‚0 disk.drbd2165.in‚0( proc.memory.availableMB‚ 1110.9453125* proc.memory.percent‚17.119479179382324 proc.cpu.threads‚17$ memory_percent‚93.21932792663574 disk.rbd1.out‚ 33.927734375$ disk.dm-0.out‚126567.97021484375 disk.sda3.out‚ 674.01171875 disk.drbd2165.out‚0$ disk.nvme0c0n1.in‚29053.43359375' disk.nvme0n1.out‚126567.97021484375 network.sent‚ 1.4698435e+07 network.recv‚ 3.18333e+06 disk.rbd3.out‚0) disk.nvme0c0n1.out‚126567.97021484375 disk.rbd1.in‚ 0.47314453125* disk./.usagePercent‚37.450261576210124 disk.sda.out‚ 674.01171875 disk.sda2.in‚0 disk.sda.in‚ 74.91015625 disk.dm-0.in‚29053.43359375 disk.sda3.in‚ 74.91015625 disk.rbd3.in‚0# proc.memory.rssMB‚ 2804.85546875 cpu‚25.55273527766334öÇV”È ‚(:‰ ô—éÒ¥£°÷ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚282.988' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚56.5976 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0*2¬,É ‚(:¡ ô—éҸȬù disk.rbd0.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda3.in‚ 74.91015625 disk.rbd3.in‚0$ disk.nvme0c0n1.in‚29053.85546875 disk.rbd1.out‚ 33.990234375$ memory_percent‚93.21630001068115 disk.dm-1.out‚ 678.578125 disk.drbd2165.in‚0 disk.drbd2165.out‚0) proc.memory.availableMB‚ 1111.44140625 proc.cpu.threads‚17 disk.rbd0.out‚0" disk.nvme0n1.in‚29053.85546875# proc.memory.rssMB‚ 2804.85546875 cpu‚25.55286898143737* disk./.usagePercent‚37.450332234289576 disk.rbd3.out‚0 disk.rbd1.in‚ 0.47314453125 network.sent‚ 1.4703491e+07) disk.nvme0c0n1.out‚126588.88134765625 disk.dm-0.in‚29053.85546875' disk.nvme0n1.out‚126588.88134765625 disk.sda2.out‚0* proc.memory.percent‚17.119479179382324 network.recv‚ 3.186519e+06 disk.sda.in‚ 74.91015625 disk.dm-1.in‚ 74.91015625$ disk.dm-0.out‚126588.88134765625 disk.sda2.in‚0$ disk./.usageGB‚558.0370140075684 disk.sda.out‚ 678.578125 disk.sda3.out‚ 678.578125ZߥO4Ê / _wandbruntime‚12771  _runtime‚12771’ó ÈË Â & lr-AdamW/pg2‚0.0004809117680506382 trainer/global_step‚7699 % _timestamp‚1.7843025874497676e+09 _runtime‚12771.666948185 _step‚631 & lr-AdamW/pg1‚0.0004809117680506382ò_~ÍÌ Ç & lr-AdamW/pg1‚0.0004809117680506382 & lr-AdamW/pg2‚0.0004809117680506382 trainer/global_step‚7699 % _timestamp‚1.7843025874497676e+09 _runtime‚12771.666948185 _step‚631÷`ŽT4Í / _wandbruntime‚12771  _runtime‚12771^Î þO NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚66692774 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚607097071505408 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚65698685 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 % _timestamp‚1.7843025874508228e+09 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7242958086656 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2220667703280128 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚66682801 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36633600 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚66692774 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚65698679 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2213424771366912 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚255 _runtime‚12771.668156706 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.al»^ï¦located‚19754746 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2213424745193472 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 _step‚632 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚66682801 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚597751591276544 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚19754741 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9345884953600 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚607097476230144 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚260 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚21851936 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2219699013774202 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7230563695482 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚41606682 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7244303476224 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7244303476224 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 132391459 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404724736 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2212468475506688 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2220669074843136 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2213424771366912 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 trainer/global_step‚7699 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 132381480 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚65698679 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2220667703280128 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚41606422 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚597751223185408 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2212468450078720 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2213424745193472 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2219700383802492 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2220669074843136 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7242958086656 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚607097476230144 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚260 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚21851936 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2219699013774202 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7230563695482 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚41606682 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7244303476224 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7244303476224 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 132391459 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404724736 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2212468475506688 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2220669074843136 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2213424771366912 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 trainer/global_step‚7699 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 132381480 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚65698679 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2220667703280128 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚41606422 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚597751223185408 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2212468450078720 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2213424745193472 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2219700383802492 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2220669074843136 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7242958086656 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7242958086656 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2220667703280128 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚66682801 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36633600 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚66692774 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚65698679 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2213424771366912 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚255 _runtime‚12771.668156706 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚19754746 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2213424745193472 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 _step‚632 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚66682801 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚597751591276544 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚19754741 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9345884953600øJÜÕø4Ð / _wandbruntime‚12773  _runtime‚12773qWÅ'Ñ ˆN F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚65706891 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7231487161728 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 % _timestamp‚1.7843025888349464e+09 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2213702745923584 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚21854632 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚597828709593088 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚66691279 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9347073399808 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21854386 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 132408157 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 trainer/global_step‚7699 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2220948000615936 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚607175746366464 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚65706885 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 132398164 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2212746331856896 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚19757289 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36626432 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9347036773376 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2213702745923584 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7243883122176 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚65706891 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚41611921 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2220948000615936 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 _runtime‚12773.052325925 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2220946629045760 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚19757284 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 132408157 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404717568 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚597829077684224 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚66701266 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2219979189047026 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚66701266 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚251 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚41611670 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2212746357284864 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7245228518912 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚66691279¢ª³›ìÒ N ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚19757289 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36626432 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9347036773376 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2213702745923584 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7243883122176 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚65706891 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚41611921 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2220948000615936 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 _runtime‚12773.052325925 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2220946629045760 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚19757284 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 132408157 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404717568 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚597829077684224 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚66701266 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2219979189047026 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚66701266 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚251 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚41611670 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2212746357284864 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7245228518912 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚66691279 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚65706891 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7231487161728 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 % _timestamp‚1.7843025888349464e+09 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2213702745923584 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚21854632 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚597828709593088 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚66691279 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9347073399808 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21854386 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 132408157 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 trainer/global_step‚7699 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2220948000615936 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚607175746366464 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚65706885 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 132398164 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2212746331856896ù¾Îç}4Ó / _wandbruntime‚12773  _runtime‚12773nèì†Ô ‰ % _timestamp‚1.7843025889211638e+09 _step‚634 epoch‚19 3loss/SRqqt1/assignment_loss‚0.40460850253438224 3loss/FBt2/assignment_loss‚0.0003198681111797553 0loss/FRt1/detection_loss‚0.48152550131319605 0loss/FRt1/assignment_loss‚1.4038718948288569 0loss/FBt2/detection_loss‚0.40487496541285023 &loss/total_loss‚6.8770291471547464 3loss/SRqqt2/assignment_loss‚0.38799267533177145 0loss/FBt1/assignment_loss‚0.0081250365907996 /loss/FRt2/detection_loss‚1.3754093519409665 1loss/SRqqt1/detection_loss‚0.7848971067161159 /loss/FBt1/detection_loss‚0.3914246065683211 trainer/global_step‚7699 _runtime‚12773.138087265 0loss/FRt2/assignment_loss‚0.6687154261330467 1loss/SRqqt2/detection_loss‚0.5652642116732606ý ¹RÕ Ž &loss/total_loss‚6.8770291471547464 3loss/SRqqt2/assignment_loss‚0.38799267533177145 0loss/FBt1/assignment_loss‚0.0081250365907996 /loss/FRt2/detection_loss‚1.3754093519409665 1loss/SRqqt1/detection_loss‚0.7848971067161159 /loss/FBt1/detection_loss‚0.3914246065683211 trainer/global_step‚7699 _runtime‚12773.138087265 0loss/FRt2/assignment_loss‚0.6687154261330467 1loss/SRqqt2/detection_loss‚0.5652642116732606 % _timestamp‚1.7843025889211638e+09 _step‚634 epoch‚19 3loss/SRqqt1/assignment_loss‚0.40460850253438224 3loss/FBt2/assignment_loss‚0.0003198681111797553 0loss/FRt1/detection_loss‚0.48152550131319605 0loss/FRt1/assignment_loss‚1.4038718948288569 0loss/FBt2/detection_loss‚0.40487496541285023úc´ýr‘Ö ‚(:† ƒ˜éÒã½êö gpu.0.gpu‚28.0 gpu.0.memory‚9, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚338.21' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚67.642 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0™tù)× ‚(:ž ƒ˜éÒ›ô¤ù disk.dm-1.out‚ 679.44921875 network.sent‚ 1.4708646e+07* disk./.usagePercent‚37.450625106908745$ disk./.usageGB‚558.0413780212402 disk.rbd3.in‚0' disk.nvme0c0n1.out‚126605.466796875% disk.nvme0n1.out‚126605.466796875 disk.rbd1.out‚ 33.990234375 network.recv‚ 3.190395e+06 disk.rbd0.out‚0 disk.dm-0.in‚29054.05859375" disk.nvme0n1.in‚29054.05859375$ memory_percent‚93.21632385253906 disk.drbd2165.in‚0 disk.rbd3.out‚0 disk.rbd1.in‚ 0.47314453125 proc.cpu.threads‚17 disk.rbd0.in‚0 cpu‚25.553060814497467 disk.drbd2165.out‚0 disk.sda.in‚ 74.91015625% proc.memory.availableMB‚ 1111.4375$ disk.nvme0c0n1.in‚29054.05859375 disk.sda2.out‚0 disk.dm-1.in‚ 74.91015625 disk.sda1.in‚0 disk.sda3.in‚ 74.91015625# proc.memory.rssMB‚ 2804.85546875* proc.memory.percent‚17.119479179382324" disk.dm-0.out‚126605.466796875 disk.sda1.out‚0 disk.sda3.out‚ 679.44921875 disk.sda2.in‚0 disk.sda.out‚ 679.44921875WIK)”Ø ‚(:‰ ’˜éÒ¤©Ÿø gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚278.726' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.7452 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0´(=°,Ù ‚(:¡ ’˜éÒ…üöø disk.sda1.in‚0$ disk.nvme0c0n1.in‚29054.48046875) disk.nvme0c0n1.out‚126615.87255859375 disk.rbd1.out‚ 34.052734375 disk.sda1.out‚0" disk.nvme0n1.in‚29054.48046875 disk.sda2.in‚0# memory_percent‚93.2163953781128 disk.drbd2165.out‚0 disk.rbd3.in‚0 cpu‚25.553273274857823 disk.sda.in‚ 74.91015625 disk.sda3.out‚ 679.640625* proc.memory.percent‚17.119479179382324 disk.dm-1.in‚ 74.91015625 disk.sda3.in‚ 74.91015625* disk./.usagePercent‚37.451248741262155 disk.sda.out‚ 679.640625 disk.rbd0.out‚0 disk.rbd3.out‚0 disk.sda2.out‚0) proc.memory.availableMB‚ 1111.42578125 network.sent‚ 1.4771851e+07 disk.drbd2165.in‚0' disk.nvme0n1.out‚126615.87255859375 proc.cpu.threads‚17 network.recv‚ 3.194976e+06 disk.dm-0.in‚29054.48046875# proc.memory.rssMB‚ 2804.85546875$ disk./.usageGB‚558.0506706237793 disk.rbd1.in‚ 0.47314453125 disk.dm-1.out‚ 679.640625 disk.rbd0.in‚0$ disk.dm-0.out‚126615.87255859375õ(.ŸÚ ‚(:” ¡˜éÒéîó÷ gpu.0.gpu‚49.0 gpu.0.memory‚15, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚110.415' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.083000000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0èÙë»Û ‚(:Ž ¡˜éÒ€Øù* disk./.usagePercent‚37.451537773768315 disk.sda.out‚680.0625# disk.dm-0.out‚126633.8095703125 disk.drbd2165.in‚0 disk.rbd3.in‚0 disk.sda2.out‚0 disk.sda.in‚ 74.91015625 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.rbd3.out‚0 disk.nvme0n1.in‚ 29166.59375$ memory_percent‚93.21494102478027( proc.memory.availableMB‚ 1111.6640625 disk.dm-1.out‚680.0625 disk.drbd2165.out‚0 disk.rbd1.out‚ 34.052734375 cpu‚25.55348337909348 disk.sda3.out‚680.0625! disk.nvme0c0n1.in‚ 29166.59375$ disk./.usageGB‚558.0549774169922 disk.sda1.out‚0 network.sent‚ 1.4776897e+07 disk.dm-1.in‚ 74.91015625 disk.dm-0.in‚ 29166.59375 disk.sda3.in‚ 74.91015625( disk.nvme0c0n1.out‚126633.8095703125 proc.cpu.threads‚17 network.recv‚ 3.198565e+06 disk.sda1.in‚0& disk.nvme0n1.out‚126633.8095703125 disk.sda2.in‚0 disk.rbd1.in‚ 0.47314453125# proc.memory.rssMB‚ 2804.85546875* proc.memory.percent‚17.119479179382324%¤žÜ ‚(:“ °˜éÒ‚Ôàø gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚280.405' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚56.08099999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÅÅX+*Ý ‚(:Ÿ °˜éÒ·ýìù network.sent‚ 1.478142e+07) disk.nvme0c0n1.out‚126644.46923828125' disk.nvme0n1.out‚126644.46923828125 disk.sda3.out‚ 680.33203125 proc.cpu.threads‚17 network.recv‚ 3.202008e+06 disk.sda2.out‚0 disk.rbd1.in‚ 0.47314453125 disk.sda.in‚ 74.91015625 disk.sda.out‚ 680.33203125$ memory_percent‚93.21494102478027( proc.memory.availableMB‚ 1111.6640625# proc.memory.rssMB‚ 2804.85546875* disk./.usagePercent‚37.451946105785424 disk.rbd3.out‚0 disk.rbd1.out‚ 34.115234375* proc.memory.percent‚17.119479179382324 disk.rbd3.in‚0 disk.rbd0.in‚0$ disk.dm-0.out‚126644.46923828125 disk.sda1.in‚0 disk.drbd2165.in‚0" disk.nvme0c0n1.in‚ 29166.796875 disk.sda2.in‚0 disk.sda1.out‚0 disk.nvme0n1.in‚ 29166.796875 disk.dm-1.out‚ 680.33203125 disk.rbd0.out‚0 disk.dm-0.in‚ 29166.796875 cpu‚25.55367042535537 disk.sda3.in‚ 74.91015625$ disk./.usageGB‚558.0610618591309 disk.drbd2165.out‚0 disk.dm-1.in‚ 74.91015625¢[/&’Þ ‚(:‡ ¿˜éÒ¦ßÏö gpu.0.gpu‚88.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚300.52' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚60.104 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0;„„ß ‚(:‘ ¿˜éÒñå¹ø disk.rbd3.in‚0 disk.rbd3.out‚0 disk.sda.out‚ 680.52734375 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.out‚0$ memory_percent‚93.21646690368652* disk./.usagePercent‚37.452421255768975 disk.drbd2165.in‚0( proc.memory.availableMB‚ 1111.4140625 cpu‚25.55390517548697 network.sent‚ 1.4786508e+07 disk.dm-1.in‚ 74.91015625! disk.nvme0c0n1.in‚ 29167.21875 disk.sda2.in‚0 disk.rbd1.out‚ 34.115234375 network.recv‚ 3.205639e+06 disk.dm-1.out‚ 680.52734375 disk.dm-0.out‚126655.9140625 disk.sda3.in‚ 74.91015625% disk.nvme0c0n1.out‚126655.9140625# disk.nvme0n1.out‚126655.9140625 proc.cpu.threads‚17 disk.sda.in‚ 74.91015625 disk.drbd2165.out‚0 disk.nvme0n1.in‚ 29167.21875 disk.rbd1.in‚ 0.47314453125# proc.memory.rssMB‚ 2804.85546875 disk.rbd0.in‚0 disk.dm-0.in‚ 29167.21875 disk.sda3.out‚ 680.52734375$ disk./.usageGB‚558.0681419372559 disk.rbd0.out‚0* proc.memory.percent‚17.119479179382324©:äE4à / _wandbruntime‚12850  _runtime‚12850ÖßìÈá  & lr-AdamW/pg1‚0.0004783549203261068 & lr-AdamW/pg2‚0.0004783549203261068 trainer/global_step‚7749 % _timestamp‚1.7843026667277563e+09 _runtime‚12850.944933705 _step‚635dj32Íâ Ç & lr-AdamW/pg1‚0.0004783549203261068 & lr-AdamW/pg2‚0.0004783549203261068 trainer/global_step‚7749 % _timestamp‚1.7843026667277563e+09 _runtime‚12850.944933705 _step‚635û](ÖÀ4ã / _wandbruntime‚12850  _runtime‚12850pYh(ä þO ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2234615427279360 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 133227511 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2226362564415488 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2227324807872512 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2227324781699072 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚263 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚611031300364288 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚601625543569408 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2233639339796110 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 _runtime‚12850.946127458 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚41870768 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚67108553 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 % _timestamp‚1.7843026667287989e+09 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚19881896 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2234614055716352 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚67108553 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2226362538987520 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7278145408912 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 133217532 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚67118526 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2227324781699072 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9405388703744 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚66108985 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7290619406848 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7276800808590 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚19881891 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2234615427279360 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚67118526 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2227324807872512 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2233640709824400 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2234614055716352 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚21989140 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21988877 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 trainer/global_step‚7749 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404724736 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚601625911660544 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 _step‚636k¨ãmM'å ƒP YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 _step‚636 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2234615427279360 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 133227511 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2226362564415488 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2227324807872512 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2227324781699072 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚263 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚611031300364288 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚601625543569408 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2233639339796110 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 _runtime‚12850.946127458 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚41870768 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚67108553 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 % _timestamp‚1.7843026667287989e+09 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚19881896 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2234614055716352 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚67108553 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2226362538987520 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7278145408912 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 133217532 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚67118526 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2227324781699072 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9405388703744 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚66108985 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7290619406848 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7276800808590 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚19881891 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2234615427279360 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚67118526 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2227324807872512 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2233640709824400 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2234614055716352 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚21989140 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21988877 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 trainer/global_step‚7749 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404724736 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 eQDeviceStats﷘ƼMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚601625911660544 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136üQy;ž4æ / _wandbruntime‚12852  _runtime‚12852¼HÑu'ç ˆN D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7291544449536 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚259 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚611109572640768 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 133244209 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚67117031 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 _runtime‚12852.330752463 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚41876010 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7290199052800 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚601703398068224 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚19884434 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7279068875270 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 % _timestamp‚1.7843026681134067e+09 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2233918145040532 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2227602808602624 trainer/global_step‚7749 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 133234216 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2227602808602624 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9406579290112 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2227602782429184 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚66117185 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚19884439 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚66117185 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚611109977358336 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2234894353052160 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚66117191 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2234892981481984 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2226640446193664 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚66117191 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9406542663680 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _step‚637 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36626432 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _step‚637 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36626432 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7291544449536 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚259 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚611109572640768 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 133244209 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚67117031 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 _runtime‚12852.330752463 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚41876010 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7290199052800 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚601703398068224 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚19884434 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7279068875270 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 % _timestamp‚1.7843026681134067e+09 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2233918145040532 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2227602808602624 trainer/global_step‚7749 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 133234216 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2227602808602624 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9406579290112 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2227602782429184 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚66117185 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚19884439 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚66117185 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚611109977358336 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640ý¶*Rb4é / _wandbruntime‚12852  _runtime‚12852Ò‡µ ê … 1loss/SRqqt1/detection_loss‚0.8723394628745978 1loss/SRqqt2/detection_loss‚0.6425329770202657 /loss/FBt2/detection_loss‚0.5251046657133446 %loss/total_loss‚7.575784213762573 epoch‚19 2loss/SRqqt2/assignment_loss‚0.4500420248729306 2loss/FBt1/assignment_loss‚0.000361393168716106 3loss/FBt2/assignment_loss‚0.0006780409854312179 /loss/FRt1/detection_loss‚0.5136856003304697 trainer/global_step‚7749 _step‚638 /loss/FRt2/detection_loss‚1.6021640700271096 /loss/FBt1/detection_loss‚0.4639147448285324 % _timestamp‚1.7843026682001772e+09 _runtime‚12852.417103462 0loss/FRt1/assignment_loss‚1.4472828674294593 0loss/FRt2/assignment_loss‚0.5894038025143875 1loss/SRqqt1/assignment_loss‚0.468274563997328 ¾™ë Š _runtime‚12852.417103462 0loss/FRt1/assignment_loss‚1.4472828674294593 0loss/FRt2/assignment_loss‚0.5894038025143875 1loss/SRqqt1/assignment_loss‚0.468274563997328 1loss/SRqqt1/detection_loss‚0.8723394628745978 1loss/SRqqt2/detection_loss‚0.6425329770202657 /loss/FBt2/detection_loss‚0.5251046657133446 %loss/total_loss‚7.575784213762573 epoch‚19 2loss/SRqqt2/assignment_loss‚0.4500420248729306 2loss/FBt1/assignment_loss‚0.000361393168716106 3loss/FBt2/assignment_loss‚0.0006780409854312179 /loss/FRt1/detection_loss‚0.5136856003304697 trainer/global_step‚7749 _step‚638 /loss/FRt2/detection_loss‚1.6021640700271096 /loss/FBt1/detection_loss‚0.4639147448285324 % _timestamp‚1.7843026682001772e+09þ õ,…Ÿì ‚(:” ΘéÒê¦ñö gpu.0.gpu‚90.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚281.431' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚56.286199999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0c¬í ‚(: ΘéÒÁȪù disk.sda2.in‚0) disk./.usagePercent‚37.45277685023404$ disk./.usageGB‚558.0734405517578 disk.rbd0.out‚0 disk.sda1.in‚0 disk.rbd3.out‚0 disk.rbd1.in‚ 0.47314453125 disk.nvme0n1.in‚ 29167.4375& disk.nvme0n1.out‚126670.9912109375 network.sent‚ 1.47921e+07 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.drbd2165.out‚0 cpu‚25.554091080316955 disk.sda.out‚ 680.703125 disk.rbd0.in‚0 disk.dm-0.in‚ 29167.4375 disk.sda2.out‚0 network.recv‚ 3.209047e+06 disk.dm-1.in‚ 74.91015625 disk.sda3.in‚ 74.91015625 disk.rbd1.out‚ 34.177734375# memory_percent‚93.2163953781128 proc.cpu.threads‚17# disk.dm-0.out‚126670.9912109375 disk.sda3.out‚ 680.703125 disk.sda.in‚ 74.91015625 disk.dm-1.out‚ 680.703125 disk.nvme0c0n1.in‚ 29167.4375( disk.nvme0c0n1.out‚126670.9912109375) proc.memory.availableMB‚ 1111.42578125# proc.memory.rssMB‚ 2804.85546875 disk.rbd3.in‚0* proc.memory.percent‚17.119479179382324OI«žî ‚(:“ ݘéÒƒ™ßø gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚283.847' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚56.76939999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Úë&þ/ï ‚(:¤ ݘéÒÇòù disk.rbd0.out‚0 disk.drbd2165.out‚0 disk.rbd3.in‚0 disk.rbd3.out‚0" disk.nvme0n1.in‚29272.93359375# disk./.usageGB‚558.080451965332' disk.nvme0n1.out‚126683.88623046875 disk.sda2.in‚0 network.recv‚ 3.213995e+06 disk.dm-1.in‚ 74.91015625 disk.dm-0.in‚29272.93359375 disk.sda1.in‚0$ memory_percent‚93.21660995483398 network.sent‚ 1.4855353e+07 disk.sda.in‚ 74.91015625* proc.memory.percent‚17.119479179382324 disk.rbd1.out‚ 34.177734375$ disk.dm-0.out‚126683.88623046875 disk.sda1.out‚0 disk.drbd2165.in‚0) disk./.usagePercent‚37.45324739208197 disk.rbd1.in‚ 0.50830078125 disk.rbd0.in‚0' proc.memory.availableMB‚ 1111.390625# proc.memory.rssMB‚ 2804.85546875$ disk.nvme0c0n1.in‚29272.93359375 disk.sda2.out‚0 cpu‚25.554355486254604 proc.cpu.threads‚17 disk.sda.out‚ 681.01953125 disk.dm-1.out‚ 681.01953125 disk.sda3.in‚ 74.91015625 disk.sda3.out‚ 681.01953125) disk.nvme0c0n1.out‚126683.88623046875ßtk”ð ‚(:‰ ì˜éÒ½ýöö gpu.0.gpu‚90.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚281.966' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚56.3932 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0~é¦#ñ ‚(:˜ ì˜éÒ¶†™ù disk.nvme0n1.in‚ 29273.34375 disk.sda.out‚ 681.22265625( disk.nvme0c0n1.out‚126696.8310546875 disk.rbd1.out‚ 34.240234375 disk.dm-1.in‚ 74.91015625! disk.nvme0c0n1.in‚ 29273.34375 proc.cpu.threads‚17 network.sent‚ 1.4859959e+07$ disk./.usageGB‚558.0864791870117 disk.sda2.in‚0# memory_percent‚93.2166576385498( proc.memory.availableMB‚ 1111.3828125# proc.memory.rssMB‚ 2804.85546875 disk.sda.in‚ 74.91015625 disk.dm-1.out‚ 681.22265625# disk.dm-0.out‚126696.8310546875 disk.sda1.out‚0 disk.sda3.in‚ 74.91015625 disk.drbd2165.out‚0* proc.memory.percent‚17.119479179382324 disk.rbd0.in‚0) disk./.usagePercent‚37.45365188398607 disk.dm-0.in‚ 29273.34375 disk.sda1.in‚0& disk.nvme0n1.out‚126696.8310546875 disk.sda2.out‚0 cpu‚25.55452759146118 network.recv‚ 3.217249e+06 disk.rbd0.out‚0 disk.sda3.out‚ 681.22265625 disk.drbd2165.in‚0 disk.rbd1.in‚ 0.50830078125 disk.rbd3.in‚0 disk.rbd3.out‚0H¹”ò ‚(:‰ û˜éÒ²™©ù gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚290.327' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚58.0654 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0L…d#ó ‚(:˜ û˜éÒÇ™Åû$ memory_percent‚93.21668148040771) proc.memory.availableMB‚ 1111.37890625 disk.rbd3.in‚0 disk.sda2.in‚0 disk.sda.out‚ 681.5390625 disk.sda1.out‚0 proc.cpu.threads‚17) disk.nvme0c0n1.out‚126708.04345703125' disk.nvme0n1.out‚126708.04345703125 disk.sda3.in‚ 74.91015625# proc.memory.rssMB‚ 2804.85546875 disk.rbd1.in‚ 0.50830078125 disk.rbd1.out‚ 34.240234375 disk.rbd0.out‚0 cpu‚25.554744848179162 disk.sda2.out‚0 network.sent‚ 1.4865596e+07 disk.rbd3.out‚0 disk.drbd2165.out‚0 disk.nvme0n1.in‚ 29273.5625 disk.sda.in‚ 74.91015625 disk.rbd0.in‚0 disk.sda1.in‚0 disk.sda3.out‚ 681.5390625* proc.memory.percent‚17.119479179382324 network.recv‚ 3.220657e+06 disk.nvme0c0n1.in‚ 29273.5625 disk.dm-0.in‚ 29273.5625$ disk.dm-0.out‚126708.04345703125) disk./.usagePercent‚37.45409426500524$ disk./.usageGB‚558.0930709838867 disk.drbd2165.in‚0 disk.dm-1.in‚ 74.91015625 disk.dm-1.out‚ 681.5390625ûÀnö”ô ‚(:‰ Š™éÒøçì÷ gpu.0.gpu‚90.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚286.263' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚57.2526 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0xj© 'õ ‚(:œ Š™éÒ»¥ýø disk.dm-1.out‚ 681.7109375 disk.rbd0.out‚0$ disk.dm-0.out‚126721.36376953125 disk.drbd2165.in‚0* proc.memory.percent‚17.119479179382324 disk.drbd2165.out‚0) disk.nvme0c0n1.out‚126721.36376953125) proc.memory.availableMB‚ 1111.37890625 disk.rbd0.in‚0 disk.sda3.in‚ 74.91015625 disk.rbd3.in‚0# proc.memory.rssMB‚ 2804.85546875 disk.rbd1.out‚ 34.318359375 disk.sda.in‚ 74.91015625" disk.nvme0c0n1.in‚ 29273.984375 network.recv‚ 3.223974e+06 disk.sda1.in‚0 disk.sda3.out‚ 681.7109375 disk.sda2.out‚0$ memory_percent‚93.21668148040771 cpu‚25.55490447973676 disk.dm-0.in‚ 29273.984375 disk.sda1.out‚0 proc.cpu.threads‚17 network.sent‚ 1.487011e+07) disk./.usagePercent‚37.45455175046892 disk.dm-1.in‚ 74.91015625 disk.nvme0n1.in‚ 29273.984375' disk.nvme0n1.out‚126721.36376953125 disk.sda2.in‚0$ disk./.usageGB‚558.0998878479004 disk.rbd1.in‚ 0.50830078125 disk.sda.out‚ 681.7109375 disk.rbd3.out‚0'ü…éŸö ‚(:” ™™éÒ„Ùíö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚294.967' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚58.993399999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0–¼›,÷ ‚(:¡ ™™éÒ·¶ù$ disk./.usageGB‚558.1042976379395 disk.dm-1.out‚ 682.02734375" disk.nvme0n1.in‚29433.72265625% disk.nvme0n1.out‚126732.755859375 disk.rbd1.out‚ 34.318359375 disk.sda.in‚ 74.91015625 disk.rbd0.in‚0 disk.rbd0.out‚0) disk./.usagePercent‚37.45484769517851 disk.sda.out‚ 682.02734375 disk.sda3.out‚ 682.02734375 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.rbd1.in‚ 0.50830078125) proc.memory.availableMB‚ 1111.37109375 disk.dm-1.in‚ 74.91015625 disk.sda1.in‚0$ disk.nvme0c0n1.in‚29433.72265625 disk.sda2.in‚0 cpu‚25.555071458007543 proc.cpu.threads‚17 network.recv‚ 3.227556e+06 disk.dm-0.in‚29433.72265625" disk.dm-0.out‚126732.755859375 disk.rbd3.in‚0' disk.nvme0c0n1.out‚126732.755859375$ memory_percent‚93.21672916412354 disk.sda1.out‚0 disk.sda2.out‚0# proc.memory.rssMB‚ 2804.85546875 disk.sda3.in‚ 74.91015625* proc.memory.percent‚17.119479179382324 network.sent‚ 1.4875688e+07 disk.rbd3.out‚0jÅH 4ø / _wandbruntime‚12930  _runtime‚12930ËQÂÉù à % _timestamp‚1.7843027459879127e+09 _runtime‚12930.20505626 _step‚639 ' lr-AdamW/pg1‚0.00047578956682554197 ' lr-AdamW/pg2‚0.00047578956682554197 trainer/global_step‚7799#žìÎú È ' lr-AdamW/pg1‚0.00047578956682554197 ' lr-AdamW/pg2‚0.00047578956682554197 trainer/global_step‚7799 % _timestamp‚1.7843027459879127e+09 _runtime‚12930.20505626 _step‚639ÿž×zŒ4û / _wandbruntime‚12930  _runtime‚12930¯V›ñü þO P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚614964642968576 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2240256627896320 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7335589947904 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚605500232044544 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36633600 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚66519279 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7335589947904 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚67544278 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 aLDeviceStatsMonitor.on_train_batch_start/requeste"›ÃÑ…"d_bytes.large_pool.allocated‚2240256653324288 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚20009041 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9464815648768 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2247579665818018 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2241224818204672 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404724736 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚66519285 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2248561779715584 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚42135496 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2241224818204672 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2248561779715584 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22126192 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2248560408152576 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 134063563 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚66519285 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚20009046 _step‚640 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7336935337472 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22126450 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7323037921698 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2241224844378112 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚614965047693312 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 % _timestamp‚1.7843027459888759e+09Ûˆœy (ý ƒP JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚263 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚605499863953408 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22126192 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2248560408152576 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 134063563 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚66519285 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚20009046 _step‚640 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7336935337472 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22126450 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7323037921698 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2241224844378112 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚614965047693312 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 % _timestamp‚1.7843027459888759e+09 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚614964642968576 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2240256627896320 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7335589947904 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚605500232044544 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36633600 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚66519279 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7335589947904 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚67544278 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2240256653324288 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚20009041 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9464815648768 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2247579665818018 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2241224818204672 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404724736 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚66519285 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2248561779715584 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚42135496 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2241224818204672 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2248561779715584 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 134080261 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚66527491 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚20011589 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚20011584 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚605577718452224 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9466006105088 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚66527485 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7323961387944 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚67542783 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚67552770 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚255 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 % _timestamp‚1.7843027473742888e+09 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7337860380160 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚67552770 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2241502818934784 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 134080261 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2240534535102464 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚66527491 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚42140471 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 _step‚641 trainer/global_step‚7799 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2248839333918208 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2241502845108224 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2247859841090842 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2247858471062440 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7336514983424 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚22129137 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚67542783 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚42140726 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚605577350361088 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 _runtime‚12931.59161188 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9465969478656 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2241502818934784 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7325305988378 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7336514983424 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 134070268 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987^ðé²€!ŒN N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9465969478656 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2241502818934784 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7325305988378 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7336514983424 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 134070268 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36626432 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2248840705488384 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2241502845108224 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚615043724557312 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 2958ãŸNa81216 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 134080261 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚66527491 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚20011589 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚20011584 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚605577718452224 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9466006105088 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚66527485 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7323961387944 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚67542783 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚67552770 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚255 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 % _timestamp‚1.7843027473742888e+09 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7337860380160 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚67552770 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2241502818934784 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 134080261 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2240534535102464 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚66527491 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚42140471 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 _step‚641 trainer/global_step‚7799 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2248839333918208 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2241502845108224 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2247859841090842 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2247858471062440 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7336514983424 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚22129137 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚67542783 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚42140726 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚605577350361088 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 _runtime‚12931.59161188 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008<!4!/ _wandbruntime‚12931  _runtime‚12931º)7 ‚!† 2loss/SRqqt1/assignment_loss‚0.5049698215992389 % _timestamp‚1.7843027474602935e+09 0loss/FRt1/assignment_loss‚1.4533955106290721 1loss/FRt2/assignment_loss‚0.49939583673089893 2loss/FBt2/assignment_loss‚0.018484341850538613 1loss/SRqqt1/detection_loss‚0.7952440896560293 /loss/FBt2/detection_loss‚0.4439563062172383 %loss/total_loss‚7.458783194566066 epoch‚19 _runtime‚12931.677210371 /loss/FRt1/detection_loss‚0.4816707723504716 /loss/FRt2/detection_loss‚1.4201655478788737 1loss/SRqqt2/detection_loss‚0.5606990391856216 /loss/FBt1/detection_loss‚0.8232504241459537 _step‚642 2loss/SRqqt2/assignment_loss‚0.4039684007317327 2loss/FBt1/assignment_loss‚0.053583103590395975 trainer/global_step‚7799JÓ`ƒ!‹ 2loss/FBt2/assignment_loss‚0.018484341850538613 1loss/SRqqt1/detection_loss‚0.7952440896560293 /loss/FBt2/detection_loss‚0.4439563062172383 %loss/total_loss‚7.458783194566066 epoch‚19 _runtime‚12931.677210371 /loss/FRt1/detection_loss‚0.4816707723504716 /loss/FRt2/detection_loss‚1.4201655478788737 1loss/SRqqt2/detection_loss‚0.5606990391856216 /loss/FBt1/detection_loss‚0.8232504241459537 _step‚642 2loss/SRqqt2/assignment_loss‚0.4039684007317327 2loss/FBt1/assignment_loss‚0.053583103590395975 trainer/global_step‚7799 2loss/SRqqt1/assignment_loss‚0.5049698215992389 % _timestamp‚1.7843027474602935e+09 0loss/FRt1/assignment_loss‚1.4533955106290721 1loss/FRt2/assignment_loss‚0.49939583673089893‚>Y„!‚(:’ ¨™éÒ§´Üö gpu.0.gpu‚99.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚297.04' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚59.40800000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0žÒkÐ,…!‚(:¡ ¨™éÒóí­ù disk.rbd3.out‚0# proc.memory.rssMB‚ 2804.85546875 disk.dm-1.out‚ 682.2578125) disk.nvme0c0n1.out‚126741.73193359375 disk.sda.out‚ 682.2578125 disk.sda3.out‚ 682.2578125$ disk./.usageGB‚558.1134986877441 disk.rbd0.out‚0 disk.sda.in‚ 74.91015625) proc.memory.availableMB‚ 1111.07421875 proc.cpu.threads‚17 network.recv‚ 3.230783e+06 disk.sda1.in‚0 disk.drbd2165.in‚0 disk.rbd3.in‚0 cpu‚25.555296397004927* disk./.usagePercent‚37.455465185351095 disk.dm-0.in‚ 29433.9609375 disk.drbd2165.out‚0# disk.nvme0c0n1.in‚ 29433.9609375 disk.sda2.in‚0# memory_percent‚93.2185411453247* proc.memory.percent‚17.119479179382324$ disk.dm-0.out‚126741.73193359375! disk.nvme0n1.in‚ 29433.9609375' disk.nvme0n1.out‚126741.73193359375 network.sent‚ 1.4880783e+07 disk.dm-1.in‚ 74.91015625 disk.rbd0.in‚0 disk.sda1.out‚0 disk.sda2.out‚0 disk.sda3.in‚ 74.91015625 disk.rbd1.in‚ 0.50830078125 disk.rbd1.out‚ 34.380859375Ûª¸QŸ†!‚(:” ·™éÒ¼åùö gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚269.062' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚53.812400000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0/^e)‡!‚(:ž ·™éÒѬ©ù disk.sda3.in‚ 75.26953125 disk.sda3.out‚ 682.59375' disk.nvme0n1.out‚126757.82861328125 disk.rbd1.out‚ 34.380859375 disk.rbd3.out‚0 disk.rbd0.out‚0$ memory_percent‚93.22006702423096$ disk.nvme0c0n1.in‚29434.36328125 disk.dm-0.in‚29434.36328125$ disk.dm-0.out‚126757.82861328125 disk.sda1.in‚0 disk.drbd2165.in‚0" disk.nvme0n1.in‚29434.36328125 disk.rbd0.in‚0 disk.rbd1.in‚ 0.50830078125 network.recv‚ 3.235527e+06) disk./.usagePercent‚37.45574141748054$ disk./.usageGB‚558.1176147460938 disk.rbd3.in‚0 proc.cpu.threads‚17 disk.sda2.in‚0) proc.memory.availableMB‚ 1110.82421875* proc.memory.percent‚17.119479179382324 disk.sda2.out‚0 disk.sda.in‚ 75.26953125 disk.drbd2165.out‚0 cpu‚25.555463692560235 network.sent‚ 1.4943756e+07 disk.sda.out‚ 682.59375 disk.dm-1.out‚ 682.59375# proc.memory.rssMB‚ 2804.85546875) disk.nvme0c0n1.out‚126757.82861328125 disk.dm-1.in‚ 75.26953125 disk.sda1.out‚0œ¾ ”ˆ!‚(:‰ Æ™éÒØØáö gpu.0.gpu‚99.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚325.841' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚65.1682 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ðœVQ&‰!‚(:› Æ™éÒÀ¶ƒù cpu‚25.555689404625788 network.sent‚ 1.4948917e+07# disk.nvme0n1.out‚126765.5859375 disk.dm-0.in‚29434.56640625$ memory_percent‚93.21873188018799* proc.memory.percent‚17.119479179382324) disk./.usagePercent‚37.45603249804697% disk.nvme0c0n1.out‚126765.5859375 disk.rbd3.out‚0 disk.sda.in‚ 75.26953125 disk.dm-1.in‚ 75.26953125 disk.dm-0.out‚126765.5859375 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd2165.in‚0 network.recv‚ 3.238625e+06 disk.sda.out‚ 682.81640625 disk.dm-1.out‚ 682.81640625 disk.rbd0.in‚0 disk.rbd0.out‚0) proc.memory.availableMB‚ 1111.04296875 proc.cpu.threads‚17 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd1.out‚ 34.443359375$ disk.nvme0c0n1.in‚29434.56640625 disk.sda3.in‚ 75.26953125 disk.sda3.out‚ 682.81640625 disk.rbd3.in‚0$ disk./.usageGB‚558.1219520568848" disk.nvme0n1.in‚29434.56640625 disk.rbd1.in‚ 0.50830078125 disk.drbd2165.out‚0# proc.memory.rssMB‚ 2804.85546875›I1”Š!‚(:‰ Õ™éÒ³žéö gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚263.646' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.7292 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ðS»30‹!‚(:¥ Õ™éÒþúù disk.sda1.in‚0 disk.rbd1.out‚ 34.443359375 disk.sda.in‚ 75.26953125) proc.memory.availableMB‚ 1110.74609375 cpu‚25.555874555821706# disk.nvme0c0n1.in‚ 29490.6796875 network.recv‚ 3.241945e+06 disk.dm-0.in‚ 29490.6796875$ disk.dm-0.out‚126776.22314453125# proc.memory.rssMB‚ 2804.85546875) disk.nvme0c0n1.out‚126776.22314453125 disk.sda2.out‚0 disk.rbd0.out‚0 network.sent‚ 1.4954007e+07 disk.sda3.in‚ 75.26953125 disk.drbd2165.out‚0 disk.rbd3.out‚0! disk.nvme0n1.in‚ 29490.6796875 disk.rbd1.in‚ 0.50830078125 proc.cpu.threads‚17* disk./.usagePercent‚37.456660740535995 disk.dm-1.out‚ 683.05859375* proc.memory.percent‚17.119479179382324 disk.drbd2165.in‚0 disk.rbd3.in‚0 disk.sda2.in‚0 disk.dm-1.in‚ 75.26953125 disk.sda1.out‚0 disk.rbd0.in‚0$ memory_percent‚93.22054386138916 disk.sda.out‚ 683.05859375 disk.sda3.out‚ 683.05859375' disk.nvme0n1.out‚126776.22314453125$ disk./.usageGB‚558.1313133239746°åˆÌžŒ!‚(:“ ä™éÒ«¥€÷ gpu.0.gpu‚99.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚343.042' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚68.60839999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0$›Ks'!‚(:œ ä™éÒåÁßù disk.dm-1.in‚ 75.2734375& disk.nvme0n1.out‚126789.4267578125# disk./.usageGB‚558.135555267334 disk.rbd1.in‚ 0.50830078125 disk.sda.out‚ 683.5234375 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.dm-0.in‚29577.42578125$ disk.nvme0c0n1.in‚29577.42578125 disk.rbd1.out‚ 34.521484375 network.recv‚ 3.24472e+06 disk.sda.in‚ 75.2734375# disk.dm-0.out‚126789.4267578125 disk.sda3.out‚ 683.5234375" disk.nvme0n1.in‚29577.42578125 network.sent‚ 1.4958135e+07* disk./.usagePercent‚37.456945420914074 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd2165.out‚0 disk.rbd3.out‚0( proc.memory.availableMB‚ 1110.9921875# proc.memory.rssMB‚ 2804.85546875 disk.sda3.in‚ 75.2734375 disk.drbd2165.in‚0 disk.rbd3.in‚0 disk.sda2.in‚0$ memory_percent‚93.21904182434082 disk.sda2.out‚0 cpu‚25.556076366205954 disk.dm-1.out‚ 683.5234375( disk.nvme0c0n1.out‚126789.4267578125* proc.memory.percent‚17.119479179382324 proc.cpu.threads‚17#¦ƒI4Ž!/ _wandbruntime‚13009  _runtime‚1300958(-Ç!Á & lr-AdamW/pg1‚0.0004732158723168054 & lr-AdamW/pg2‚0.0004732158723168054 trainer/global_step‚7849 $ _timestamp‚1.784302825017753e+09 _runtime‚13009.234920995 _step‚643N„\Ì!Æ _runtime‚13009.234920995 _step‚643 & lr-AdamW/pg1‚0.0004732158723168054 & lr-AdamW/pg2‚0.0004732158723168054 trainer/global_step‚7849 $ _timestamp‚1.784302825017753e+09ƒ?ô 4‘!/ _wandbruntime‚13009  _runtime‚13009ê>Fê(’!þO C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2255124880883712 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚66929585 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7370619635128 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 _runtime‚13009.236277246 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7383251268096 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2262506760588800 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2255124880883712 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7381905878528 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2255124854710272 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚66929585 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36633600 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9524223518720 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 _step‚644 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7383251268096 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚67960057 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7381905878528 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404724736 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚618898812580864 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚66929579 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚609374184337408 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2262506760588800 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22263251 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0Ù‡L“!ƒP ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 134889636 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7369275034806 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22263251 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2255124880883712 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚66929585 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7370619635128 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 _runtime‚13009.236277246 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7383251268096 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2262506760588800 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2255124880883712 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7381905878528 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2255124854710272 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚66929585 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36633600 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9524223518720 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 _step‚644 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7383251268096 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚67960057 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7381905878528 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404724736 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚618898812580864 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚66929579 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚609374184337408 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2262506760588800 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚42404718 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7384176310784 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 % _timestamp‚1.7843028264032543e+09 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚67968535 _runtime‚13010.620590512 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9525407051776 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2254428598583296 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7370198501052 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚67968535 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7371543101486 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 _step‚645 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚66937791 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2255402881613824 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚42404971 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚20138739 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚248 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2262787057924608 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2255402855440384 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 134916313 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚66937785 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 trainer/global_step‚7849 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚618977077796864 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚67978522 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚20138734 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872r·'–!N J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 134906320 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22265984 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚609452038836224 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2261798797084348 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 134906320 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚22266232 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚42404718 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7384176310784 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 % _timestamp‚1.7843028264032543e+09 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚67968535 _runtime‚13010.620590512 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9525407051776 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2254428598583296 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7370198501052 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚67968535 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7371543101486 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 _step‚645 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚66937791 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2255402881613824 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚42404971 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚20138739 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚248 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2262787057924608 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2255402855440384 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 134916313 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚66937785 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 trainer/global_step‚7849 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚618977077796864 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚67978522 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚20138734 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9525443678208 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440…[°Ì¼4—!/ _wandbruntime‚13010  _runtime‚130103½¦‘ ˜!… 0loss/FRt1/assignment_loss‚1.4810325264690867 1loss/FBt2/assignment_loss‚0.00474095825569198 .loss/FRt2/detection_loss‚1.412284110615845 1loss/SRqqt2/detection_loss‚0.6431461306949685 /loss/FBt1/detection_loss‚0.4208182982867584 0loss/FRt2/assignment_loss‚0.5209064649777774 2loss/SRqqt1/assignment_loss‚0.4341067334968259 3loss/SRqqt2/assignment_loss‚0.42910886035291557 2loss/FBt1/assignment_loss‚0.015226625701403762 1loss/SRqqt1/detection_loss‚0.7596305784787324 _step‚646 /loss/FRt1/detection_loss‚0.5553725843808685 0loss/FBt2/detection_loss‚0.37213140513239334 epoch‚19 % _timestamp‚1.7843028264904578e+09 %loss/total_loss‚7.048505276843267 trainer/global_step‚7849 _runtime‚13010.7073670998›Ó)™!Š 0loss/FRt2/assignment_loss‚0.5209064649777774 2loss/SRqqt1/assignment_loss‚0.4341067334968259 3loss/SRqqt2/assignment_loss‚0.42910886035291557 2loss/FBt1/assignment_loss‚0.015226625701403762 1loss/SRqqt1/detection_loss‚0.7596305784787324 _step‚646 /loss/FRt1/detection_loss‚0.5553725843808685 0loss/FBt2/detection_loss‚0.37213140513239334 epoch‚19 % _timestamp‚1.7843028264904578e+09 %loss/total_loss‚7.048505276843267 trainer/global_step‚7849 _runtime‚13010.707367099 0loss/FRt1/assignment_loss‚1.4810325264690867 1loss/FBt2/assignment_loss‚0.00474095825569198 .loss/FRt2/detection_loss‚1.412284110615845 1loss/SRqqt2/detection_loss‚0.6431461306949685 /loss/FBt1/detection_loss‚0.4208182982867584†8"¶O’š!‚(:‡ ó™éÒ¤Àë÷ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚269.59' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.918 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0žä̓/›!‚(:¤ ó™éÒ‘Æ…ù disk.sda2.in‚0) disk./.usagePercent‚37.45685505025448 disk.rbd0.out‚0 disk.sda2.out‚0$ disk./.usageGB‚558.1342086791992 disk.sda1.in‚0' disk.nvme0n1.out‚126801.84130859375 disk.sda3.in‚ 75.2734375) proc.memory.availableMB‚ 1111.17578125# proc.memory.rssMB‚ 2804.85546875* proc.memory.percent‚17.119479179382324 cpu‚25.556187231122095 disk.drbd2165.out‚0 disk.dm-0.in‚29577.62890625 disk.sda1.out‚0 network.sent‚ 1.4963246e+07) disk.nvme0c0n1.out‚126801.84130859375 disk.rbd1.in‚ 0.50830078125 disk.rbd0.in‚0$ disk.dm-0.out‚126801.84130859375$ disk.nvme0c0n1.in‚29577.62890625 disk.rbd1.out‚ 34.521484375 disk.sda.out‚ 683.87109375 disk.dm-1.out‚ 683.87109375$ memory_percent‚93.21792125701904 network.recv‚ 3.248437e+06 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.sda.in‚ 75.2734375 disk.dm-1.in‚ 75.2734375 proc.cpu.threads‚17 disk.sda3.out‚ 683.87109375 disk.drbd2165.in‚0" disk.nvme0n1.in‚29577.62890625«×jQ•œ!‚(:Š ‚šéÒ÷Ï÷ö gpu.0.gpu‚100.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚342.514' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚68.5028 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0­‰+!‚(:  ‚šéÒµò²ù$ disk.dm-0.out‚126820.93408203125* proc.memory.percent‚17.119479179382324 network.recv‚ 3.253e+06* disk./.usagePercent‚37.457369369390776 disk.sda3.in‚ 75.2734375 disk.drbd2165.out‚0' disk.nvme0n1.out‚126820.93408203125 disk.sda.out‚ 686.22265625 disk.sda1.out‚0 cpu‚25.556389254670304 disk.rbd3.out‚0# proc.memory.rssMB‚ 2804.85546875 network.sent‚ 1.5026198e+07 disk.rbd3.in‚0) disk.nvme0c0n1.out‚126820.93408203125" disk.nvme0n1.in‚29578.05078125 disk.sda.in‚ 75.2734375 disk.dm-1.in‚ 75.2734375$ memory_percent‚93.21794509887695' proc.memory.availableMB‚ 1111.171875 proc.cpu.threads‚17$ disk./.usageGB‚558.1418724060059 disk.drbd2165.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-0.in‚29578.05078125 disk.dm-1.out‚ 686.22265625 disk.rbd0.out‚0 disk.sda1.in‚0 disk.sda3.out‚ 686.22265625$ disk.nvme0c0n1.in‚29578.05078125 disk.rbd1.in‚ 0.50830078125 disk.rbd1.out‚ 34.583984375 disk.rbd0.in‚0ê>|딞!‚(:‰ ‘šéÒòšñö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚263.646' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.7292 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚00GhÅ+Ÿ!‚(:  ‘šéÒíþ†ù disk.drbd2165.out‚0 disk.rbd3.in‚0) disk.nvme0c0n1.out‚126837.42724609375 disk.sda.out‚ 688.1796875 disk.drbd2165.in‚0* proc.memory.percent‚17.119479179382324 network.sent‚ 1.5031295e+07 disk.rbd1.in‚ 0.50830078125 disk.dm-1.out‚ 688.1796875$ disk./.usageGB‚558.1459808349609 disk.sda.in‚ 75.2734375$ disk.dm-0.out‚126837.47412109375 disk.sda3.in‚ 75.2734375 disk.sda3.out‚ 688.1796875( proc.memory.availableMB‚ 1111.4140625) disk./.usagePercent‚37.45764508950515" disk.nvme0n1.in‚29590.48828125' disk.nvme0n1.out‚126837.43505859375# proc.memory.rssMB‚ 2804.85546875 cpu‚25.556478956002866 network.recv‚ 3.256619e+06 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-1.in‚ 75.2734375 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda1.in‚0 disk.sda1.out‚0$ disk.nvme0c0n1.in‚29590.48828125$ memory_percent‚93.21646690368652 proc.cpu.threads‚17 disk.dm-0.in‚29590.48828125 disk.rbd3.out‚0 disk.rbd1.out‚ 34.583984375ÛRÓ” !‚(:‰  šéÒ›âñö gpu.0.gpu‚96.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚333.229' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚66.6458 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ê¡K.¡!‚(:£  šéÒ „Ùù network.recv‚ 3.259861e+06 disk.drbd2165.out‚0( proc.memory.availableMB‚ 1111.3515625) disk./.usagePercent‚37.45812433560925$ disk./.usageGB‚558.1531219482422 disk.rbd3.in‚0$ disk.nvme0c0n1.in‚29676.51171875 disk.rbd0.in‚0 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.sda.out‚ 688.62109375 disk.dm-1.out‚ 688.62109375 disk.dm-0.in‚29676.51171875 disk.sda3.in‚ 75.2734375 disk.sda2.in‚0 disk.rbd1.out‚ 34.708984375 disk.sda.in‚ 75.2734375$ disk.dm-0.out‚126847.96240234375 disk.dm-1.in‚ 75.2734375 disk.sda1.in‚0# proc.memory.rssMB‚ 2804.85546875 network.sent‚ 1.5036486e+07 disk.rbd3.out‚0' disk.nvme0n1.out‚126847.96240234375 disk.sda2.out‚0 disk.rbd1.in‚ 0.50830078125 disk.sda3.out‚ 688.62109375 cpu‚25.556698104909785 proc.cpu.threads‚17) disk.nvme0c0n1.out‚126847.96240234375" disk.nvme0n1.in‚29676.51171875 disk.rbd0.out‚0$ memory_percent‚93.21684837341309* proc.memory.percent‚17.119479179382324«ú÷Ý”¢!‚(:‰ ¯šéÒêÜ›÷ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚273.148' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.6296 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0C瘒£!‚(:™ ¯šéÒ­Û¶ù network.recv‚ 3.263355e+06 disk.rbd1.out‚ 34.708984375 disk.drbd2165.out‚0$ memory_percent‚93.21713447570801# proc.memory.rssMB‚ 2804.85546875) disk./.usagePercent‚37.45912046092498 disk.sda.in‚ 75.2734375 disk.dm-1.in‚ 75.2734375 disk.rbd0.out‚0 disk.dm-0.in‚ 29676.921875 disk.sda2.in‚0 disk.sda2.out‚0 cpu‚25.55686325317507$ disk./.usageGB‚558.1679649353027 disk.dm-1.out‚ 690.1484375 disk.sda1.out‚0 disk.sda3.in‚ 75.2734375 disk.rbd3.in‚0 disk.nvme0n1.in‚ 29676.921875* proc.memory.percent‚17.119479179382324 proc.cpu.threads‚17 disk.sda1.in‚0 disk.sda3.out‚ 690.1484375 disk.drbd2165.in‚0$ disk.dm-0.out‚126863.30712890625 disk.rbd1.in‚ 0.50830078125 disk.rbd3.out‚0" disk.nvme0c0n1.in‚ 29676.921875' disk.nvme0n1.out‚126863.30712890625 network.sent‚ 1.1Fd_’5041536e+07 disk.sda.out‚ 690.1484375 disk.rbd0.in‚0) disk.nvme0c0n1.out‚126863.30712890625( proc.memory.availableMB‚ 1111.3046875oØbS4¤!/ _wandbruntime‚13088  _runtime‚13088”éqqÆ¥!À trainer/global_step‚7899 $ _timestamp‚1.784302904444757e+09 _runtime‚13088.66190893 _step‚647 & lr-AdamW/pg1‚0.0004706340021034861 & lr-AdamW/pg2‚0.0004706340021034861€jæ˦!Å _step‚647 & lr-AdamW/pg1‚0.0004706340021034861 & lr-AdamW/pg2‚0.0004706340021034861 trainer/global_step‚7899 $ _timestamp‚1.784302904444757e+09 _runtime‚13088.66190893‡›ÊPÖ4§!/ _wandbruntime‚13088  _runtime‚13088ýÂ%‰(¨!þO D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2276453113025024 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚42663977 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2275461687890124 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2276454484588032 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2269024917389312 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 _runtime‚13088.663133191 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚67339885 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2269024891215872 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚67339879 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7429567198720 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9583605127168 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7415512147914 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 % _timestamp‚1.7843029044458108e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7428221809152 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 _step‚648 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2276454484588032 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 135725688 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2268044831141888 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚20263341 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 trainer/global_step‚7899 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2276453113025024 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7428221809152 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2268044805713920 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22400631 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚613248504721408 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚613248872812544 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22400373 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0Í‚Þ (©!ƒP RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22400373 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2276453113025024 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚42663977 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2275461687890124 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2276454484588032 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2269024917389312 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 _runtime‚13088.663133191 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚67339885 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2269024891215872 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚67339879 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7429567198720 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9583605127168 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7415512147914 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 % _timestamp‚1.7843029044458108e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7428221809152 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 _step‚648 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2276454484588032 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 135725688 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2268044831141888 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚20263341 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 trainer/global_step‚7899 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2276453113025024 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7428221809152 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2268044805713920 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22400631 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚613248504721408 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚613248872812544 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0ˆÞ¬/@4ª!/ _wandbruntime‚13090  _runtime‚13090±±?'«!ˆN L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 135752365 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 135742372 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2275739123106256 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2268322712920064 _runtime‚13090.050710622 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚68404274 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7416435614160 _step‚649 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2269302918119424 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚248 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2276733410360832 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9584823424000 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2269302918119424 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚68394287 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚253 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚68394287 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚20265884 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7417780214594 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404717568 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9584786797568 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2276732038790656 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚67348085 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚42669230 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚42668977 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚22403341 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚622910777926656 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36626432 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7429146844672 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22403093 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2276733410360832 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚67348085 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7430492241408 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 trainer/global_step‚7899 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚20265889 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚67348091 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 % _timestamp‚1.7843029058333087e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 135752365 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2268322687492096 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0é”Dà¬!N E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚68404274 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7416435614160 _step‚649 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2269302918119424 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚248 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2276733410360832 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9584823424000 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2269302918119424 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚Ëa¡3!1175 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚68394287 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚253 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚68394287 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚20265884 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7417780214594 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404717568 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9584786797568 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2276732038790656 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚67348085 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚42669230 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚42668977 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚22403341 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚622910777926656 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36626432 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7429146844672 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22403093 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2276733410360832 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚67348085 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7430492241408 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 trainer/global_step‚7899 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚20265889 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚67348091 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 % _timestamp‚1.7843029058333087e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 135752365 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2268322687492096 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 135752365 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 135742372 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2275739123106256 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2268322712920064 _runtime‚13090.050710622‰ä|[4­!/ _wandbruntime‚13090  _runtime‚13090e}“ ®!‡ _step‚650 0loss/FRt1/assignment_loss‚1.4505033840403032 0loss/FRt2/assignment_loss‚0.5723624209511863 3loss/SRqqt1/assignment_loss‚0.48418110692358074 4loss/FBt2/assignment_loss‚0.00036329177500568144 1loss/SRqqt1/detection_loss‚0.7966902128694128 _runtime‚13090.135352292 2loss/FBt1/assignment_loss‚0.005422695622777499 .loss/FRt2/detection_loss‚1.374629377583784 %loss/total_loss‚7.273362327061392 epoch‚19 trainer/global_step‚7899 2loss/SRqqt2/assignment_loss‚0.4710775356389695 /loss/FRt1/detection_loss‚0.4863336226770821 0loss/SRqqt2/detection_loss‚0.574738115942656 /loss/FBt1/detection_loss‚0.5637472650374878 0loss/FBt2/detection_loss‚0.49331329799914625 % _timestamp‚1.7843029059183779e+09SÒÇT¯!Œ _step‚650 0loss/FRt1/assignment_loss‚1.4505033840403032 0loss/FRt2/assignment_loss‚0.5723624209511863 3loss/SRqqt1/assignment_loss‚0.48418110692358074 4loss/FBt2/assignment_loss‚0.00036329177500568144 1loss/SRqqt1/detection_loss‚0.7966902128694128 _runtime‚13090.135352292 2loss/FBt1/assignment_loss‚0.005422695622777499 .loss/FRt2/detection_loss‚1.374629377583784 %loss/total_loss‚7.273362327061392 epoch‚19 trainer/global_step‚7899 2loss/SRqqt2/assignment_loss‚0.4710775356389695 /loss/FRt1/detection_loss‚0.4863336226770821 0loss/SRqqt2/detection_loss‚0.574738115942656 /loss/FBt1/detection_loss‚0.5637472650374878 0loss/FBt2/detection_loss‚0.49331329799914625 % _timestamp‚1.7843029059183779e+09Š^ä#|”°!‚(:‰ ¾šéÒõíëö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚40.0 gpu.0.powerWatts‚299.456' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚59.8912 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0š–.±!‚(:Ž ¾šéÒ«þú" disk.dm-0.out‚126870.927734375 disk.rbd3.out‚0* proc.memory.percent‚17.119479179382324 proc.cpu.threads‚17) disk./.usagePercent‚37.52576408623087# disk./.usageGB‚559.161003112793 disk.sda1.in‚0 disk.sda1.out‚0# proc.memory.rssMB‚ 2804.85546875 network.sent‚ 1.504673e+07 network.recv‚ 3.266752e+06 disk.dm-1.out‚ 690.5703125 disk.drbd2165.in‚0 disk.drbd2165.out‚0$ memory_percent‚93.21975708007812" disk.nvme0c0n1.in‚ 29677.140625% disk.nvme0n1.out‚126870.927734375 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda.in‚ 75.2734375 disk.sda.out‚ 690.5703125 disk.dm-1.in‚ 75.2734375' disk.nvme0c0n1.out‚126870.927734375 disk.rbd0.out‚0 disk.dm-0.in‚ 29677.140625 disk.sda3.out‚ 690.5703125 disk.rbd1.in‚ 0.50830078125 disk.rbd1.out‚ 34.771484375$ proc.memory.availableMB‚1110.875 disk.rbd0.in‚0 disk.sda3.in‚ 75.2734375 disk.rbd3.in‚0 cpu‚25.557005085044125 disk.nvme0n1.in‚ 29677.140625ïÙ_M”²!‚(:‰ ÍšéÒÒ†îö gpu.0.gpu‚90.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚330.816' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚66.1632 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0h×Ô!³!‚(:– ÍšéÒÞ‘Šú disk.rbd3.in‚0 disk.nvme0n1.in‚ 29677.5625 disk.rbd1.out‚ 34.771484375 disk.sda3.out‚ 691.24609375) proc.memory.availableMB‚ 1110.51953125 network.recv‚ 3.27135e+06' disk.nvme0n1.out‚128226.65869140625 disk.sda1.in‚0 cpu‚25.557244670980783 disk.nvme0c0n1.in‚ 29677.5625) disk.nvme0c0n1.out‚128226.65869140625 disk.sda3.in‚ 75.2734375 disk.drbd2165.out‚0* proc.memory.percent‚17.119479179382324 network.sent‚ 1.510935e+07$ disk./.usageGB‚559.6631164550781 disk.drbd2165.in‚0 disk.sda2.out‚0 disk.dm-1.in‚ 75.2734375 disk.rbd0.in‚0 disk.sda1.out‚0 disk.rbd1.in‚ 0.50830078125 disk.rbd0.out‚0$ disk.dm-0.out‚128226.65869140625$ memory_percent‚93.22192668914795# proc.memory.rssMB‚ 2804.85546875 proc.cpu.threads‚17 disk.rbd3.out‚0 disk.sda2.in‚0 disk.sda.out‚ 691.24609375) disk./.usagePercent‚37.55946133393279 disk.sda.in‚ 75.2734375 disk.dm-1.out‚ 691.24609375 disk.dm-0.in‚ 29677.5625÷Å„ž´!‚(:“ ÜšéÒÈ”òö gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚268.527' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚53.70539999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0UÔÏõ*µ!‚(:Ÿ ÜšéÒŘ˜ù$ disk.nvme0c0n1.in‚29867.27734375 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 691.6015625' proc.memory.availableMB‚ 1110.484375" disk.nvme0n1.in‚29867.27734375 disk.rbd0.in‚0 disk.rbd0.out‚0* proc.memory.percent‚17.119479179382324$ disk./.usageGB‚559.6672019958496 disk.rbd3.in‚0 disk.rbd3.out‚0' disk.nvme0n1.out‚128239.32861328125 disk.drbd2165.out‚0$ memory_percent‚93.22214126586914 network.recv‚ 3.274721e+06 network.sent‚ 1.5114959e+07 disk.dm-1.in‚ 75.2734375 disk.sda1.out‚0 disk.drbd2165.in‚0 cpu‚25.557411157547378 disk.sda3.in‚ 75.2734375# proc.memory.rssMB‚ 2804.85546875) disk.nvme0c0n1.out‚128239.32861328125 disk.rbd1.out‚ 34.833984375 disk.sda.in‚ 75.2734375 disk.sda.out‚ 691.6015625 disk.dm-0.in‚29867.27734375$ disk.dm-0.out‚128239.32861328125 disk.sda1.in‚0 proc.cpu.threads‚17) disk./.usagePercent‚37.55973551800196 disk.rbd1.in‚ 0.50830078125 disk.dm-1.out‚ 691.6015625DrC$”¶!‚(:‰ ëšéÒÙŠôö gpu.0.gpu‚99.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚332.173' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚66.4346 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÄÎ,ú*·!‚(:Ÿ ëšéÒ‹Ä’ù proc.cpu.threads‚17$ disk.nvme0c0n1.in‚29867.49609375# disk.dm-0.out‚128259.6044921875# memory_percent‚93.2236909866333) disk./.usagePercent‚37.68173590844417 disk.sda.in‚ 75.2734375 disk.sda3.in‚ 75.2734375 disk.drbd2165.out‚0 disk.rbd3.out‚0 disk.dm-1.in‚ 75.2734375 disk.dm-0.in‚29867.49609375 disk.sda2.out‚0 disk.rbd1.out‚ 34.833984375 disk.rbd0.in‚0 disk.sda1.in‚0# proc.memory.rssMB‚ 2804.85546875* proc.memory.percent‚17.119479179382324 cpu‚25.55766877915192 network.sent‚ 1.5120051e+07" disk.nvme0n1.in‚29867.49609375 disk.rbd3.in‚0$ disk./.usageGB‚561.4850959777832 disk.sda1.out‚0 network.recv‚ 3.278367e+06( disk.nvme0c0n1.out‚128259.6044921875& disk.nvme0n1.out‚128259.6044921875 disk.sda2.in‚0 disk.rbd1.in‚ 0.50830078125 disk.dm-1.out‚ 691.85546875 disk.rbd0.out‚0 disk.sda3.out‚ 691.85546875 disk.sda.out‚ 691.85546875 disk.drbd2165.in‚0) proc.memory.availableMB‚ 1110.23046875Bx÷=ž¸!‚(:“ úšéÒÚÙéö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚262.054' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚52.41079999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚06yι!‚(:ˆ úšéÒ·…”ù disk.rbd0.out‚0# proc.memory.rssMB‚ 2804.85546875 disk.sda1.in‚0 disk.rbd1.out‚ 34.896484375 disk.dm-0.in‚29867.91796875 network.sent‚ 1.5125104e+07 disk.sda3.out‚692 disk.drbd2165.in‚0 disk.rbd3.in‚0 disk.sda2.in‚0# disk./.usageGB‚561.491397857666 disk.rbd3.out‚0' disk.nvme0n1.out‚128431.75439453125 disk.sda2.out‚0$ disk.dm-0.out‚128431.75439453125 disk.sda.in‚ 75.2734375 disk.dm-1.in‚ 75.2734375 disk.rbd0.in‚0 disk.drbd2165.out‚0 cpu‚25.557849735637134* disk./.usagePercent‚37.682158832890735) disk.nvme0c0n1.out‚128431.75439453125" disk.nvme0n1.in‚29867.91796875$ memory_percent‚93.22218894958496 disk.sda1.out‚0 network.recv‚ 3.281714e+06 disk.sda.out‚692 disk.rbd1.in‚ 0.50830078125( proc.memory.availableMB‚ 1110.4765625* proc.memory.percent‚17.119479179382324 disk.sda3.in‚ 75.2734375$ disk.nvme0c0n1.in‚29867.91796875 proc.cpu.threads‚17 disk.dm-1.out‚692 ¨A¿4º!/ _wandbruntime‚13168  _runtime‚13168àšÇ¾È»! % _timestamp‚1.7843029839497414e+09 _runtime‚13168.166911211 _step‚651 & lr-AdamW/pg1‚0.0004680441220142832 & lr-AdamW/pg2‚0.0004680441220142832 trainer/global_step‚7949ËïÓͼ!Ç & lr-AdamW/pg1‚0.0004680441220142832 & lr-AdamW/pg2‚0.0004680441220142832 trainer/global_step‚7949 % _timestamp‚1.7843029839497414e+09 _runtime‚13168.166911211 _step‚651‹7xj¤4½!/ _wandbruntime‚13168  _runtime‚13168ÙÇ(¾!þO G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2289400643883742 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7475883129344 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7474537739776 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9642834816000 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2290400837024256 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚271 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚617122825105408 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚67750179 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚68821534 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚42928523 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22537756 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚626766066743296 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 136561740 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2281938894622720 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚68811561 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 136571719 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7463093861344 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚20390491 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 trainer/global_step‚7949 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2290399465461248 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2282924953894912 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9642873546752 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2282924953894912 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚68811561 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7461749261022 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2290399465461248 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22538027 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚68821534 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚42928523 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22537756 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚626766066743296 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 136561740 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2281938894622720 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚68811561 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 136571719 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7463093861344 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚20390491 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 trainer/global_step‚7949 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2290399465461248 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2282924953894912 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9642873546752 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 ^IDeviceStatsMonitor.on_train_batch_start/activN)Nÿµe_bytes.large_pool.allocated‚2282924953894912 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚68811561 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7461749261022 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2290399465461248 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22538027 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 JDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9644015924224 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7476808172032 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚42933491 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚20393034 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22540457 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2289680819156566 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2283202954625024 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚265 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚260 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚67758385 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 _runtime‚13169.554033311 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚67758391 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 _step‚653 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 $ _timestamp‚1.784302985336642e+09 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2283202928451584 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2290678391226880 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚626844327437312 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 136588417 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 136578424 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7475462775296 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚42933756 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚22540717 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚68820039 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2282216801828864 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7476808172032 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚38723584 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚626844734252032 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9644054647808 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚20393039 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚67758391 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119¹Vë('Â!ŒN GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7476808172032 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚42933491 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚20393034 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22540457 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2289680819156566 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2283202954625024 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚265 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚260 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚67758385 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 _runtime‚13169.554033311 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚67758391 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 _step‚653 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 $ _timestamp‚1.784302985336642e+09 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2283202928451584 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2290678391226880 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚626844327437312 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 136588417 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 136578424 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7475462775296 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚42933756 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚22540717 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚68820039 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2282216801828864 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7476808172032 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚38723584 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚626844734252032 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9644054647808 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚20393039 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚67758391 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9644015924224 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216ªn,4Ã!/ _wandbruntime‚13169  _runtime‚13169fFX Ä!… 0loss/FRt1/assignment_loss‚1.4685506783461593 1loss/SRqqt1/detection_loss‚0.8132609096240555 1loss/SRqqt2/detection_loss‚0.6174717454777208 % _timestamp‚1.7843029854220903e+09 2loss/SRqqt1/assignment_loss‚0.4714865700410346 /loss/FRt1/detection_loss‚0.5054064908383276 /loss/FRt2/detection_loss‚1.4214006880475423 /loss/FBt2/detection_loss‚0.3981819043696766 epoch‚19 _runtime‚13169.639045044 _step‚654 3loss/FBt2/assignment_loss‚0.0018818273450376842 %loss/total_loss‚7.429820482226688 trainer/global_step‚7949 0loss/FRt2/assignment_loss‚0.5145446779182453 2loss/SRqqt2/assignment_loss‚0.3635211465774456 1loss/FBt1/assignment_loss‚0.02309622591627888 /loss/FBt1/detection_loss‚0.8310176177251646//2iÅ!Š 2loss/SRqqt1/assignment_loss‚0.4714865700410346 /loss/FRt1/detection_loss‚0.5054064908383276 /loss/FRt2/detection_loss‚1.4214006880475423 /loss/FBt2/detection_loss‚0.3981819043696766 epoch‚19 _runtime‚13169.639045044 _step‚654 3loss/FBt2/assignment_loss‚0.0018818273450376842 %loss/total_loss‚7.429820482226688 trainer/global_step‚7949 0loss/FRt2/assignment_loss‚0.5145446779182453 2loss/SRqqt2/assignment_loss‚0.3635211465774456 1loss/FBt1/assignment_loss‚0.02309622591627888 /loss/FBt1/detection_loss‚0.8310176177251646 0loss/FRt1/assignment_loss‚1.4685506783461593 1loss/SRqqt1/detection_loss‚0.8132609096240555 1loss/SRqqt2/detection_loss‚0.6174717454777208 % _timestamp‚1.7843029854220903e+09Žé›wì”Æ!‚(:‰ ‰›éÒˆÉëö gpu.0.gpu‚81.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚324.638' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚64.9276 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚019Ð,Ç!‚(:¡ ‰›éÒ—Ó½ù disk.sda3.out‚ 692.32421875 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.rbd1.in‚ 0.50830078125 disk.rbd1.out‚ 34.896484375 disk.sda.in‚ 75.2734375 disk.sda.out‚ 692.32421875 disk.dm-1.out‚ 692.32421875 network.sent‚ 1.5130202e+07( disk.nvme0c0n1.out‚130288.0634765625 disk.sda2.in‚0 disk.sda1.out‚0) proc.memory.availableMB‚ 1109.98828125 proc.cpu.threads‚17$ disk.nvme0c0n1.in‚29868.12109375 disk.sda2.out‚0 disk.rbd0.out‚0# proc.memory.rssMB‚ 2804.85546875* proc.memory.percent‚17.119479179382324 disk.rbd0.in‚0 network.recv‚ 3.285043e+06) disk./.usagePercent‚37.68195274682567 disk.sda3.in‚ 75.2734375 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.dm-1.in‚ 75.2734375$ memory_percent‚93.22516918182373$ disk./.usageGB‚561.4883270263672" disk.nvme0n1.in‚29868.12109375 disk.dm-0.in‚29868.12109375# disk.dm-0.out‚130288.0634765625 disk.sda1.in‚0 cpu‚25.558085819132668& disk.nvme0n1.out‚130288.0634765625KÐœTŸÈ!‚(:” ˜›éÒ¼•Œø gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚281.431' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚56.286199999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0t#3%É!‚(:š ˜›éÒÔþÆù disk.dm-1.in‚ 75.2734375 disk.dm-0.in‚30047.10546875 disk.sda1.out‚0" disk.nvme0n1.in‚30047.10546875 disk.sda2.out‚0 proc.cpu.threads‚17 disk.sda.in‚ 75.2734375 disk.dm-1.out‚ 692.38671875 disk.rbd1.out‚ 34.958984375' proc.memory.availableMB‚ 1110.203125 disk.rbd0.in‚0 disk.sda3.out‚ 692.38671875' disk.nvme0c0n1.out‚130295.931640625# proc.memory.rssMB‚ 2804.85546875 network.recv‚ 3.289376e+06) disk./.usagePercent‚37.68237080712908 disk.rbd0.out‚0$ disk.nvme0c0n1.in‚30047.10546875 cpu‚25.55822774171456 disk.sda2.in‚0" disk.dm-0.out‚130295.931640625 disk.rbd1.in‚ 0.50830078125 disk.sda.out‚ 692.38671875 disk.sda1.in‚0 disk.rbd3.out‚0% disk.nvme0n1.out‚130295.931640625$ memory_percent‚93.22385787963867# disk./.usageGB‚561.494556427002 disk.sda3.in‚ 75.2734375 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.rbd3.in‚0* proc.memory.percent‚17.119479179382324 network.sent‚ 1.5193178e+07šœžÊ!‚(:“ §›éÒ©Òïö gpu.0.gpu‚86.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚320.425' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚64.08500000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0„l€;/Ë!‚(:¤ §›éÒïöÙù$ disk.dm-0.out‚130310.21533203125 disk.rbd3.out‚0 disk.rbd3.in‚0 disk.dm-1.out‚ 692.80078125 disk.sda.out‚ 692.80078125 disk.rbd0.in‚0 network.recv‚ 3.293203e+06$ memory_percent‚93.22521686553955) proc.memory.availableMB‚ 1109.98046875 network.sent‚ 1.5197798e+07* disk./.usagePercent‚37.682823172442085 disk.drbd2165.out‚0 disk.sda2.in‚0 disk.dm-0.in‚30047.52734375$ disk.nvme0c0n1.in‚30047.52734375 disk.rbd1.out‚ 34.958984375* proc.memory.percent‚17.119479179382324 proc.cpu.threads‚17 disk.drbd2165.in‚0 disk.dm-1.in‚ 75.2734375 disk.rbd0.out‚0$ disk./.usageGB‚561.5012969970703 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda3.in‚ 75.2734375) disk.nvme0c0n1.out‚130310.21533203125" disk.nvme0n1.in‚30047.52734375' disk.nvme0n1.out‚130310.21533203125 disk.sda2.out‚0 disk.sda3.out‚ 692.80078125 disk.rbd1.in‚ 0.50830078125 disk.sda.in‚ 75.2734375# proc.memory.rssMB‚ 2804.85546875 cpu‚25.55844467727973ƱŸžÌ!‚(:“ ¶›éÒÁ¾÷ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚275.782' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚55.15639999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0µ%€ŸQÍ!‚(:¢ ¶›éÒõª¾ù disk.rbd3.in‚0 disk.rbd1.out‚ 35.021484375 disk.rbd0.out‚0 disk.rbd3.out‚0' disk.nvme0n1.out‚130318.29736328125 disk.dm-1.out‚ 692.94140625) proc.memory.availableMB‚ 1152.47265625 network.sent‚ 1.520344e+07 disk.drbd2165.in‚0 disk.sda.out‚ 692.94140625 disk.sda1.ou@‚}Üt‚0# proc.memory.rssMB‚ 2804.85546875) disk./.usagePercent‚37.68063840414488$ disk.nvme0c0n1.in‚30047.73046875" disk.nvme0n1.in‚30047.73046875 disk.sda2.in‚0$ disk.dm-0.out‚130318.29736328125 disk.rbd1.in‚ 0.50830078125 disk.dm-0.in‚30047.73046875$ memory_percent‚92.96586513519287 disk.sda3.out‚ 692.94140625) disk.nvme0c0n1.out‚130318.29736328125 disk.rbd0.in‚0 proc.cpu.threads‚17 network.recv‚ 3.29675e+06 disk.sda1.in‚0 disk.drbd2165.out‚0 disk.sda2.out‚0 disk.sda.in‚ 75.2734375 disk.dm-1.in‚ 75.2734375* proc.memory.percent‚17.119479179382324 cpu‚25.558643720355793$ disk./.usageGB‚561.4687423706055 disk.sda3.in‚ 75.2734375ê¨ÎpžÎ!‚(:“ Å›éÒ›Èâö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚41.0 gpu.0.powerWatts‚296.216' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚59.24320000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ˈÞD.Ï!‚(:£ Å›éÒðÓ«ù disk.sda3.in‚ 75.2734375* proc.memory.percent‚17.119479179382324 proc.cpu.threads‚17 network.recv‚ 3.300414e+06' disk.nvme0n1.out‚130329.59326171875 disk.rbd0.out‚0$ memory_percent‚92.97041893005371$ disk./.usageGB‚561.4761695861816 disk.sda1.out‚0 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.sda.in‚ 75.2734375 disk.rbd0.in‚0 disk.sda1.in‚0 disk.drbd2165.out‚0) disk./.usagePercent‚37.68113685081404) disk.nvme0c0n1.out‚130329.59326171875 disk.sda2.out‚0 disk.sda.out‚ 693.23046875 disk.drbd2165.in‚0 disk.rbd1.out‚ 35.021484375 disk.dm-0.in‚30048.15234375 disk.sda3.out‚ 693.23046875# proc.memory.rssMB‚ 2804.85546875 cpu‚25.558842018605404$ disk.nvme0c0n1.in‚30048.15234375" disk.nvme0n1.in‚30048.15234375 disk.rbd1.in‚ 0.50830078125$ disk.dm-0.out‚130329.59326171875 disk.sda2.in‚0 disk.dm-1.in‚ 75.2734375 disk.dm-1.out‚ 693.23046875( proc.memory.availableMB‚ 1151.7265625 network.sent‚ 1.5208609e+072¬ºŸÐ!‚(:” Ô›éÒªâ ø gpu.0.gpu‚90.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚284.911' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚56.982200000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0þüŸœÑ!‚(: Ô›éÒÕœù' disk.nvme0n1.out‚130342.10302734375 disk.rbd1.out‚ 35.083984375 disk.sda.in‚ 75.2734375) proc.memory.availableMB‚ 1153.23046875 network.sent‚ 1.5213717e+07$ disk./.usageGB‚560.4200096130371 disk.sda3.out‚693.4375 disk.drbd2165.out‚0 disk.sda2.in‚0 disk.rbd0.in‚0# memory_percent‚92.9612398147583 disk.rbd0.out‚0 disk.rbd3.in‚0! disk.nvme0c0n1.in‚ 30302.03125 disk.sda2.out‚0 disk.dm-1.in‚ 75.2734375 disk.dm-0.in‚ 30302.03125 disk.sda1.in‚0 disk.drbd2165.in‚0 disk.nvme0n1.in‚ 30302.03125 disk.rbd1.in‚ 0.50830078125* proc.memory.percent‚17.119479179382324* disk./.usagePercent‚37.610257068839076$ disk.dm-0.out‚130342.10302734375 disk.rbd3.out‚0 disk.sda1.out‚0# proc.memory.rssMB‚ 2804.85546875 cpu‚25.558964826002757 network.recv‚ 3.303554e+06 disk.sda3.in‚ 75.2734375 disk.sda.out‚693.4375 disk.dm-1.out‚693.4375 proc.cpu.threads‚17) disk.nvme0c0n1.out‚130342.10302734375‘T4Ò!/ _wandbruntime‚13247  _runtime‚13247¨ˆšÆÈÓ! % _timestamp‚1.7843030631627522e+09 _runtime‚13247.379919335 _step‚655 & lr-AdamW/pg1‚0.0004654463983923553 & lr-AdamW/pg2‚0.0004654463983923553 trainer/global_step‚7999\Ý#ÍÔ!Ç & lr-AdamW/pg1‚0.0004654463983923553 & lr-AdamW/pg2‚0.0004654463983923553 trainer/global_step‚7999 % _timestamp‚1.7843030631627522e+09 _runtime‚13247.379919335 _step‚655)A‹4Õ!/ _wandbruntime‚13247  _runtime‚13247Ò3Û(Ö!üO ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2296824964227072 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 _step‚656 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2303342339933940 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚68160479 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2304345817897472 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚43192967 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚630699428794368 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚620997145489408 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚43192702 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚20517646 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 % _timestamp‚1.7843030631637743e+09 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2304347189460480 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2303340969905650 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2295832983531520 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚265 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2296824990400512 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36633600 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7507986374130 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7522199059968 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚260 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404724736 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚69237313 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚20517641 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2296824964227072 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2304345817897472 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7509330974452 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2295833008959488 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22675321 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22675061 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚68160479 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚630699024069632 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚68160485 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7520853670400 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚69247286 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚620997513580544 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚69237313 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚69247286 Bj1(×!P D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36633600 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7507986374130 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7522199059968 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚260 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404724736 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚69237313 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚20517641 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2296824964227072 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2304345817897472 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7509330974452 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2295833008959488 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22675321 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22675061 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚68160479 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚630699024069632 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚68160485 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7520853670400 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚69247286 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚620997513580544 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚69237313 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚69247286 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2304345817897472 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚43192967 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚630699428794368 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚620997145489408 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚43192702 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚20517646 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 % _timestamp‚1.7843030631637743e+09 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2304347189460480 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2303340969905650 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2295832983531520 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚265 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2296824990400512 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440a°ä4Ø!/ _wandbruntime‚13248  _runtime‚13248ÉÃJø Ù!ˆN E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7523124102656 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7521778705920 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚43198228 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚69245791 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2303621145178474 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚68168685 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2296110890737664 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2297102991130624 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7521778705920 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚68168685 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚256 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7510254440810 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 137424469 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22677783 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2297102964957184 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚68168691 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7508909840376 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚621074999988224 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 % _timestamp‚1.7843030645499823e+09 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚20520184 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2303619775150072 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚20520189 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 trainer/global_step‚7999 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚68168691 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚43197967 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404717568 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 [FDeviceStatsMonitor.on_trainïílÀ_batch_end/requested_bytes.large_pool.freed‚2296110865309696 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 137414476 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚22678039 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9703067609088 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2304626115233280 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7523124102656 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 137424469 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36626432 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2304624743663104 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2304626115233280 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚69255778 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚630778104223744 _step‚657tªè¬'Ú!N E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7523124102656 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7521778705920 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚43198228 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚69245791 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2303621145178474 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚68168685 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2296110890737664 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2297102991130624 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7521778705920 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚68168685 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚256 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7510254440810 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 137424469 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22677783 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2297102964957184 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚68168691 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7508909840376 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚621074999988224 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 % _timestamp‚1.7843030645499823e+09 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚20520184 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2303619775150072 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚20520189 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 trainer/global_step‚7999 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚68168691 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚43197967 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404717568 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2296110865309696 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 137414476 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚22678039 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9703067609088 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2304626115233280 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7523124102656 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 137424469 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36626432 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2304624743663104 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2304626115233280 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚69255778 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚630778104223744 _step‚657‘âs‚a4Û!/ _wandbruntime‚13248  _runtime‚13248ƒ:N´ Ü!‡ 0loss/FRt1/assignment_loss‚1.4544011705989874 0loss/FRt2/assignment_loss‚0.5585767405950994 3loss/SRqqt1/assignment_loss‚0.48614262047815193 3loss/FBt1/assignment_loss‚0.0008269928627961942 2loss/FBt2/assignment_loss‚0.006439727235423179 /loss/FRt2/detection_loss‚1.4850454799048423 epoch‚19 _runtime‚13248.852432664 1loss/SRqqt2/detection_loss‚0.5198044672223223 /loss/FBt1/detection_loss‚1.0195008053269703 trainer/global_step‚7999 $ _timestamp‚1.784303064635482e+09 _step‚658 2loss/SRqqt2/assignment_loss‚0.4293678142856796 0loss/FRt1/detection_loss‚0.49339262258373723 /loss/FBt2/detection_loss‚0.4610310071972969 %loss/total_loss‚7.748308929127238 1loss/SRqqt1/detection_loss‚0.8337794808359321>%zWÝ!Œ /loss/FRt2/detection_loss‚1.4850454799048423 epoch‚19 _runtime‚13248.852432664 1loss/SRqqt2/detection_loss‚0.5198044672223223 /loss/FBt1/detection_loss‚1.0195008053269703 trainer/global_step‚7999 $ _timestamp‚1.784303064635482e+09 _step‚658 2loss/SRqqt2/assignment_loss‚0.4293678142856796 0loss/FRt1/detection_loss‚0.49339262258373723 /loss/FBt2/detection_loss‚0.4610310071972969 %loss/total_loss‚7.748308929127238 1loss/SRqqt1/detection_loss‚0.8337794808359321 0loss/FRt1/assignment_loss‚1.4544011705989874 0loss/FRt2/assignment_loss‚0.5585767405950994 3loss/SRqqt1/assignment_loss‚0.48614262047815193 3loss/FBt1/assignment_loss‚0.0008269928627961942 2loss/FBt2/assignment_loss‚0.006439727235423179’êÙF2žÞ!‚(:“ ã›éÒÖøö gpu.0.gpu‚90.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚329.225' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚65.84500000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0do9‡#ß!‚(:˜ ã›éÒÒ•¬ù disk.rbd0.out‚0# proc.memory.rssMB‚ 2804.86328125' disk.nvme0c0n1.out‚130351.509765625" disk.nvme0n1.in‚30302.24609375" disk.dm-0.out‚130351.509765625$ memory_percent‚92.96369552612305 cpu‚25.561061704388255 proc.cpu.threads‚17 disk.dm-1.out‚ 693.8203125 disk.dm-0.in‚30302.24609375 disk.sda1.out‚0 disk.drbd2165.out‚0 disk.rbd3.out‚0 disk.sda2.in‚0 disk.sda.in‚ 75.2734375 disk.sda1.in‚0 disk.sda2.out‚0 network.recv‚ 3.307146e+06% disk.nvme0n1.out‚130351.509765625 disk.rbd1.out‚ 35.083984375* proc.memory.percent‚17.119526863098145 network.sent‚ 1.521828e+07# disk./.usageGB‚560.427074432373 disk.rbd0.in‚0 disk.sda3.in‚ 75.2734375 disk.sda3.out‚ 693.8203125 disk.rbd3.in‚0$ disk.nvme0c0n1.in‚30302.24609375' proc.memory.availableMB‚ 1152.828125* disk./.usagePercent‚37.610731194792486 disk.dm-1.in‚ 75.2734375 disk.rbd1.in‚ 0.50830078125 disk.sda.out‚ 693.8203125 disk.drbd2165.in‚0éA m”à!‚(:‰ ò›éÒû’ù gpu.0.gpu‚50.0 gpu.0.memory‚15, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚36.0 gpu.0.powerWatts‚109.461' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.8922 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚00-Ùá!‚(:‘ ò›éÒÀ®ßú disk.rbd0.out‚0 disk.sda3.in‚ 75.2734375 disk.sda2.out‚0 network.recv‚ 3.310979e+06 disk.dm-0.in‚ 30302.65625 disk.sda1.in‚0 disk.sda3.out‚ 694.0390625 disk.rbd3.out‚0 disk.sda1.out‚0! disk.nvme0c0n1.in‚ 30302.65625* proc.memory.percent‚17.119526863098145 cpu‚25.56407035497228 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.drbd2165.out‚0 disk.rbd3.in‚0 disk.rbd1.in‚ 0.50830078125# memory_percent‚92.9631233215332' proc.memory.availableMB‚ 1152.921875# proc.memory.rssMB‚ 2804.86328125 network.sent‚ 1.5281604e+07) disk./.usagePercent‚37.61117818394727 disk.dm-1.in‚ 75.2734375( disk.nvme0c0n1.out‚130363.4248046875 disk.drbd2165.in‚0& disk.nvme0n1.out‚130363.4248046875 disk.sda2.in‚0 disk.rbd1.out‚ 35.146484375 disk.sda.in‚ 75.2734375 disk.sda.out‚ 694.0390625$ disk./.usageGB‚560.4337348937988# disk.dm-0.out‚130363.4248046875 disk.nvme0n1.in‚ 30302.65625 disk.dm-1.out‚ 694.0390625•ÞÀT4â!/ _wandbruntime‚13276  _runtime‚13276}…³=ã!‰{ @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9969059228897095 5 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.99639493227005 8"Purity/0FRt*SRqqt1FBt/event_purity‚0.863945722579956 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6415125727653503 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.20064201951026917 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6831989288330078 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7857142686843872 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3822741210460663 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6713663935661316 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5608333349227905 particle/accuracy_3_of_3‚1 particle/accuracy_3_of_5‚1 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9994588494300842 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9814814329147339 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.40740740299224854 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6425684690475464 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.03257915750145912 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6626734137535095 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6393002271652222 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6922417283058167 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6261513829231262 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.07692307978868484 epoch‚19 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.04305829480290413 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8474301695823669 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 +particle/sensitivity‚0.6375354528427124 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9615384340286255 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8842229247093201 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.21602194011211395 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.13694025576114655 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3854166865348816 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5570737719535828 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9973172545433044 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9814814329147339 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6533726453781128 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 trainer/global_step‚7999 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.4714246988296509 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.21150325238704681 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.16414682567119598 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 particle/accuracy_1_of_1‚1 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6740841269493103 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7358526587486267 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6487979292869568 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.46604618430137634 +particle/specificity‚0.9122573137283325 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.07692307978868484 jet/accuracy_5_of_5‚0 7"Purity/*FRt*SRqqt1FBt/event_purity‚0.73866868019104 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.43997693061828613 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6425684690475464 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.18280069530010223 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7114006280899048 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6513466835021973 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9921594262123108 particle/accuracy_2_of_4‚1 particle/accuracy_1_of_5‚1 particle/accuracy_4_of_5‚1 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.22825166583061218 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8479214906692505 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8474301695823669 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.04845140501856804 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.4850291311740875 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7555310130119324 ) Purity/2FRt*SRqqt2FBt/FRt_purity‚0.75 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6770833730697632 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.01807679794728756 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8009259104728699 7validation_average_jet_accuracy‚0.37242767214775085 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 _runtime‚13276.202152641 6 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.999009907245636 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.18507443368434906 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 *jet/accuracy_2_of_2‚0.1542491763830185 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7142857313156128 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7257859110832214 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7758315205574036 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7506373524665833 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.21918928623199463 *jet/accuracy_1_of_2‚0.7841231822967529 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6384945511817932 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.047466885298490524 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9464154243469238 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.21280519664287567 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.1750645488500595 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.22988121211528778 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7745379209518433 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 particle/accuracy_3_of_4‚1 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7176269888877869 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.46990740299224854 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9897685050964355 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9583333134651184 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6912494897842407 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8396018743515015 particle/accuracy_5_of_5‚0 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.20921556651592255 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9965004920959473 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7251234650611877 'particle/f_score‚0.6510772705078125 _step‚659 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.2906171977519989 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7296561002731323 jet/accuracy_2_of_5‚1 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6861165761947632 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6743914484977722 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 jet/accuracy_4_of_5‚0 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.037863560020923615 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8917797803878784 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9993476867675781 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.5941483378410339 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.048153530806303024 (particle/accuracy‚0.8532560467720032 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9993476867675781 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.23609218001365662 *jet/accuracy_1_of_1‚0.3305790424346924 , Purity/0FRt2SRqqt1FBt/FBt_purity‚0.96875 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5581051111221313 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 +jet/accuracy_3_of_4‚0.09675586968660355 6 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.842706024646759 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.22148717939853668 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9964677691459656 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 +jet/accuracy_3_of_3‚0.07046087086200714 *jet/accuracy_1_of_4‚0.9639049768447876 ,jet/accuracy_4_of_4‚0.013899499550461769 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3232397437095642 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt0SRqqt0FBt/event_purity‚0.0352289043366909 *jet/accuracy_1_of_3‚0.9151612520217896 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9963976740837097 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.24180252850055695 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8396018743515015 *jet/accuracy_2_of_4‚0.7909020185470581 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 /particle/accuracy_4_of_4‚0.9467192888259888 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9926258325576782 9"Purity/*FRt2SRqqt*FBt/event_purity‚0.2120874971151352 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9263125061988831 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.2007310539484024 , Purity/0FRt2SRqqt*FBt/FBt_purity‚0.96875 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.38010600209236145 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.05267133563756943 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.0833333358168602 & Purity/0FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7168362736701965 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.23040169477462769 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9969992637634277 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 jet/accuracy_1_of_5‚1 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 particle/accuracy_1_of_3‚1 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6240926384925842 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 jet/accuracy_3_of_5‚1 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.03257915750145912 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.22988121211528778 *jet/accuracy_2_of_3‚0.5629128813743591 particle/accuracy_2_of_5‚1 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.02725229412317276 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8009259104728699 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5562595725059509 6 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.471925288438797 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4140625 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7328191995620728 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_4‚1 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 % _timestamp‚1.7843030919843476e+09 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9992585778236389 & Purity/*FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6364977955818176 6 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.850888192653656 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.15217402577400208 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.2871319651603699 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7142202258110046 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.04272473603487015 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.28565412759780884 particle/accuracy_2_of_2‚1 particle/accuracy_2_of_3‚1 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6880826950073242 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7582266926765442 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.19880163669586182 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.14852024614810944 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.0833333358168602 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7742124199867249 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9843188524246216 )loss/val_total_loss‚7.881929976107947 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.46097368001937866 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9901471138000488 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.24087421596050262 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.15666666626930237 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.625 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2409600466489792 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 particle/accuracy_1_of_2‚1 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293?µ‹jä!Ž{ 9"Purity/*FRt2SRqqt*FBt/event_purity‚0.2120874971151352 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9263125061988831 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.2007310539484024 , Purity/0FRt2SRqqt*FBt/fáç†<FBt_purity‚0.96875 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.38010600209236145 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.05267133563756943 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.0833333358168602 & Purity/0FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7168362736701965 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.23040169477462769 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9969992637634277 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 jet/accuracy_1_of_5‚1 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 particle/accuracy_1_of_3‚1 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6240926384925842 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 jet/accuracy_3_of_5‚1 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.03257915750145912 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.22988121211528778 *jet/accuracy_2_of_3‚0.5629128813743591 particle/accuracy_2_of_5‚1 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.02725229412317276 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8009259104728699 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5562595725059509 6 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.471925288438797 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4140625 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7328191995620728 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_4‚1 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 % _timestamp‚1.7843030919843476e+09 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9992585778236389 & Purity/*FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6364977955818176 6 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.850888192653656 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.15217402577400208 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.2871319651603699 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7142202258110046 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.04272473603487015 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.28565412759780884 particle/accuracy_2_of_2‚1 particle/accuracy_2_of_3‚1 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6880826950073242 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7582266926765442 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.19880163669586182 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.14852024614810944 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.0833333358168602 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7742124199867249 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9843188524246216 )loss/val_total_loss‚7.881929976107947 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.46097368001937866 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9901471138000488 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.24087421596050262 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.15666666626930237 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.625 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2409600466489792 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 particle/accuracy_1_of_2‚1 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9969059228897095 5 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.99639493227005 8"Purity/0FRt*SRqqt1FBt/event_purity‚0.863945722579956 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6415125727653503 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.20064201951026917 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6831989288330078 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7857142686843872 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3822741210460663 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6713663935661316 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5608333349227905 particle/accuracy_3_of_3‚1 particle/accuracy_3_of_5‚1 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9994588494300842 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9814814329147339 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.40740740299224854 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6425684690475464 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.03257915750145912 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6626734137535095 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6393002271652222 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6922417283058167 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6261513829231262 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.07692307978868484 epoch‚19 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.04305829480290413 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8474301695823669 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 +particle/sensitivity‚0.6375354528427124 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9615384340286255 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8842229247093201 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.21602194011211395 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.13694025576114655 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3854166865348816 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5570737719535828 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9973172545433044 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9814814329147339 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6533726453781128 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 trainer/global_step‚7999 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.4714246988296509 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.21150325238704681 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.16414682567119598 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 particle/accuracy_1_of_1‚1 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6740841269493103 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7358526587486267 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6487979292869568 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.46604618430137634 +particle/specificity‚0.9122573137283325 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.07692307978868484 jet/accuracy_5_of_5‚0 7"Purity/*FRt*SRqqt1FBt/event_purity‚0.73866868019104 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.43997693061828613 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6425684690475464 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.18280069530010223 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7114006280899048 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6513466835021973 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9921594262123108 particle/accuracy_2_of_4‚1 particle/accuracy_1_of_5‚1 particle/accuracy_4_of_5‚1 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.22825166583061218 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8479214906692505 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8474301695823669 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.04845140501856804 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.4850291311740875 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7555310130119324 ) Purity/2FRt*SRqqt2FBt/FRt_purity‚0.75 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6770833730697632 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.01807679794728756 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8009259104728699 7validation_average_jet_accuracy‚0.37242767214775085 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 _runtime‚13276.202152641 6 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.999009907245636 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.18507443368434906 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 *jet/accuracy_2_of_2‚0.1542491763830185 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7142857313156128 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7257859110832214 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7758315205574036 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7506373524665833 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.21918928623199463 *jet/accuracy_1_of_2‚0.7841231822967529 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6384945511817932 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.047466885298490524 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9464154243469238 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.21280519664287567 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.1750645488500595 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.22988121211528778 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7745379209518433 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 particle/accuracy_3_of_4‚1 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7176269888877869 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.46990740299224854 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9897685050964355 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9583333134651184 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6912494897842407 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8396018743515015 particle/accuracy_5_of_5‚0 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.20921556651592255 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9965004920959473 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7251234650611877 'particle/f_score‚0.6510772705078125 _step‚659 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.2906171977519989 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7296561002731323 jet/accuracy_2_of_5‚1 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6861165761947632 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6743914484977722 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 jet/accuracy_4_of_5‚0 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.037863560020923615 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8917797803878784 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9993476867675781 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.5941483378410339 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.048153530806303024 (particle/accuracy‚0.8532560467720032 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9993476867675781 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.23609218001365662 *jet/accuracy_1_of_1‚0.3305790424346924 , Purity/0FRt2SRqqt1FBt/FBt_purity‚0.96875 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5581051111221313 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 +jet/accuracy_3_of_4‚0.09675586968660355 6 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.842706024646759 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.22148717939853668 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9964677691459656 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 +jet/accuracy_3_of_3‚0.07046087086200714 *jet/accuracy_1_of_4‚0.9639049768447876 ,jet/accuracy_4_of_4‚0.013899499550461769 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3232397437095642 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt0SRqqt0FBt/event_purity‚0.0352289043366909 *jet/accuracy_1_of_3‚0.9151612520217896 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9963976740837097 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.24180252850055695 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8396018743515015 *jet/accuracy_2_of_4‚0.7909020185470581 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 /particle/accuracy_4_of_4‚0.9467192888259888 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9926258325576782“‡mG<“å!‚(:ˆ œéÒ÷æõö gpu.0.gpu‚89.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚41.0 gpu.0.powerWatts‚262.895' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.579 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0±¢æ!‚(:ˆ œéÒƒðÁù disk.dm-0.in‚ 30302.875 disk.drbd2165.in‚0 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.nvme0c0n1.in‚ 30302.875 disk.dm-1.in‚ 75.2734375 disk.sda1.in‚0$ memory_percent‚88.11709880828857 disk.rbd0.in‚0 disk.drbd2165.out‚0 disk.nvme0n1.in‚ 30302.875) proc.memory.percent‚17.23177433013916) disk./.usagePercent‚37.60889024461376 disk.sda.out‚ 694.23828125 disk.sda3.in‚ 75.2734375 network.sent‚ 1.5285411e+07 network.recv‚ 3.314457e+06 disk.sda3.out‚ 694.23828125# disk.nvme0n1.out‚132314.1796875# proc.memory.rssMB‚ 2823.25390625 cpu‚25.563477275688363 disk.sda2.in‚0 disk.dm-0.out‚132314.1796875 disk.rbd1.in‚ 0.50830078125 disk.sda.in‚ 75.2734375 disk.dm-1.out‚ 694.23828125% disk.nvme0c0n1.out‚132314.1796875 proc.cpu.threads‚17 disk.sda1.out‚0 disk.sda2.out‚0) proc.memory.availableMB‚ 1946.89453125 disk.rbd0.out‚0$ disk./.usageGB‚560.3996429443359 disk.rbd1.out‚ 35.146484375»Užç!‚(:“ œéÒß¡µø gpu.0.gpu‚99.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚329.609' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚65.92179999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0øöW&è!‚(:› œéÒáØ—ù disk.drbd2165.out‚0 disk.rbd0.in‚0 disk.sda3.in‚ 75.2734375 disk.dm-1.in‚ 75.2734375 disk.dm-1.out‚ 694.6953125' proc.memory.availableMB‚ 1947.015625 disk.sda.out‚ 694.6953125$ disk./.usageGB‚560.4086875915527# disk.dm-0.out‚132322.8955078125 disk.rbd3.in‚0( disk.nvme0c0n1.out‚132322.8955078125 disk.rbd1.out‚ 35.208984375 proc.cpu.threads‚17 disk.sda.in‚ 75.2734375) disk./.usagePercent‚37.60949723847745 disk.rbd0.out‚0 disk.sda1.in‚0" disk.nvme0n1.in‚30410.83203125 disk.sda2.in‚0$ disk.nvme0c0n1.in‚30410.83203125& disk.nvme0n1.out‚132322.8955078125# proc.memory.rssMB‚ 2823.25390625) proc.memory.percent‚17.23177433013916 disk.sda3.out‚ 694.6953125 network.sent‚ 1.5343357e+07 network.recv‚ 3.318644e+06 disk.drbd2165.in‚0 disk.rbd3.out‚0$ memory_percent‚88.11635971069336 disk.dm-0.in‚30410.83203125 disk.sda1.out‚0 disk.rbd1.in‚ 0.50830078125 disk.sda2.out‚0 cpu‚25.563707266941773²Kn[žé!‚(:“ ŸœéÒÛòêö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚279.55' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚55.910000000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0FJí_*ê!‚(:Ÿ ŸœéÒÇé©ù disk.sda2.out‚0$ disk.dm-0.out‚132339.68603515625 disk.sda1.out‚0$ memory_percent‚88.11638355255127" disk.nvme0n1.in‚30416.16796875 disk.dm-0.in‚30416.16796875' disk.nvme0n1.out‚132339.68603515625 disk.sda2.in‚0 disk.drbd2165.out‚0 cpu‚25.563784975137548$ disk./.usageGB‚560.4130630493164 disk.rbd1.out‚ 35.208984375 disk.sda.in‚ 75.2734375 disk.dm-1.in‚ 75.2734375 disk.sda3.out‚ 695.0859375 disk.drbd2165.in‚0 disk.rbd3.in‚0 disk.rbd3.out‚0 network.sent‚ 1.5347897e+07 network.recv‚ 3.32222e+06 disk.rbd1.in‚ 0.50830078125 disk.dm-1.out‚ 695.0859375 disk.sda1.in‚0 disk.sda3.in‚ 75.2734375) disk./.usagePercent‚37.60979087911922$ disk.nvme0c0n1.in‚30416.16796875 disk.sda.out‚ 695.0859375) proc.memory.availableMB‚ 1947.01171875# proc.memory.rssMB‚ 2823.25390625) proc.memory.percent‚17.23177433013916) disk.nvme0c0n1.out‚132339.68603515625 disk.rbd0.in‚0 disk.rbd0.out‚0 proc.cpu.threads‚17¦¡© ”ë!‚(:‰ ®œéÒç’‡÷ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚40.0 gpu.0.powerWatts‚232.459' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚46.4918 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0„Ð×ì!‚(:” ®œéÒù„‘ú disk.sda1.in‚0 disk.sda3.in‚ 75.2734375 disk.drbd2165.in‚0 disk.rbd3.in‚0( disk.nvme0c0n1.out‚132349.0986328125 disk.sda2.out‚0) disk./.usagePercent‚37.61025860488428 disk.sda2.in‚0 disk.rbd1.out‚ 35.271484375) proc.memory.availableMB‚ 1946.76171875 disk.rbd1.in‚ 0.50830078125 disk.sda.out‚ 695.3046875 disk.dm-1.in‚ 75.2734375 disk.sda1.out‚0 disk.nvme0n1.in‚ 30416.40625& disk.nvme0n1.out‚132349.0986328125# proc.memory.rssMB‚ 2823.25390625! disk.nvme0c0n1.in‚ 30416.40625 disk.rbd0.out‚0 network.sent‚ 1.5353498e+07$ disk./.usageGB‚560.4200325012207 disk.sda.in‚ 75.2734375 disk.dm-1.out‚ 695.3046875 disk.drbd2165.out‚0 disk.rbd3.out‚0 network.recv‚ 3.325725e+06 disk.rbd0.in‚0 disk.dm-0.in‚ 30416.40625$ memory_percent‚88.11790943145752) proc.memory.percent‚17.23177433013916 proc.cpu.threads‚17# disk.dm-0.out‚132349.0986328125 disk.sda3.out‚ 695.3046875 cpu‚25.563897304421598ðÜÕ7•í!‚(:Š ½œéÒŽÊøö gpu.0.gpu‚100.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚326.764' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚65.3528 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0†' Ê/î!‚(:¤ ½œéÒì‚Éù$ disk.nvme0c0n1.in‚30416.80859375 disk.rbd0.out‚0# proc.memory.rssMB‚ 2823.25390625 network.sent‚ 1.5358588e+07 network.recv‚ 3.329095e+06) disk.nvme0c0n1.out‚132364.11865234375 disk.sda2.in‚0 disk.sda.in‚ 75.2734375 disk.dm-1.out‚ 695.51953125 disk.sda1.out‚0 disk.sda3.in‚ 75.2734375$ disk./.usageGB‚560.4182167053223 disk.sda2.out‚0 disk.sda.out‚ 695.51953125 disk.drbd2165.out‚0 disk.sda1.in‚0) proc.memory.percent‚17.23177433013916 proc.cpu.threads‚17 disk.rbd0.in‚0 cpu‚25.564089581121543 disk.rbd1.out‚ 35.271484375" disk.nvme0n1.in‚30416.80859375 disk.rbd1.in‚ 0.50830078125) proc.memory.availableMB‚ 1946.68359375* disk./.usagePercent‚37.610136745297986 disk.rbd3.in‚0 disk.sda3.out‚ 695.51953125 disk.drbd2165.in‚0 disk.rbd3.out‚0' disk.nvme0n1.out‚132364.11865234375 disk.dm-1.in‚ 75.2734375 disk.dm-0.in‚30416.80859375$ disk.dm-0.out‚132364.11865234375$ memory_percent‚88.11838626861572„yl4ï!/ _wandbruntime‚13360  _runtime‚13360=ÕþÉð!à ' lr-AdamW/pg1‚0.00046284099808463627 ' lr-AdamW/pg2‚0.00046284099808463627 trainer/global_step‚8049 $ _timestamp‚1.784303176387864e+09 _runtime‚13360.605071387 _step‚660:Ÿ)Îñ!È trainer/global_step‚8049 $ _timestamp‚1.784303176387864e+09 _runtime‚13360.605071387 _step‚660 ' lr-AdamW/pg1‚0.00046284099808463627 ' lr-AdamW/pg2‚0.00046284099808463627”ÿªèÊ4ò!/ _wandbruntime‚13360  _runtime‚13360–Ö b$ó!þO ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2314552982167552 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚43498471 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚20653714 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚32439296 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚69732025 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚69722052 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚229 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2313553006800896 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2314552982167552 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚68702785 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚624919578120192 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚8049 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚69722052 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7574357604864 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7573012215296 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚634688767965696 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7561397219002 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22844757 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚634689168496128 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚68702791 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22844528 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2321113033991608 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7574357604864 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 138434816 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2322125994382848 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚234 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚43498237 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 _runtime‚13360.606370189 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 138424837 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚624919946211328 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2322127365945856 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚20653709 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9769222284800 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚68702785 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2321114404019898 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2313552981372928 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚0nüí¢2322125994382848 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚68702791 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9769189845504 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7560052618680 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚624919946211328 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2322127365945856 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚20653709 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9769222284800 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚68702785 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2321114404019898 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2313552981372928 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2322125994382848 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚68702791 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9769189845504 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7560052618680 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚69732025 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚69722052 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚229 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2313553006800896 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2314552982167552 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚68702785 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚624919578120192 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚8049 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚69722052 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7574357604864 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7573012215296 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚634688767965696 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7561397219002 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22844757 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚634689168496128 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚68702791 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22844528 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2321113033991608 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7574357604864 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 138434816 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2322125994382848 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚234 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚43498237•ÍÉ94õ!/ _wandbruntime‚13361  _runtime‚13361à³¾ 'ö!‡N REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚20656252 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9770431148544 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚217 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7575282647552 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚634767463244288 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚624997064527872 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 400523264 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚634767863767552 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚32432128 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚69740517 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2313830863151104 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7560976084926 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7573937250816 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚222 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2313830888579072 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 138451514 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚68710997 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚43503435 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2314831009071104 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7575282647552 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 138441521 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2314830982897664 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7573937250816 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚624997432619008 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 _step‚662 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚69730530 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2322404920148480 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2322406291718656 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22847183 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7562320685360ÃC'÷!ŒN EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2322406291718656 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22847183 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7562320685360 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚20656252 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9770431148544 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚217 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7575282647552 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚634767463244288 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚624997064527872 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 400523264 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚634767863767552 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚32432128 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚69740517 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2313830863151104 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7560976084926 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7573937250816 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚222 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2313830888579072 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 138451514 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚68710997 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚43503435 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2314831009071104 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7575282647552 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 138441521 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2314830982897664 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7573937250816 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚624997432619008 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 _step‚662 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚69730530 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2322404920148480–¨lh4ø!/ _wandbruntime‚13362  _runtime‚13362áož ù!† &loss/total_loss‚7.2010555006791765 epoch‚20 _runtime‚13362.078231044 1loss/SRqqt1/assignment_loss‚0.386516614057051 2loss/FBt2/assignment_loss‚0.023814845456052565 /loss/FRt1/detection_loss‚0.5046767616870577 1loss/SRqqt2/detection_loss‚0.5475334005943583 0loss/FRt2/assignment_loss‚0.5422013869968586 2loss/SRqqt2/assignment_loss‚0.4477733263522245 0loss/FBt1/detection_loss‚0.45330959060466136 $ _timestamp‚1.784303177861283e+09 _step‚663 0loss/FRt1/assignment_loss‚1.3793326630478813 3loss/FBt1/assignment_loss‚0.0010746715796792845 1loss/SRqqt1/detection_loss‚0.8924304416632859 /loss/FBt2/detection_loss‚0.5107029976297013 trainer/global_step‚8049 /loss/FRt2/detection_loss‚1.5116888010103642r®åŠú!‹ 1loss/SRqqt1/detection_loss‚0.8924304416632859 /loss/FBt2/detection_loss‚0.5107029976297013 trainer/global_step‚8049 /loss/FRt2/detection_loss‚1.5116888010103642 &loss/total_loss‚7.2010555006791765 epoch‚20 _runtime‚13362.078231044 1loss/SRqqt1/assignment_loss‚0.386516614057051 2loss/FBt2/assignment_loss‚0.023814845456052565 /loss/FRt1/detection_loss‚0.5046767616870577 1loss/SRqqt2/detection_loss‚0.5475334005943583 0loss/FRt2/assignment_loss‚0.5422013869968586 2loss/SRqqt2/assignment_loss‚0.4477733263522245 0loss/FBt1/detection_loss‚0.45330959060466136 $ _timestamp‚1.7843031ÿëß´‡77861283e+09 _step‚663 0loss/FRt1/assignment_loss‚1.3793326630478813 3loss/FBt1/assignment_loss‚0.0010746715796792845—vךžû!‚(:“ ÌœéÒÌÖö÷ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚295.687' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚59.13740000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ë@9O&ü!‚(:› ÌœéÒ¯“„ú disk.dm-1.in‚ 75.2734375' disk.nvme0c0n1.out‚132372.673828125% disk.nvme0n1.out‚132372.673828125 disk.rbd1.in‚ 0.50830078125 disk.dm-1.out‚ 695.68359375 disk.dm-0.in‚30417.01171875 disk.sda1.out‚0" disk.nvme0n1.in‚30417.01171875' proc.memory.availableMB‚ 1945.796875 disk.sda2.in‚0 network.sent‚ 1.5363631e+07$ disk./.usageGB‚560.4249229431152 disk.sda.out‚ 695.68359375$ disk.nvme0c0n1.in‚30417.01171875 disk.sda.in‚ 75.2734375" disk.dm-0.out‚132372.673828125 disk.sda3.in‚ 75.2734375 disk.drbd2165.out‚0 disk.rbd3.in‚0) proc.memory.percent‚17.23177433013916* disk./.usagePercent‚37.610586806543175 disk.rbd0.in‚0 disk.sda1.in‚0 disk.sda3.out‚ 695.68359375 disk.drbd2165.in‚0 disk.rbd3.out‚0$ memory_percent‚88.12379837036133 cpu‚25.56420273139279 network.recv‚ 3.332387e+06 disk.rbd0.out‚0 disk.sda2.out‚0 disk.rbd1.out‚ 35.333984375# proc.memory.rssMB‚ 2823.25390625 proc.cpu.threads‚17Ý-”ý!‚(:‰ ÛœéÒå‘„÷ gpu.0.gpu‚32.0 gpu.0.memory‚10, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚335.402' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚67.0804 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0š”ÑÁþ!‚(:” ÛœéÒ±ªãù) disk./.usagePercent‚37.61168379882739 disk.dm-1.out‚ 696.0546875 disk.rbd0.out‚0 disk.drbd2165.out‚0! disk.nvme0c0n1.in‚ 30561.53125) proc.memory.percent‚17.23177433013916 disk.sda.in‚ 75.2734375 disk.dm-1.in‚ 75.2734375# proc.memory.rssMB‚ 2823.25390625 network.sent‚ 1.5426012e+07 network.recv‚ 3.337332e+06' disk.nvme0n1.out‚132384.17138671875 disk.sda2.out‚0 disk.rbd1.in‚ 0.50830078125 disk.sda.out‚ 696.0546875 disk.sda3.in‚ 75.2734375 disk.rbd3.in‚0 proc.cpu.threads‚17 disk.sda2.in‚0 disk.dm-0.in‚ 30561.53125 disk.sda3.out‚ 696.0546875 disk.nvme0n1.in‚ 30561.53125' proc.memory.availableMB‚ 973.9296875 cpu‚25.56439295456919 disk.rbd1.out‚ 35.333984375$ disk./.usageGB‚560.4412689208984$ disk.dm-0.out‚132384.17138671875 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd2165.in‚0) disk.nvme0c0n1.out‚132384.17138671875$ memory_percent‚94.05560493469238 disk.rbd0.in‚0 disk.rbd3.out‚0U”ÿ!‚(:‰ êœéÒÈçêö gpu.0.gpu‚91.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚277.374' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.4748 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0`Ϲ¥€"‚(:ˆ êœéÒ¢ã’ù disk.sda3.out‚ 696.3984375 disk.nvme0n1.in‚30561.75# proc.memory.rssMB‚ 2823.25390625 cpu‚25.564491161095702 proc.cpu.threads‚17* disk./.usagePercent‚37.611986911747934$ disk./.usageGB‚560.4457855224609 disk.sda.in‚ 75.2734375 disk.rbd3.out‚0 disk.rbd0.in‚0 disk.rbd0.out‚0# disk.dm-0.out‚132396.8408203125 disk.sda1.in‚0 disk.drbd2165.in‚0 disk.rbd1.in‚ 0.50830078125 disk.sda2.in‚0# memory_percent‚94.0556526184082 disk.sda3.in‚ 75.2734375 network.sent‚ 1.5431603e+07 disk.dm-1.in‚ 75.2734375 disk.dm-1.out‚ 696.3984375 disk.dm-0.in‚30561.75& disk.nvme0n1.out‚132396.8408203125 disk.sda2.out‚0 disk.sda.out‚ 696.3984375 disk.drbd2165.out‚0 disk.rbd3.in‚0( disk.nvme0c0n1.out‚132396.8408203125& proc.memory.availableMB‚ 973.921875) proc.memory.percent‚17.23177433013916 network.recv‚ 3.340436e+06 disk.nvme0c0n1.in‚30561.75 disk.rbd1.out‚ 35.396484375 disk.sda1.out‚0âšP“"‚(:ˆ ùœéÒ¦Ûôö gpu.0.gpu‚10.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚40.0 gpu.0.powerWatts‚327.587' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚65.5174 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚01Pz‚"‚(:” ùœéÒÕì€ù disk.sda3.out‚ 696.796875 disk.rbd1.out‚ 35.396484375 disk.sda.out‚ 696.796875 disk.sda2.in‚0 disk.rbd1.in‚ 0.50830078125 disk.sda1.in‚0 cpu‚25.56466164118848 proc.cpu.threads‚17 disk.drbd2165.out‚0 disk.dm-1.out‚ 696.796875$ disk.dm-0.out‚132406.94189453125* disk./.usagePercent‚37.612421356533545 disk.rbd3.out‚0) disk.nvme0c0n1.out‚132406.94189453125 disk.nvme0n1.in‚ 30561.953125 disk.rbd0.out‚0 disk.sda1.out‚0) proc.memory.percent‚17.23177433013916 network.recv‚ 3.344449e+06$ disk./.usageGB‚560.4522590637207 disk.rbd3.in‚0 disk.sda2.out‚0 disk.drbd2165.in‚0 disk.sda.in‚ 75.2734375 disk.dm-1.in‚ 75.2734375 disk.rbd0.in‚0 network.sent‚ 1.5436885e+07" disk.nvme0c0n1.in‚ 30561.953125' disk.nvme0n1.out‚132406.94189453125 disk.dm-0.in‚ 30561.953125# memory_percent‚94.0558910369873# proc.memory.rssMB‚ 2823.25390625 disk.sda3.in‚ 75.2734375' proc.memory.availableMB‚ 973.8828125ûößb’ƒ"‚(:‡ ˆéÒ¶Üêö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚269.59' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.918 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Îlà„"‚(:† ˆéÒ¶Ó¦ù disk.drbd2165.in‚0 disk.rbd1.out‚ 35.458984375( proc.memory.availableMB‚ 974.26953125 disk.nvme0c0n1.in‚ 30562.375 disk.dm-1.in‚ 75.2734375 disk.rbd0.in‚0 disk.dm-0.in‚ 30562.375 cpu‚25.564811434248654 disk.sda3.in‚ 75.2734375 disk.nvme0n1.in‚ 30562.375 disk.sda.in‚ 75.2734375 disk.sda.out‚ 696.921875 network.recv‚ 3.347561e+06# proc.memory.rssMB‚ 2823.25390625 disk.sda1.out‚0) disk./.usagePercent‚37.61289701853216 disk.sda3.out‚ 696.921875 disk.rbd3.in‚0 disk.rbd1.in‚ 0.50830078125 disk.rbd0.out‚0 proc.cpu.threads‚17 disk.sda1.in‚0$ disk./.usageGB‚560.4593467712402 disk.rbd3.out‚0' disk.nvme0c0n1.out‚132419.607421875 disk.sda2.in‚0) proc.memory.percent‚17.23177433013916 network.sent‚ 1.5441913e+07 disk.sda2.out‚0 disk.dm-1.out‚ 696.921875" disk.dm-0.out‚132419.607421875# memory_percent‚94.0535306930542 disk.drbd2165.out‚0% disk.nvme0n1.out‚132419.607421875”ìÑi4…"/ _wandbruntime‚13439  _runtime‚13439¹òƆ"À $ _timestamp‚1.784303255723776e+09 _runtime‚13439.94096282 _step‚664 & lr-AdamW/pg1‚0.0004602280884311194 & lr-AdamW/pg2‚0.0004602280884311194 trainer/global_step‚8099\_ôˇ"Å & lr-AdamW/pg1‚0.0004602280884311194 & lr-AdamW/pg2‚0.0004602280884311194 trainer/global_step‚8099 $ _timestamp‚1.784303255723776e+09 _runtime‚13439.94096282 _step‚664˜^´Ú4ˆ"/ _wandbruntime‚13439  _runtime‚13439FA`N(‰"þO [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚70157777 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚20780859 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚69113091 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚638623791437312 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7607634332110 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 _runtime‚13439.942196058 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 139270868 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22981295 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2336072346819072 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7619328145920 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚265 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚43762159 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚20780864 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 % _timestamp‚1.7843032557248645e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2336073718382080 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2327447070281728 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚628793898504192 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚69113085 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚69113085 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚43761894 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚638623388809728 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22981035 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2328453018673152 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚70147804 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚70147804 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2327447095709696 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2328453044846592 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2328453044846592 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 139270868 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22981295 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2336072346819072 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7619328145920 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚265 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚43762159 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚20780864 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 % _timestamp‚1.7843032557248645e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2336073718382080 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2327447070281728 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚628793898504192 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚69113085 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚69113085 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚43761894 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚638623388809728 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22981035 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2328453018673152 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973™ „qž‹"‚(:“ —éÒ´Üñö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚288.679' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚57.73579999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0fD÷ÈŒ"‚(: —éÒŽåÖú disk.sda.out‚697.25 disk.dm-1.out‚697.25) disk.nvme0c0n1.out‚132429.81103515625) disk./.usagePercent‚37.61319629133968 disk.sda1.in‚0' disk.nvme0n1.out‚132429.81103515625 disk.sda3.in‚ 75.2734375 disk.sda3.out‚697.25 disk.rbd3.in‚0 disk.rbd1.in‚ 0.50830078125 disk.dm-1.in‚ 75.2734375 disk.dm-0.in‚30676.87890625 proc.cpu.threads‚17 network.recv‚ 3.351156e+06 disk.rbd0.in‚0( proc.memory.availableMB‚ 973.78515625 cpu‚25.56497519183089$ disk.dm-0.out‚132429.81103515625 network.sent‚ 1.5446989e+07 disk.sda1.out‚0 disk.drbd2165.in‚0$ disk.nvme0c0n1.in‚30676.87890625" disk.nvme0n1.in‚30676.87890625 disk.sda2.out‚0 disk.sda.in‚ 75.2734375 disk.rbd1.out‚ 35.458984375) proc.memory.percent‚17.23177433013916 disk.rbd3.out‚0 disk.rbd0.out‚0# proc.memory.rssMB‚ 2823.25390625$ disk./.usageGB‚560.4638061523438 disk.drbd2165.out‚0 disk.sda2.in‚0$ memory_percent‚94.05648708343506ŠF"4"/ _wandbruntime‚13441  _runtime‚13441Á€¢ò Ž"ˆN RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 139277573 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2335333535286340 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2328731045576704 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2327724977487872 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚259 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9830684971520 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 139287566 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚43767141 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7621598578176 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚254 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚70156282 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚70156282 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚628871384911872 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚69121297 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7621598578176 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚638702069883392 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚22983993 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 trainer/global_step‚8099 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚70156282 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚628871384911872 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚69121297 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7621598578176 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚638702069883392 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚22983993 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 trainer/global_step‚8099 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 139277573 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2335333535286340 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2328731045576704 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2327724977487872 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚259 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9830684971520 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 139287566 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚43767141 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7621598578176 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚254 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚70156282 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 QDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚20908014 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9889308833280 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9889274296832 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7652526844896 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2348995056063714 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2350018699255296 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2350020070818304 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚8149 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚44025881 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚20908009 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2342353081352192 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚70573556 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚251 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚69523391 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚69523385 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚44025625 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 VADeviceStatsMonitor.on_train_bƉµûatch_start/requested_bytes.all.freed‚2348993686035424 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚632668586979328 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7665644076544 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7653871445218 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚642557895812608 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2350020070818304 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2342353081352192 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚69523385 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚70583529 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2342353055178752 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2341341159190528 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚70583529 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7666989466112 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚642557493185024 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 _step‚669 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7666989466112 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 140096941 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚23117616 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008G¦/ (¢"ƒP WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚632668218888192 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2341341184618496 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 140106920 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚20908014 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9889308833280 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9889274296832 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7652526844896 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2348995056063714 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2350018699255296 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2350020070818304 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚8149 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚44025881 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚20908009 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2342353081352192 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚70573556 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚251 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚69523391 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚69523385 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚44025625 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2348993686035424 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚632668586979328 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7665644076544 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7653871445218 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚642557895812608 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2350020070818304 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2342353081352192 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚69523385 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚70583529 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2342353055178752 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2341341159190528 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚70583529 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7666989466112 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚642557493185024 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 _step‚669 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7666989466112 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 140096941 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚23117616 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚256 _runtime‚13519.450742313 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 _runtime‚13520.839616503 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 140113625 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7654794911576 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚44030934 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2342631082082304 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2342631055908864 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9890508808704 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚70592021 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚23120622 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2350298996591104 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7666569112064 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 trainer/global_step‚8149 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7666569112064 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚44031179 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚642636179575296 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2349273861308248 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚70592021 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚245 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚20910552 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7667914508800 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7667914508800 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚69531591 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚632745705295872 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2341619040968704 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2342631055908864 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚70582034 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2349272491279846 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 140123618 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚70582034 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚240 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 140123618 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚642636582195712 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2342631082082304 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2350297625020928 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 % _timestamp‚1.7843033366222415e+09µšä[v¥"N YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9890508808704 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚70592021 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚23120622 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2350298996591104 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7666569112064 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 trainer/global_step‚8149 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7666569112064 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚44031179 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚642636179575296 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2349273861308248 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚70592021 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚245 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚20910552 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7667914508800 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7667914508800 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚69531591 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚632745705295872 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2341619040968704 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2342631055908864 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚70582034 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2349272491279846 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 140123618 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚70582034 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚240 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 140123618 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚642636582195712 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed°%‚‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2342631082082304 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2350297625020928 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 % _timestamp‚1.7843033366222415e+09 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 _runtime‚13520.839616503 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 140113625 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7654794911576 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚44030934 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2342631082082304 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2342631055908864ž¸}4¦"/ _wandbruntime‚13520  _runtime‚13520÷x! §"ƒ 3loss/SRqqt2/assignment_loss‚0.39532647484947686 1loss/SRqqt1/detection_loss‚0.8094529799170426 1loss/SRqqt2/detection_loss‚0.5712752017649689 %loss/total_loss‚6.867584658816361 $ _timestamp‚1.784303336706433e+09 0loss/FRt1/assignment_loss‚1.3891840126139647 _runtime‚13520.92337114 2loss/SRqqt1/assignment_loss‚0.4505740547743665 2loss/FBt1/assignment_loss‚0.005862602037120407 .loss/FRt2/detection_loss‚1.348179918278399 trainer/global_step‚8149 0loss/FRt2/assignment_loss‚0.5229532306925614 1loss/FBt2/assignment_loss‚0.00555863645839478 0loss/FRt1/detection_loss‚0.47628176375619524 /loss/FBt1/detection_loss‚0.5081932308359278 /loss/FBt2/detection_loss‚0.3847425528379433 epoch‚20 _step‚671kgì¨"ˆ 0loss/FRt2/assignment_loss‚0.5229532306925614 1loss/FBt2/assignment_loss‚0.00555863645839478 0loss/FRt1/detection_loss‚0.47628176375619524 /loss/FBt1/detection_loss‚0.5081932308359278 /loss/FBt2/detection_loss‚0.3847425528379433 epoch‚20 _step‚671 3loss/SRqqt2/assignment_loss‚0.39532647484947686 1loss/SRqqt1/detection_loss‚0.8094529799170426 1loss/SRqqt2/detection_loss‚0.5712752017649689 %loss/total_loss‚6.867584658816361 $ _timestamp‚1.784303336706433e+09 0loss/FRt1/assignment_loss‚1.3891840126139647 _runtime‚13520.92337114 2loss/SRqqt1/assignment_loss‚0.4505740547743665 2loss/FBt1/assignment_loss‚0.005862602037120407 .loss/FRt2/detection_loss‚1.348179918278399 trainer/global_step‚8149ŸÒoz”©"‚(:‰ ñéÒ‚¾“ø gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚265.534' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.1068 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Qš‹¶ª"‚(:‹ ñéÒþ•Ìø' disk.nvme0c0n1.out‚132529.646484375 disk.rbd1.in‚ 0.50830078125 disk.rbd3.out‚0" proc.memory.availableMB‚974.25 network.sent‚ 1.5572126e+07 disk.sda1.out‚0 disk.drbd2165.out‚0 proc.cpu.threads‚17% disk.nvme0n1.out‚132529.646484375 disk.sda.out‚ 706.15625 disk.dm-1.in‚ 75.2734375 disk.dm-1.out‚ 706.15625 disk.dm-0.in‚ 30817.6953125# disk.nvme0c0n1.in‚ 30817.6953125 disk.sda2.out‚0" disk.dm-0.out‚132529.646484375 disk.sda1.in‚0 cpu‚25.565880038647222 network.recv‚ 3.373868e+06* disk./.usagePercent‚37.615967316890305$ disk./.usageGB‚560.5050964355469 disk.rbd0.in‚0 disk.rbd0.out‚0# proc.memory.rssMB‚ 2823.25390625) proc.memory.percent‚17.23177433013916! disk.nvme0n1.in‚ 30817.6953125 disk.sda3.in‚ 75.2734375 disk.sda2.in‚0 disk.rbd1.out‚ 35.646484375 disk.sda3.out‚ 706.15625 disk.drbd2165.in‚0 disk.rbd3.in‚0 disk.sda.in‚ 75.2734375$ memory_percent‚94.05364990234375cF…•”«"‚(:‰ €žéÒå£çö gpu.0.gpu‚84.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚40.0 gpu.0.powerWatts‚112.997' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.5994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ã@&¬"‚(:› €žéÒŒ¬ºù disk.sda.out‚ 709.41796875 disk.rbd0.in‚0 disk.sda3.in‚ 75.2734375 disk.sda3.out‚ 709.41796875* disk./.usagePercent‚37.616679017835494$ disk./.usageGB‚560.5157012939453 disk.rbd1.out‚ 35.771484375 disk.rbd0.out‚0# proc.memory.rssMB‚ 2823.25390625 disk.dm-1.out‚ 709.41796875 disk.rbd3.in‚0 cpu‚25.56609882717608 disk.sda1.out‚0 disk.drbd2165.out‚0 disk.rbd3.out‚0# memory_percent‚94.0542221069336 network.sent‚ 1.5635114e+07 disk.rbd1.in‚ 0.50830078125 disk.dm-1.in‚ 75.2734375 disk.dm-0.in‚ 30818.1171875$ disk.dm-0.out‚132552.87646484375% proc.memory.availableMB‚ 974.15625 disk.drbd2165.in‚0! disk.nvme0n1.in‚ 30818.1171875 disk.sda.in‚ 75.2734375 disk.sda1.in‚0 disk.sda2.out‚0# disk.nvme0c0n1.in‚ 30818.1171875) disk.nvme0c0n1.out‚132552.87646484375) proc.memory.percent‚17.23177433013916 proc.cpu.threads‚17' disk.nvme0n1.out‚132552.87646484375 network.recv‚ 3.378167e+06 disk.sda2.in‚0ðö %”­"‚(:‰ žéÒ ¥ãö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚290.327' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚58.0654 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0£×P7®"‚(:Ž žéÒÍ™‡ú disk.dm-1.out‚ 713.1171875" disk.nvme0c0n1.in‚ 30937.609375' proc.memory.availableMB‚ 973.8984375) proc.memory.percent‚17.23177433013916 disk.rbd0.out‚0# proc.memory.rssMB‚ 2823.25390625 disk.sda.out‚ 713.1171875$ disk.nvme0n1.out‚132576.51171875$ memory_percent‚94.05579566955566 disk.dm-0.in‚ 30937.609375 disk.sda1.in‚0 disk.sda3.in‚ 75.2734375 disk.rbd1.in‚ 0.50830078125) disk./.usagePercent‚37.61665444111221$ disk./.usageGB‚560.5153350830078 disk.rbd0.in‚0 disk.sda3.out‚ 713.1171875 disk.rbd3.in‚0 cpu‚25.56622573380229 proc.cpu.threads‚17 network.recv‚ 3.380901e+06! disk.dm-0.out‚132576.51171875 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.drbd2165.out‚0& disk.nvme0c0n1.out‚132576.51171875 disk.nvme0n1.in‚ 30937.609375 disk.sda2.in‚0 network.sent‚ 1.5640015e+07 disk.rbd3.out‚0 disk.sda2.out‚0 disk.rbd1.out‚ 35.771484375 disk.sda.in‚ 75.2734375 disk.dm-1.in‚ 75.2734375«$¥”¯"‚(:‰ žžéÒÝ„÷ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚39.0 gpu.0.powerWatts‚110.656' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.1312 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0üñî^ °"‚(:• žžéÒ¡óù* disk./.usagePercent‚37.617124982960135 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda3.out‚ 713.61328125 disk.rbd1.out‚ 35.833984375 disk.sda.in‚ 75.2734375 disk.sda.out‚ 713.61328125( proc.memory.availableMB‚ 973.92578125# proc.memory.rssMB‚ 2823.25390625 disk.nvme0n1.in‚ 30937.828125 disk.rbd1.in‚ 0.50830078125 proc.cpu.threads‚17# disk./.usageGB‚560.522346496582" disk.nvme0c0n1.in‚ 30937.828125 disk.rbd0.in‚0) proc.memory.percent‚17.23177433013916 network.sent‚ 1.5644312e+07 disk.sda3.in‚ 75.2734375 disk.drbd2165.out‚0 disk.sda2.out‚0 disk.dm-1.out‚ 713.61328125 cpu‚25.566355503070238 disk.rbd0.out‚0 disk.rbd3.in‚0' disk.nvme0c0n1.out‚132584.330078125 disk.sda2.in‚0 disk.dm-1.in‚ 75.2734375 disk.dm-0.in‚ 30937.828125" disk.dm-0.out‚132584.330078125 disk.drbd2165.in‚0 disk.rbd3.out‚0% disk.nvme0n1.out‚132584.330078125 network.recv‚ 3.384293e+06# memory_percent‚94.0556287765503û5i ”±"‚(:‰ ­žéÒȆ÷ gpu.0.gpu‚90.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚260.702' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.1404 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0VÕ ;)²"‚(:ž ­žéÒä×­ø* disk./.usagePercent‚37.617595524808074 disk.dm-1.out‚ 714.23046875 disk.rbd0.out‚0 disk.sda1.in‚0 disk.rbd3.in‚0" disk.nvme0n1.in‚30938.96484375 disk.sda.in‚ 75.2734375 disk.dm-0.in‚30938.96484375 disk.sda3.in‚ 75.2734375 disk.drbd2165.out‚0( proc.memory.availableMB‚ 973.92578125# proc.memory.rssMB‚ 2823.25390625) proc.memory.percent‚17.23177433013916 cpu‚25.566563918447432 disk.sda.out‚ 714.23046875 disk.rbd3.out‚0$ disk.nvme0c0n1.in‚30938.96484375 disk.rbd1.out‚ 35.833984375 proc.cpu.threads‚17 disk.dm-1.in‚ 75.2734375 disk.drbd2165.in‚0( disk.nvme0c0n1.out‚132594.8115234375 disk.sda2.out‚0 disk.sda3.out‚ 714.23046875& disk.nvme0n1.out‚132594.8115234375 disk.rbd1.in‚ 0.50830078125$ disk./.usageGB‚560.5293579101562# disk.dm-0.out‚132594.8115234375 disk.sda2.in‚0 network.recv‚ 3.38791e+06 disk.sda1.out‚0 network.sent‚ 1.5649449e+07 disk.rbd0.in‚0# memory_percent‚94.0556287765503ÑùH™4³"/ _wandbruntime‚13598  _runtime‚13598®Ì]0Æ´"À % _timestamp‚1.7843034144957829e+09 _runtime‚13598.713008498 _step‚672 % lr-AdamW/pg1‚0.000454980412847442 % lr-AdamW/pg2‚0.000454980412847442 trainer/global_step‚8199OáØV˵"Å % lr-AdamW/pg1‚0.000454980412847442 % lr-AdamW/pg2‚0.000454980412847442 trainer/global_step‚8199 % _timestamp‚1.7843034144957829e+09 _runtime‚13598.713008498 _step‚672 ŒÈoÍ4¶"/ _wandbruntime‚13598  _runtime‚13598¥ã2ß(·"þO @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚32439296 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7700108558326 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2355235248099328 _runtime‚13598.714293384 _step‚673 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚23254778 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚646492223581696 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7698763958004 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚69933691 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2356253117857792 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚21035164 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9949316218368 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2362934012057332 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚245 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚636542907363328 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2355235273527296 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚646491823051264 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚69933685 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚70999308 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2363966423254528 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2356253091684352 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7713305396736 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚44290187 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 trainer/global_step‚8199 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2356253117857792 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 % _timestamp‚1.7843034144969347e+09 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚71009281 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2363965051691520 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚69933691 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚23255023 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2363965051691520 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚71009281 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7713305396736 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 400530432 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚636542539272192 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚250 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚70999308 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7711960007168 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2356253091684352Ð@ (¸"ƒP A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2356253117857792 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 % _timestamp‚1.7843034144969347e+09 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚71009281 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2363965051691520 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚69933691 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚23255023 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2363965051691520 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚71009281 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7713305396736 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 400530432 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚636542539272192 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚250 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚70999308 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7711960007168 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2356253091684352 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚32439296 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7700108558326 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2355235248099328 _runtime‚13598.714293384 _step‚673 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚23254778 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚646492223581696 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7698763958004 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚69933691 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2356253117857792 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚21035164 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9949316218368 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2362934012057332 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚245 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚636542907363328 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2355235273527296 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚646491823051264 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚69933685 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚70999308 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2363966423254528 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2356253091684352 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7713305396736 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚44290187 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 trainer/global_step‚8199¡Âd%4¹"/ _wandbruntime‚13600  _runtime‚13600Éì®Iº"‡N E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚243 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚44295271 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2363214187330156 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2355513129877504 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 \JDeviceStatsMonitor.on_train_batch_end/allocõò˜ÖK$ated_bytes.small_pool.allocated‚ 7714230439424 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 140949677 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚636620393771008 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9950520396288 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7714230439424 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚71017773 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 140959670 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7699687424250 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7712885042688 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 trainer/global_step‚8199 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 _step‚674 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚71017773 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚69941891 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚23257807 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2355513155305472 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 140949677 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7712885042688 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚69941897 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2364245349027328 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚646570513644032 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚71007786 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 400523264 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 $ _timestamp‚1.784303415882406e+09 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚44295514 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23257569 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9950487964160 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7701032024684 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 _runtime‚13600.099904236 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚21037702 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚71007786 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚32432128 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚69941891Õ$vø'»"ŒN K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚646570914167296 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2356531092414464 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2364243977457152 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2356531118587904 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2356531092414464 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2364245349027328 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 140959670 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7714230439424 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚71017773 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 140959670 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7699687424250 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7712885042688 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 trainer/global_step‚8199 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 _step‚674 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚71017773 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚69941891 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚23257807 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2355513155305472 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 140949677 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7712885042688 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚69941897 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2364245349027328 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚646570513644032 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚71007786 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 400523264 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 $ _timestamp‚1.784303415882406e+09 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚44295514 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23257569 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9950487964160 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7701032024684 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 _runtime‚13600.099904236 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚21037702 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚71007786 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚32432128 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚69941891 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚243 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚44295271 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2363214187330156 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2355513129877504 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7714230439424 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 140949677 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚636620393771008 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9950520396288 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 FDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7759621327360 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚71435033 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚640417227747328 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚677 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚44555438 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 141769045 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7758275937792 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚650426527299072 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚70343991 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚70343991 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚21162314 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7746345671434 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2377911404127744 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚21162309 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2369129362436096 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚650426927829504 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10009667642880 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 _runtime‚13677.677742532 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚71425060 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚23393366 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2377911404127744 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2369129337008128 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2370153154363392 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2370153128189952 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚32439296 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2377912775690752 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7759621327360 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7758275937792 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚23393129 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820Jø© (Î"„P RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7746345671434 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2377911404127744 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚21162309 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2369129362436096 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚650426927829504 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10009667642880 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 _runtime‚13677.677742532 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚71425060 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚23393366 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2377911404127744 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2369129337008128 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2370153154363392 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2370153128189952 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚32439296 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2377912775690752 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7759621327360 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7758275937792 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚23393129 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚71425060 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 400530432 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚242 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚640416859656192 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚70343985 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2370153154363392 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 trainer/global_step‚8249 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7745001071112 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 $ _timestamp‚1.784303493460442e+09 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7759621327360 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚71435033 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚640417227747328 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚677 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚44555438 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 141769045 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7758275937792 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚650426527299072 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚70343991 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚70343991 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚21162314 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736¥AØ@4Ï"/ _wandbruntime‚13679  _runtime‚13679%B'Ð"‰N K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _step‚678 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 _runtime‚13679.06712824 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚44560745 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2378190329893376 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7759200973312 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚21164852 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚237 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚70352191 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚32432128 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 141785729 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 141795722 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚232 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚70352197 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚71443525 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚650505216081408 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2378191701463552 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2370431128920064 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚640494714155008 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2369407244214272 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚650505616604672 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚71433538 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚70352191 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7760546370048 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2369407218786304 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 141795722 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2370431155093504 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚23396125 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7747269137792 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚640494346063872 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2377153143323662 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175³ÔrOÞÑ"ŽN J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚70352191 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚32432128 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 141785729 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 141795722 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚232 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚70352197 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚71443525 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚650505216081408 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2378191701463552 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2370431128920064 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚640494714155008 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2369407244214272 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚650505616604672 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚71433538 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚70352191 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7760546370048 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2369407218786304 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 141795722 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2370431155093504 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚23396125 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7747269137792 WHDeviceStatsMonitor.on_train_batch_end/requested_Sß?H6bytes.small_pool.current‚ 1344600434 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚640494346063872 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2377153143323662 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _step‚678 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 _runtime‚13679.06712824 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚44560745 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2378190329893376 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7759200973312 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚21164852 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚237 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0¦:kæ4Ò"/ _wandbruntime‚13679  _runtime‚13679»0¿Ó"€ /loss/FRt1/assignment_loss‚1.426089064261088 /loss/FRt1/detection_loss‚0.5101969869345706 /loss/FBt2/detection_loss‚0.4841062747291289 $ _timestamp‚1.784303494935464e+09 0loss/FRt2/assignment_loss‚0.5458488995646253 1loss/FBt2/assignment_loss‚0.00938858977990959 epoch‚20 _runtime‚13679.15238578 /loss/FRt2/detection_loss‚1.3932766475921703 1loss/SRqqt2/detection_loss‚0.5832579267024864 /loss/FBt1/detection_loss‚0.5301128309685736 $loss/total_loss‚7.25543856560056 trainer/global_step‚8249 2loss/SRqqt1/assignment_loss‚0.4932152617059872 2loss/SRqqt2/assignment_loss‚0.4205825633711494 2loss/FBt1/assignment_loss‚0.039585315542594125 1loss/SRqqt1/detection_loss‚0.8197782044482761 _step‚679&$¾ Ô"… /loss/FRt2/detection_loss‚1.3932766475921703 1loss/SRqqt2/detection_loss‚0.5832579267024864 /loss/FBt1/detection_loss‚0.5301128309685736 $loss/total_loss‚7.25543856560056 trainer/global_step‚8249 2loss/SRqqt1/assignment_loss‚0.4932152617059872 2loss/SRqqt2/assignment_loss‚0.4205825633711494 2loss/FBt1/assignment_loss‚0.039585315542594125 1loss/SRqqt1/detection_loss‚0.8197782044482761 _step‚679 /loss/FRt1/assignment_loss‚1.426089064261088 /loss/FRt1/detection_loss‚0.5101969869345706 /loss/FBt2/detection_loss‚0.4841062747291289 $ _timestamp‚1.784303494935464e+09 0loss/FRt2/assignment_loss‚0.5458488995646253 1loss/FBt2/assignment_loss‚0.00938858977990959 epoch‚20 _runtime‚13679.15238578§½0†Õ"‚(:’ ‡ŸéÒÃîæö gpu.0.gpu‚90.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚281.47' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚56.29400000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 Ú22ýÖ"‚(:ò ‡ŸéÒØ­›ù disk.rbd3.out‚0 disk.sda2.out‚0 disk.rbd1.in‚ 0.50830078125( proc.memory.availableMB‚ 974.52734375 cpu‚25.565886193367167 disk.dm-0.in‚ 31133.4453125 disk.sda1.in‚0 disk.rbd3.in‚0) disk.nvme0c0n1.out‚132693.03955078125! disk.nvme0n1.in‚ 31133.4453125 disk.sda2.in‚0 network.recv‚ 3.409873e+06' disk.nvme0n1.out‚132693.03955078125( disk./.usagePercent‚37.6205401234669$ disk.dm-0.out‚132693.03955078125 disk.sda3.in‚ 75.2734375 network.sent‚ 1.5737754e+07 disk.sda.in‚ 75.2734375 disk.dm-1.out‚ 721.4296875 disk.rbd1.out‚ 36.021484375 proc.cpu.threads‚17 disk.dm-1.in‚ 75.2734375 disk.drbd2165.out‚0) proc.memory.percent‚17.23177433013916 disk.sda1.out‚0# disk.nvme0c0n1.in‚ 31133.4453125$ memory_percent‚94.05195713043213# proc.memory.rssMB‚ 2823.25390625 disk.sda3.out‚ 721.4296875$ disk./.usageGB‚560.5732345581055 disk.sda.out‚ 721.4296875 disk.drbd2165.in‚0¡úÞVŸ×"‚(:” –ŸéÒí‡ëö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚284.867' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚56.973400000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÔI\”7Ø"‚(:¬ –ŸéÒÿ¢þø disk.dm-1.out‚ 723.5859375 disk.sda1.out‚0 disk.drbd2165.out‚0( disk.nvme0c0n1.out‚132704.5244140625 disk.sda2.in‚0 disk.rbd1.in‚ 0.50830078125 network.recv‚ 3.414042e+06 disk.sda1.in‚0 disk.sda3.in‚ 75.2734375 disk.sda3.out‚ 723.5859375 network.sent‚ 1.580056e+07$ disk./.usageGB‚560.5778350830078! disk.nvme0n1.in‚ 31137.0546875& disk.nvme0n1.out‚132704.5244140625 disk.sda2.out‚0# proc.memory.rssMB‚ 2823.25390625 disk.sda.in‚ 75.2734375 disk.dm-1.in‚ 75.2734375 disk.rbd3.out‚0 disk.sda.out‚ 723.5859375& disk.rbd0.in‚1.759218604441423e+13 disk.rbd0.out‚0 disk.drbd2165.in‚0 disk.rbd3.in‚0 disk.dm-0.in‚ 31137.0546875# disk.dm-0.out‚132704.5244140625 disk.rbd1.out‚ 36.083984375' proc.memory.availableMB‚ 974.5546875) proc.memory.percent‚17.23177433013916* disk./.usagePercent‚37.620848868553196# disk.nvme0c0n1.in‚ 31137.0546875$ memory_percent‚94.05179023742676 cpu‚25.566015511943736 proc.cpu.threads‚17<ú†9Ù"‚(:® ¥ŸéÒ׿òø disk.sda3.out‚ 725.2109375 proc.cpu.threads‚17 disk.rbd1.in‚ 0.50830078125 disk.rbd1.out‚ 36.083984375 disk.sda3.in‚ 75.2734375 network.sent‚ 1.5805708e+07 disk.sda.out‚ 725.2109375 disk.dm-0.in‚ 31139.4609375 cpu‚25.566253273572126 disk.rbd3.out‚0# disk.nvme0c0n1.in‚ 31139.4609375 disk.rbd0.out‚0$ disk./.usageGB‚560.5871353149414 disk.rbd3.in‚0 disk.sda2.in‚0 disk.dm-1.in‚ 75.2734375 disk.drbd2165.out‚0$ memory_percent‚94.05176639556885 network.recv‚ 3.417756e+06* disk./.usagePercent‚37.621473014921676( disk.nvme0c0n1.out‚132717.8623046875 disk.sda.in‚ 75.2734375& disk.rbd0.in‚1.759218604441423e+13) proc.memory.percent‚17.23177433013916 disk.sda1.in‚0 disk.sda1.out‚0! disk.nvme0n1.in‚ 31139.4609375& disk.nvme0n1.out‚132717.8623046875 disk.sda2.out‚0 disk.dm-1.out‚ 725.2109375# disk.dm-0.out‚132717.8623046875( proc.memory.availableMB‚ 974.55859375 disk.drbd2165.in‚0# proc.memory.rssMB‚ 2823.25390625.~˪•Ú"‚(:Š ¥ŸéÒÀ¶æø gpu.0.gpu‚100.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚328.937' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚65.7874 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0M ¬rŸÛ"‚(:” ´ŸéÒزäö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚284.911' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚56.982200000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Õ% =Ü"‚(:² ´ŸéÒ’âïø disk.sda2.out‚0$ memory_percent‚94.05210018157959 proc.cpu.threads‚17 disk.drbd2165.out‚0# disk.nvme0c0n1.in‚ 31139.6796875 disk.rbd1.in‚ 0.50830078125 disk.sda.in‚ 75.2734375$ disk./.usageGB‚560.5917549133301 disk.sda3.out‚ 725.55859375 disk.drbd2165.in‚0$ disk.dm-0.out‚132730.55224609375# proc.memory.rssMB‚ 2823.25390625) disk.nvme0c0n1.out‚132730.55224609375 disk.rbd1.out‚ 36.146484375 disk.rbd0.out‚0 disk.dm-0.in‚ 31139.6796875( proc.memory.availableMB‚ 974.50390625) proc.memory.percent‚17.23177433013916 disk.dm-1.out‚ 725.55859375 disk.sda.out‚ 725.55859375 disk.dm-1.in‚ 75.2734375& disk.rbd0.in‚1.759218604441423e+13 network.sent‚ 1.5810242e+07 disk.sda3.in‚ 75.2734375 disk.rbd3.in‚0 disk.rbd3.out‚0' disk.nvme0n1.out‚132730.55224609375 network.recv‚ 3.42073e+06) disk./.usagePercent‚37.62178304004564 disk.sda1.in‚0 disk.sda1.out‚0! disk.nvme0n1.in‚ 31139.6796875 disk.sda2.in‚0 cpu‚25.566344675581107ë°”Ý"‚(:‰ ßéÒˆ›ø gpu.0.gpu‚97.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚344.528' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚68.9056 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0M1ÄÛ=Þ"‚(:² ßéÒÊõ·ø disk.sda2.out‚0# proc.memory.rssMB‚ 2823.25390625 proc.cpu.threads‚17 disk.drbd2165.out‚0$ memory_percent‚94.05388832092285' proc.memory.availableMB‚ 974.2109375 disk.rbd3.out‚0) proc.memory.percent‚17.23177433013916 disk.sda3.out‚ 725.94140625 disk.drbd2165.in‚0) disk.nvme0c0n1.out‚132741.32177734375! disk.nvme0n1.in‚ 31248.0859375 disk.sda2.in‚0 disk.sda.in‚ 75.2734375 disk.rbd0.out‚0' disk.nvme0n1.out‚132741.32177734375 cpu‚25.566546561735116 disk.dm-1.out‚ 725.94140625& disk.rbd0.in‚1.759218604441423e+13 disk.dm-0.in‚ 31248.0859375 disk.sda3.in‚ 75.2734375 disk.rbd3.in‚0# disk.nvme0c0n1.in‚ 31248.0859375 network.sent‚ 1.5815897e+07$ disk./.usageGB‚560.5981025695801$ disk.dm-0.out‚132741.32177734375 disk.sda1.out‚0 disk.rbd1.in‚ 0.50830078125 network.recv‚ 3.424334e+06) disk./.usagePercent‚37.62220903658262 disk.rbd1.out‚ 36.146484375 disk.sda.out‚ 725.94140625 disk.dm-1.in‚ 75.2734375 disk.sda1.in‚0˜btý”ß"‚(:‰ ÒŸéÒŽ¼ïö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚266.886' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.3772 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚02iÔ6à"‚(:« ÒŸéÒ—ŠÍù disk.rbd1.in‚ 0.50830078125 disk.drbd2165.in‚0 disk.rbd3.in‚0 disk.rbd3.out‚0) disk.nvme0c0n1.out‚132818.02001953125 disk.sda1.out‚0* disk./.usagePercent‚37.622681626490824 disk.rbd1.out‚ 36.146484375 disk.sda.out‚ 726.21875) proc.memory.percent‚17.23177433013916! disk.nvme0n1.in‚ 31248.5078125 disk.sda3.out‚ 726.21875# disk.nvme0c0n1.in‚ 31248.5078125 disk.sda2.in‚0 disk.sda.in‚ 75.2734375 disk.dm-1.in‚ 75.2734375( proc.memory.availableMB‚ 974.43359375 cpu‚25.566669160129873 network.sent‚ 1.5820541e+07$ disk./.usageGB‚560.6051445007324 disk.drbd2165.out‚0 disk.dm-1.out‚ 726.21875 disk.dm-0.in‚ 31248.5078125 disk.sda1.in‚0 proc.cpu.threads‚17 disk.sda3.in‚ 75.2734375 disk.rbd0.out‚0# proc.memory.rssMB‚ 2823.25390625 network.recv‚ 3.427894e+06' disk.nvme0n1.out‚132818.02001953125& disk.rbd0.in‚1.759218604441423e+13$ disk.dm-0.out‚132818.02001953125$ memory_percent‚94.05252933502197 disk.sda2.out‚0ÂR4á"/ _wandbruntime‚13756  _runtime‚13756F¸½Êâ"Ä _step‚680 ' lr-AdamW/pg1‚0.00044970471981326194 ' lr-AdamW/pg2‚0.00044970471981326194 trainer/global_step‚8299 % _timestamp‚1.7843035727267754e+09 _runtime‚13756.943953134|EÏã"É _runtime‚13756.943953134 _step‚680 ' lr-AdamW/pg1‚0.00044970471981326194 ' lr-AdamW/pg2‚0.00044970471981326194 trainer/global_step‚8299 % _timestamp‚1.7843035727267754e+09¨ÀóuS4ä"/ _wandbruntime‚13756  _runtime‚13756Àgó(å"€P G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚70754291 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚71860785 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚70754291 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚250 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚644291180040192 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7805937257984 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7804591868416 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2384053190868992 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚654360752702976 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2384053164695552 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7792582784542 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2391857756563968 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7804591868416 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2383023451344896 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 142615076 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚44819883 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚644291548131328 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2383023425916928 _step‚681 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚23530674 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚71850812 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10069607199232 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 142605097 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚70754285 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚21289464 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7791238184220 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5›ˆa/æ"…P ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2390816034129438 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2384053164695552 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2384053190868992 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7805937257984 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2391857756563968 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 _runtime‚13756.945171155 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚654361155330560 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2390814664101148 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚70754285 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚71850812 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚71860785 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚44820138 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚255 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 % _timestamp‚1.7843035727278495e+09 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚23530424 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2391859128126976 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2391859128126976 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 trainer/global_step‚8299 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚21289459 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10069572662784 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2391857756563968 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7804591868416 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2383023451344896 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 142615076 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚44819883 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚644291548131328 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2383023425916928 _step‚681 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚23530674 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚71850812 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10069607199232 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 142605097 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 L?DeviceStatsMonitor.on_train_batch_start/active.lÿ`ƒ)Ü arge_pool.freed‚70754285 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚21289464 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7791238184220 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚70754291 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚71860785 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚70754291 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚250 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚644291180040192 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7805937257984 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7804591868416 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2384053190868992 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚654360752702976 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2384053164695552 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7792582784542 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 JDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 142631774 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚21292002 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2383301333123072 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2384331165425664 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7806862300672 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7793506250900 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2392138053899776 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚71859290 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2384331165425664 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7805516903936 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚644368666447872 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2384331191599104 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 _step‚682 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 142621781 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚44825096 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚23533339 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚70762497 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23533094 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚70762491 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚71859290 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 142631774 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7806862300672 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚70762497 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚21292007 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2391094839373972 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚654439832192512 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10070797653504 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2391093469345570 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 QDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0=tT'é"N M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2384331165425664 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7806862300672 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7793506250900 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2392138053899776 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚71859290 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2384331165425664 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7805516903936 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚644368666447872 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2384331191599104 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 _step‚682 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 142621781 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚44825096 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚23533339 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚70762497 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23533094 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚70762491 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚71859290 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 142631774 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7806862300672 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚70762497 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚21292007 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2391094839373972 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚654439832192512 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10070797653504 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2391093469345570 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 QDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10070763124224 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 142631774 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚21292002 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2383301333123072 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175ªrUQ4ê"/ _wandbruntime‚13758  _runtime‚13758]Aš ë"ƒ $loss/total_loss‚7.51223448380711 trainer/global_step‚8299 _runtime‚13758.414567958 /loss/FRt2/assignment_loss‚0.554162521577324 /loss/FRt1/detection_loss‚0.5100564868449012 0loss/FBt1/detection_loss‚0.37639837668890636 epoch‚20 3loss/SRqqt2/assignment_loss‚0.48988887372855927 2loss/FBt1/assignment_loss‚0.000153253583031391 /loss/FBt2/detection_loss‚0.4780254941486186 _step‚683 2loss/SRqqt1/assignment_loss‚0.4232942234594626 3loss/FBt2/assignment_loss‚0.0025403918966564313 .loss/FRt2/detection_loss‚1.528515135884206 1loss/SRqqt2/detection_loss‚0.5910771224137411 $ _timestamp‚1.784303574197665e+09 0loss/FRt1/assignment_loss‚1.5367530629124944 0loss/SRqqt1/detection_loss‚1.021369540669209°«è;ì"ˆ _runtime‚13758.414567958 /loss/FRt2/assignment_loss‚0.554162521577324 /loss/FRt1/detection_loss‚0.5100564868449012 0loss/FBt1/detection_loss‚0.37639837668890636 epoch‚20 3loss/SRqqt2/assignment_loss‚0.48988887372855927 2loss/FBt1/assignment_loss‚0.000153253583031391 /loss/FBt2/detection_loss‚0.4780254941486186 _step‚683 2loss/SRqqt1/assignment_loss‚0.4232942234594626 3loss/FBt2/assignment_loss‚0.0025403918966564313 .loss/FRt2/detection_loss‚1.528515135884206 1loss/SRqqt2/detection_loss‚0.5910771224137411 $ _timestamp‚1.784303574197665e+09 0loss/FRt1/assignment_loss‚1.5367530629124944 0loss/SRqqt1/detection_loss‚1.021369540669209 $loss/total_loss‚7.51223448380711 trainer/global_step‚8299«•´f0”í"‚(:‰ áŸéÒŽ€ùö gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚334.874' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚66.9748 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0"JÅŒ3î"‚(:¨ áŸéÒÙÓ³ù disk.rbd3.out‚0 disk.sda2.in‚0 disk.sda1.in‚0 network.sent‚ 1.582619e+07 disk.sda.out‚ 728.3984375 proc.cpu.threads‚17) disk./.usagePercent‚37.62279119771548 disk.rbd3.in‚0 disk.rbd1.in‚ 0.50830078125 disk.rbd1.out‚ 36.208984375 disk.dm-1.out‚ 728.3984375 disk.sda2.out‚0$ memory_percent‚94.05410289764404( proc.memory.availableMB‚ 974.17578125 disk.rbd0.out‚0# disk.nvme0c0n1.in‚ 31248.7265625% disk.nvme0n1.out‚132834.279296875# proc.memory.rssMB‚ 2823.25390625$ disk./.usageGB‚560.6067771911621 disk.sda.in‚ 75.2734375 disk.dm-0.in‚ 31248.7265625" disk.dm-0.out‚132834.279296875 disk.sda3.out‚ 728.3984375 disk.drbd2165.out‚0) proc.memory.percent‚17.23177433013916 cpu‚25.566905536384713 network.recv‚ 3.43155e+06 disk.dm-1.in‚ 75.2734375 disk.sda1.out‚0 disk.sda3.in‚ 75.2734375 disk.drbd2165.in‚0& disk.rbd0.in‚1.759218604441423e+13' disk.nvme0c0n1.out‚132834.279296875! disk.nvme0n1.in‚ 31248.7265625b»}ùžï"‚(:“ ðŸéÒݨ”ø gpu.0.gpu‚94.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚283.318' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚56.66359999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Œ`z':ð"‚(:¯ ðŸéÒáÛÂø disk.sda2.in‚0 disk.dm-1.in‚ 75.2734375 disk.rbd0.out‚0 disk.dm-0.in‚ 31248.9296875 disk.sda1.out‚0$ disk./.usageGB‚560.6103477478027 disk.rbd1.out‚ 36.208984375 disk.sda.out‚ 731.40234375 disk.sda3.out‚ 731.40234375 cpu‚25.567035476067584 network.recv‚ 3.435891e+06* disk./.usagePercent‚37.623030820767525 disk.sda2.out‚0 disk.dm-1.out‚ 731.40234375# disk.dm-0.out‚132911.7001953125 disk.sda3.in‚ 75.2734375# proc.memory.rssMB‚ 2823.25390625& disk.rbd0.in‚1.759218604441423e+13 disk.rbd3.out‚0# disk.nvme0c0n1.in‚ 31248.9296875 network.sent‚ 1.5889079e+07& disk.nvme0n1.out‚132911.7001953125 disk.rbd1.in‚ 0.50830078125 disk.sda1.in‚0 disk.drbd2165.out‚0 disk.rbd3.in‚0 disk.sda.in‚ 75.2734375! disk.nvme0n1.in‚ 31248.9296875 disk.drbd2165.in‚0( disk.nvme0c0n1.out‚132911.7001953125' proc.memory.availableMB‚ 974.1171875) proc.memory.percent‚17.23177433013916 proc.cpu.threads‚17# memory_percent‚94.0544605255127ò.OçŸñ"‚(:” ÿŸéÒˆ½ëö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚41.0 gpu.0.powerWatts‚267.174' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚53.434799999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÇT<ò"‚(:± ÿŸéÒ÷ïÒù disk.dm-1.in‚ 75.2734375& disk.rbd0.in‚1.759218604441423e+13$ disk.dm-0.out‚132935.39501953125( proc.memory.availableMB‚ 974.17578125 disk.drbd2165.out‚0) disk.nvme0c0n1.out‚132935.39501953125 disk.sda2.out‚0 disk.dm-1.out‚ 733.80859375 disk.sda1.out‚0) proc.memory.percent‚17.23177433013916 cpu‚25.56722960292471 proc.cpu.threads‚17 disk.dm-0.in‚ 31311.6484375! disk.nvme0n1.in‚ 31311.6484375 disk.sda3.in‚ 75.2734375 disk.sda3.out‚ 733.80859375 disk.drbd2165.in‚0# proc.memory.rssMB‚ 2823.25390625$ disk./.usageGB‚560.6195983886719 disk.rbd1.out‚ 36.271484375 disk.rbd0.out‚0 disk.sda2.in‚0 disk.sda.out‚ 733.80859375 disk.sda1.in‚0) disk./.usagePercent‚37.62365163903806 disk.sda.in‚ 75.2734375' disk.nvme0n1.out‚132935.39501953125 network.recv‚ 3.43969e+06 disk.rbd3.in‚0 disk.rbd3.out‚0$ memory_percent‚94.05410289764404 disk.rbd1.in‚ 0.50830078125 network.sent‚ 1.5894299e+07# disk.nvme0c0n1.in‚ 31311.6484375ê4-.ó"‚(:’ Ž éÒË÷ gpu.0.gpu‚90.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚283.03' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚56.60599999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0HÔn94ô"‚(:© Ž éÒ¥¥°ø) proc.memory.percent‚17.23177433013916 cpu‚25.567306188296946 disk.sda2.out‚0 disk.rbd1.in‚ 0.50830078125 disk.sda.in‚ 75.2734375# disk.dm-0.out‚132948.1962890625 disk.rbd3.in‚0$ memory_percent‚94.05412673950195# proc.memory.rssMB‚ 2823.25390625 network.sent‚ 1.5898468e+07 disk.sda1.in‚0$ disk.nvme0c0n1.in‚31352.07421875( disk.nvme0c0n1.out‚132948.1962890625" disk.nvme0n1.in‚31352.07421875 disk.dm-1.out‚ 734.53125 disk.sda3.out‚ 734.53125 disk.rbd1.out‚ 36.271484375 proc.cpu.threads‚17* disk./.usagePercent‚37.623930687250386$ disk./.usageGB‚560.6237564086914& disk.rbd0.in‚1.759218604441423e+13 disk.rbd3.out‚0 network.recv‚ 3.442158e+06 disk.sda1.out‚0 disk.sda3.in‚ 75.2734375 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.dm-1.in‚ 75.2734375 disk.rbd0.out‚0& disk.nvme0n1.out‚132948.1962890625 disk.sda2.in‚0& proc.memory.availableMB‚ 974.171875 disk.sda.out‚ 734.53125 disk.dm-0.in‚31352.07421875å_)œžõ"‚(:“  éÒÊ…÷ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚294.864' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚58.97279999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ó ÞG6ö"‚(:«  éÒ’çú disk.dm-1.in‚ 75.2734375 disk.sda1.out‚0# proc.memory.rssMB‚ 2823.25390625) proc.memory.percent‚17.23177433013916 cpu‚25.56744354108629 disk.dm-1.out‚ 735.765625 disk.rbd0.out‚0 disk.sda1.in‚0 disk.rbd3.in‚0 proc.cpu.threads‚17* disk./.usagePercent‚37.624203847289415 disk.sda.out‚ 735.765625& disk.rbd0.in‚1.759218604441423e+13$ memory_percent‚94.05426979064941' proc.memory.availableMB‚ 974.1484375 disk.rbd1.in‚ 0.50830078125 disk.rbd1.out‚ 36.333984375 disk.drbd2165.out‚0' disk.nvme0n1.out‚132962.27978515625$ disk./.usageGB‚560.6278266906738 disk.dm-0.in‚ 31352.2890625 disk.sda3.out‚ 735.765625 disk.rbd3.out‚0) disk.nvme0c0n1.out‚132962.27978515625$ disk.dm-0.out‚132962.27978515625 disk.drbd2165.in‚0# disk.nvme0c0n1.in‚ 31352.2890625 disk.sda.in‚ 75.2734375 disk.sda3.in‚ 75.2734375! disk.nvme0n1.in‚ 31352.2890625 network.sent‚ 1.5903553e+07 network.recv‚ 3.44555e+06 disk.sda2.in‚0 disk.sda2.out‚0|Dš4÷"/ _wandbruntime‚13836  _runtime‚13836çÇêƒÈø" & lr-AdamW/pg1‚0.0004470567900336539 & lr-AdamW/pg2‚0.0004470567900336539 trainer/global_step‚8349 % _timestamp‚1.7843036522248566e+09 _runtime‚13836.442119655 _step‚684:ÅÍù"Ç _step‚684 & lr-AdamW/pg1‚0.0004470567900336539 & lr-AdamW/pg2‚0.0004470567900336539 trainer/global_step‚8349 % _timestamp‚1.7843036522248566e+09 _runtime‚13836.442119655¬ü3!)4ú"/ _wandbruntime‚13836  _runtime‚13836ãSÛ©Êû"ÿO NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/rezQ,F;'served_bytes.small_pool.peak‚ 1719664640 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2397953227374592 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 143451128 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚648165500424192 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚23667599 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚658295460495872 _step‚685 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚72276564 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚231 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2405805480563200 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2405804109000192 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚648165868515328 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚72286537 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2405805480563200 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚658295861026304 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2396917514825728 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2397953227374592 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚71164591 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚21416614 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚21416609 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2404756360151346 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚45084208 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚23667825 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7850907799040 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 400530432 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2396917540253696 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7852253188608 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚71164585 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2405804109000192 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚45084439 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7837475297328 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 _runtime‚13836.443421209 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2397953201201152 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2397953201201152 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚72276564 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚32439296 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10129992510976 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820~:)S (ü"„P GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 _runtime‚13836.443421209 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2397953201201152 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2397953201201152 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚72276564 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚32439296 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10129992510976 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2397953227374592 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 143451128 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚648165500424192 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚23667599 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚658295460495872 _step‚685 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚72276564 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚231 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2405805480563200 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2405804109000192 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚648165868515328 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚72286537 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2405805480563200 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚658295861026304 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2396917514825728 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2397953227374592 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚71164591 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚21416614 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚21416609 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2404756360151346 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚45084208 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚23667825 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7850907799040 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 400530432 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2396917540253696 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7852253188608 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚71164585 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2405804109000192 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚45084439 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7837475297328 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175­ÎP×Ú4ý"/ _wandbruntime‚13837  _runtime‚13837ûå»'þ"‰N XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2406084406336000 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2397195422031872 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚71172791 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 143457833 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _step‚686 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚72285042 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚45089703 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚32432128 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2398231228104704 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚658374552988160 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 400523264 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7853178231296 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚648243354923008 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10131198065152 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚71172791 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23670324 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚72285042 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2406083034765824 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 trainer/global_step‚8349 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚45089476 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10131165633024 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚72295029 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚658374152464896 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 143467826 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7851832834560 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚227 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2398231201931264 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚72295029 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2397195396603904 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2398231201931264 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚222 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2406084406336000 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 QDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 143467826 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 143457833 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7851832834560 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚648242986831872 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 % _timestamp‚1.7843036536137042e+09 _runtime‚13837.83112934 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2405035165395880ñ ›U ÿ"ŽN NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚71172791 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 143457833 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _step‚686 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚72285042 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚45089703 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚32432128 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2398231228104704 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚658374552988160 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 400523264 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7853178231296 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚648243354923008 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10131198065152 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚71172791 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23670324 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚72285042 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2406083034765824 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 trainer/global_step‚8349 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚45089476 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10131165633024 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚72295029 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚658374152464896 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 143467826 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7851832834560 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚227 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2398231201931264 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚72295029 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2397195396603904 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2398231201931264 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚222 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2406084406336000 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 QDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 143467826 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 143457833 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7851832834560 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚648242986831872 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 % _timestamp‚1.7843036536137042e+09 _runtime‚13837.83112934 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2405035165395880 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2406084406336000 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2397195422031872®^þO„4€#/ _wandbruntime‚13837  _runtime‚13837éþéS#€ % _timestamp‚1.7843036536989932e+09 _step‚687 2loss/SRqqt2/assignment_loss‚0.4315748395554406 1loss/FBt2/assignment_loss‚0.03807274823441481 1loss/SRqqt1/detection_loss‚0.8135087143065856 .loss/FBt2/detection_loss‚0.424389393916526 trainer/global_step‚8349 _runtime‚13837.915945473 /loss/FRt1/detection_loss‚0.5108887013361375 1loss/SRqqt2/detection_loss‚0.5673414266192871 /loss/FBt1/detection_loss‚0.7675511546378189 0loss/FRt2/assignment_loss‚0.6135828283863451 2loss/SRqqt1/assignment_loss‚0.4122319688261606 /loss/FRt2/detection_loss‚1.4643090867786084 0loss/FRt1/assignment_loss‚1.4851631196092976 1loss/FBt1/assignment_loss‚0.05543320068017783 #loss/total_loss‚7.5840471828868 epoch‚20³x™ ‚#… /loss/FBt1/detection_loss‚0.7675511546378189 0loss/FRt2/assignment_loss‚0.6135828283863451 2loss/SRqqt1/assignment_loss‚0.4122319688261606 /loss/FRt2/detection_loss‚1.4643090867786084 0loss/FRt1/assignment_loss‚1.4851631196092976 1loss/FBt1/assignment_loss‚0.05543320068017783 #loss/total_loss‚7.5840471828868 epoch‚20 % _timestamp‚1.7843036536989932e+09 _step‚687 2loss/SRqqt2/assignment_loss‚0.4315748395554406 1loss/FBt2/assignment_loss‚0.03807274823441481 1loss/SRqqt1/detection_loss‚0.8135087143065856 .loss/FBt2/detection_loss‚0.424389393916526 trainer/global_step‚8349 _runtime‚13837.915945473 /loss/FRt1/detection_loss‚0.5108887013361375 1loss/SRqqt2/detection_loss‚0.5673414266192871¯ßôáj”ƒ#‚(:‰ ¬ éÒ—ô‡÷ gpu.0.gpu‚100.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚245.915' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚49.183 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ½* <„#‚(:± ¬ éÒá²Ðù disk.rbd1.in‚ 0.50830078125& disk.rbd0.in‚1.759218604441423e+13 proc.cpu.threads‚17$ disk.nvme0c0n1.in‚31352.69921875 disk.sda1.out‚0 disk.drbd2165.in‚0) proc.memory.percent‚17.23177433013916 disk.rbd3.out‚0 disk.sda2.out‚0 disk.rbd1.out‚ 36.333984375( proc.memory.availableMB‚ 974.50390625# proc.memory.rssMB‚ 2823.25390625$ disk./.usageGB‚560.6389541625977 disk.dm-0.in‚31352.69921875$ disk.dm-0.out‚132979.41455078125 network.recv‚ 3.448676e+06 disk.sda3.out‚ 737.0234375 disk.drbd2165.out‚0 disk.rbd3.in‚0$ memory_percent‚94.05210018157959 disk.sda2.in‚0 disk.sda.in‚ 75.2734375 disk.sda.out‚ 737.0234375 network.sent‚ 1.5908674e+07) disk.nvme0c0n1.out‚132979.41455078125 disk.dm-1.in‚ 75.2734375 disk.dm-1.out‚ 737.0234375 disk.rbd0.out‚0 disk.sda3.in‚ 75.2734375( disk./.usagePercent‚37.6249506212668 disk.sda1.in‚0 cpu‚25.56762444820531" disk.nvme0n1.in‚31352.69921875' disk.nvme0n1.out‚132979.41455078125…¾÷”…#‚(:‰ » éÒ†õ¥÷ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚278.198' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.6396 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ªÇ&ò>†#‚(:³ » éÒÔƒÚú# disk.dm-0.out‚132991.8173828125 disk.sda2.in‚0 disk.rbd0.out‚0 network.sent‚ 1.5971617e+07 disk.sda3.out‚ 738.13671875 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.rbd3.in‚0( disk.nvme0c0n1.out‚132991.8173828125 disk.rbd1.in‚ 0.50830078125* disk./.usagePercent‚37.624953949364745 disk.sda1.out‚0& disk.nvme0n1.out‚132991.8173828125 disk.sda2.out‚0 disk.sda.out‚ 738.13671875& disk.rbd0.in‚1.759218604441423e+13) proc.memory.percent‚17.23177433013916 network.recv‚ 3.453595e+06 disk.sda3.in‚ 75.2734375 disk.rbd1.out‚ 36.396484375$ disk./.usageGB‚560.6390037536621" disk.nvme0n1.in‚31352.91796875' proc.memory.availableMB‚ 974.4921875 disk.sda1.in‚0 disk.rbd3.out‚0$ disk.nvme0c0n1.in‚31352.91796875 disk.dm-1.out‚ 738.13671875 cpu‚25.567475310155146 proc.cpu.threads‚17 disk.dm-0.in‚31352.91796875$ memory_percent‚94.05217170715332# proc.memory.rssMB‚ 2823.25390625 disk.sda.in‚ 75.2734375 disk.dm-1.in‚ 75.2734375År½Hž‡#‚(:“ Ê éÒ–¥†÷ gpu.0.gpu‚90.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚297.28' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚59.455999999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ëèT=ˆ#‚(:² Ê éÒÆâƒú disk.sda.out‚ 740.4921875 cpu‚25.567297107952992 network.recv‚ 3.456818e+06) disk./.usagePercent‚37.62551409384966 network.sent‚ 1.597673e+07 disk.rbd1.in‚ 0.50830078125 disk.sda.in‚ 75.2734375 disk.dm-1.out‚ 740.4921875 disk.dm-0.in‚31438.14453125 disk.drbd2165.out‚0( proc.memory.availableMB‚ 974.48828125) disk.nvme0c0n1.out‚133003.76025390625$ disk.nvme0c0n1.in‚31438.14453125 disk.sda1.in‚0 disk.sda3.out‚ 740.4921875$ memory_percent‚94.05219554901123 disk.rbd0.out‚0 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.rbd3.out‚0# proc.memory.rssMB‚ 2823.25390625$ disk./.usageGB‚560.6473503112793 disk.sda2.out‚0$ disk.dm-0.out‚133003.76025390625 disk.sda3.in‚ 75.2734375) proc.memory.percent‚17.23177433013916' disk.nvme0n1.out‚133003.76025390625 disk.sda2.in‚0 disk.dm-1.in‚ 75.2734375& disk.rbd0.in‚1.759218604441423e+13 disk.rbd3.in‚0 proc.cpu.threads‚17" disk.nvme0n1.in‚31438.14453125 disk.rbd1.out‚ 36.396484375 vLª”‰#‚(:‰ Ù éÒÝñèö gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚301.632' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚60.3264 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0›¨±?<Š#‚(:± Ù éÒïÉù& disk.rbd0.in‚1.759218604441423e+13$ disk.nvme0c0n1.in‚31438.34765625" disk.nvme0n1.in‚31438.34765625( proc.memory.availableMB‚ 974.48828125 disk.sda1.out‚0 disk.sda3.out‚ 741.19140625 disk.drbd2165.out‚0 disk.rbd1.in‚ 0.50830078125$ memory_percent‚94.05219554901123 disk.rbd1.out‚ 36.521484375# proc.memory.rssMB‚ 2823.25390625 network.sent‚ 1.5981818e+07$ disk./.usageGB‚560.6516189575195 disk.sda.in‚ 75.2734375 disk.rbd0.out‚0 disk.rbd3.out‚0 disk.sda2.in‚0) proc.memory.percent‚17.23177433013916 cpu‚25.567082509748033 disk.sda.out‚ 741.19140625 disk.drbd2165.in‚0' disk.nvme0c0n1.out‚133020.470703125% disk.nvme0n1.out‚133020.470703125 proc.cpu.threads‚17 disk.dm-1.out‚ 741.19140625 disk.dm-0.in‚31438.34765625 disk.sda1.in‚0 disk.sda3.in‚ 75.2734375 disk.sda2.out‚0* disk./.usagePercent‚37.625800566280475 disk.dm-1.in‚ 75.2734375" disk.dm-0.out‚133020.470703125 disk.rbd3.in‚0 network.recv‚ 3.460526e+06]é'Ÿ‹#‚(:” è éÒÒüëö gpu.0.gpu‚99.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚167.198' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚33.439600000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0تg4:Œ#‚(:¯ è éÒ…›ù network.recv‚ 3.463618e+06& disk.rbd0.in‚1.759218604441423e+13) disk.nvme0c0n1.out‚133028.57275390625' disk.nvme0n1.out‚133028.57275390625 disk.rbd1.out‚ 36.521484375 cpu‚25.566904289063135 disk.sda2.out‚0 disk.drbd2165.out‚0 disk.sda1.in‚0 disk.sda1.out‚0$ memory_percent‚94.05226707458496) proc.memory.percent‚17.23177433013916 disk.rbd3.out‚0 disk.rbd1.in‚ 0.50830078125 disk.dm-1.in‚ 75.2734375# proc.memory.rssMB‚ 2823.25390625 disk.dm-0.in‚31438.56640625$ disk.dm-0.out‚133028.57275390625 disk.drbd2165.in‚0' proc.memory.availableMB‚ 974.4765625 proc.cpu.threads‚17 disk.sda.out‚ 741.421875$ disk./.usageGB‚560.6583061218262 disk.sda3.out‚ 741.421875 disk.sda2.in‚0 disk.dm-1.out‚ 741.421875 network.sent‚ 1.5986896e+07) disk./.usagePercent‚37.62624934748799 disk.sda3.in‚ 75.2734375$ disk.nvme0c0n1.in‚31438.56640625" disk.nvme0n1.in‚31438.56640625 disk.rbd0.out‚0 disk.rbd3.in‚0 disk.sda.in‚ 75.2734375n7»4#/ _wandbruntime‚13915  _runtime‚13915Â{γÊŽ#Ä _runtime‚13915.482696247 _step‚688 ' lr-AdamW/pg1‚0.00044440236469843594 ' lr-AdamW/pg2‚0.00044440236469843594 trainer/global_step‚8399 % _timestamp‚1.7843037312655241e+09J•^Ï#É _step‚688 ' lr-AdamW/pg1‚0.00044440236469843594 ' lr-AdamW/pg2‚0.00044440236469843594 trainer/global_step‚8399 % _timestamp‚1.7843037312655241e+09 _runtime‚13915.482696247°´¦4#/ _wandbruntime‚13915  _runtime‚13915†&Z¢(‘#€P VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7897223729664 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7883712410436 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚652039820808192 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚45348874 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2411853237706752 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚72712289 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7885057010758 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 trainer/global_step‚8399 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚72712289 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2410811629162496 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2419751832999424 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚21543759 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 144287180 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚21543764 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7898569119232 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 400530432 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7898569119232 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚71574885 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2418696686173254 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚32439296 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚23804882 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2418695316144964 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2419751832999424 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2411853263880192 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10190380839424 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 % _timestamp‚1.7843037312666945e+09 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 144277201 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚233 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2419750461436416 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0aywC’#…P ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2411853237706752 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚72712289 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7885057010758 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 trainer/global_step‚8399 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚72712289 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2410811629162496 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2419751832999424 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚21543759 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 144287180 XKDevicÉkÍÈeStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚21543764 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7898569119232 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 400530432 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7898569119232 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚71574885 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2418696686173254 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚32439296 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚23804882 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2418695316144964 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2419751832999424 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2411853263880192 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10190380839424 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 % _timestamp‚1.7843037312666945e+09 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 144277201 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚233 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2419750461436416 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7897223729664 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7883712410436 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚652039820808192 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚45348874±¸uƒz4“#/ _wandbruntime‚13916  _runtime‚13916‡àÛ'”#ŠN TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚72710794 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2420029387202048 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚662308863380992 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚71583091 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7898148765184 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10191588597248 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 144303878 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2412131264610304 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7898148765184 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚32432128 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚224 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 % _timestamp‚1.7843037326539185e+09 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7899494161920 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚72720781 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2412131238436864 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 144303878 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚23807779 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚71583097 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚71583097 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 144293885 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚662309263904256 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚652117307215872 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10191556165120 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2412131238436864 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚45354086 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2418975491417788 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚229 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2420030758772224 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚21546307 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚45353857 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2420030758772224 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2411089485512704 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚72720781 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚71583091 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 400523264 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7885980477116 _runtime‚13916.871369258 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1-“{f'•#N N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7899494161920 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚72720781 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2412131238436864 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 144303878 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚23807779 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚71583097 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚71583097 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 144293885 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚662309263904256 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚652117307215872 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10191556165120 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2412131238436864 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚45354086 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2418975491417788 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚229 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2420030758772224 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚21546307 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚45353857 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2420030758772224 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2411089485512704 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚72720781 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚71583091 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 400523264 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7885980477116 _runtime‚13916.871369258 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚72710794 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2420029387202048 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚662308863380992 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚71583091 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7898148765184 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10191588597248 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 144303878 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2412131264610304 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7898148765184 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚32432128 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚224 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 % _timestamp‚1.7843037326539185e+09²åD¤.4–#/ _wandbruntime‚13916  _runtime‚13916ø)º- —#… 2loss/SRqqt1/assignment_loss‚0.4248798229691244 .loss/FRt2/detection_loss‚1.458859250495243 epoch‚20 _runtime‚13916.955805128 /loss/FRt1/assignment_loss‚1.452995238227091 0loss/FRt1/detection_loss‚0.47767557066670785 /loss/FBt1/detection_loss‚0.7404644908319824 /loss/FBt2/detection_loss‚0.5522085275546748 2loss/SRqqt2/assignment_loss‚0.4008925987495786 3loss/FBt2/assignment_loss‚8.234796101521827e-06 1loss/SRqqt1/detection_loss‚0.8075897492660186 %loss/total_loss‚7.608039814225133 trainer/global_step‚8399 0loss/FRt2/assignment_loss‚0.6393680229151338 2loss/FBt1/assignment_loss‚0.000148532422191483 1loss/SRqqt2/detection_loss‚0.6529497753312864 % _timestamp‚1.7843037327388518e+09 _step‚691}Ifò˜#Š /loss/FRt1/assignment_loss‚1.452995238227091 0loss/FRt1/detection_loss‚0.47767557066670785 /loss/FBt1/detection_loss‚0.7404644908319824 /loss/FBt2/detection_loss‚0.5522085275546748 2loss/SRqqt2/assignment_loss‚0.4008925987495786 3loss/FBt2/assignment_loss‚8.234796101521827e-06 1loss/SRqqt1/detection_loss‚0.8075897492660186 %loss/total_loss‚7.608039814225133 trainer/global_step‚8399 0loss/FRt2/assignment_loss‚0.6393680229151338 2loss/FBt1/assignment_loss‚0.000148532422191483 1loss/SRqqt2/detection_loss‚0.6529497753312864 % _timestamp‚1.7843037327388518e+09 _step‚691 2loss/SRqqt1/assignment_loss‚0.4248798229691244 .loss/FRt2/detection_loss‚1.458859250495243 epoch‚20 _runtime‚13916.955805128³A?”™#‚(:‰ ÷ éÒË›ø gpu.0.gpu‚52.0 gpu.0.memory‚16, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚332.173' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚66.4346 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0™€Ôñ@š#‚(:µ ÷ éÒÀŽ®ù disk.rbd1.out‚ 36.583984375 disk.drbd2165.in‚0& disk.rbd0.in‚1.759218604441423e+13 disk.rbd0.out‚0$ disk.dm-0.out‚133046.95654296875 disk.rbd1.in‚ 0.50830078125# proc.memory.rssMB‚ 2804.86328125$ disk./.usageGB‚560.6642570495605 disk.sda3.in‚ 75.2734375 disk.sda3.out‚ 742.12109375 disk.rbd3.in‚0 disk.rbd3.out‚0) disk.nvme0c0n1.out‚133046.95654296875 disk.sda2.in‚0 disk.sda2.out‚0" disk.nvme0n1.in‚31438.98828125 disk.sda.out‚ 742.12109375* proc.memory.percent‚17.119526863098145 network.sent‚ 1.5992297e+07' disk.nvme0n1.out‚133046.95654296875 disk.sda1.in‚0 disk.sda1.out‚0$ disk.nvme0c0n1.in‚31438.98828125 cpu‚25.56719005508443$ memory_percent‚93.93725395202637 disk.drbd2165.out‚0 network.recv‚ 3.466844e+06) disk./.usagePercent‚37.62664871924141 disk.dm-0.in‚31438.98828125 proc.cpu.threads‚17 disk.sda.in‚ 75.2734375 disk.dm-1.in‚ 75.2734375 disk.dm-1.out‚ 742.12109375' proc.memory.availableMB‚ 993.3203125*Åe&“›#‚(:ˆ †¡éÒÀ€ùø gpu.0.gpu‚49.0 gpu.0.memory‚15, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚335.505' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚67.101 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ˆxq™>œ#‚(:³ †¡éÒùÔ´ù# proc.memory.rssMB‚ 2804.86328125 disk.rbd3.in‚0 disk.sda2.out‚0 disk.sda.out‚ 742.3359375 disk.sda1.in‚0" disk.nvme0n1.in‚31505.77734375 disk.dm-0.in‚31505.77734375 disk.sda1.out‚0$ disk./.usageGB‚560.6688461303711 disk.rbd1.in‚ 0.50830078125 disk.dm-1.in‚ 75.2734375 disk.dm-1.out‚ 742.3359375 proc.cpu.threads‚17 disk.sda3.in‚ 75.2734375 network.sent‚ 1.6054959e+07) disk.nvme0c0n1.out‚133056.99365234375 disk.sda2.in‚0 disk.sda.in‚ 75.2734375$ memory_percent‚93.93823146820068 network.recv‚ 3.471199e+06( disk./.usagePercent‚37.6269566963051 disk.rbd1.out‚ 36.583984375$ disk.dm-0.out‚133056.99365234375 disk.sda3.out‚ 742.3359375 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.rbd3.out‚0$ disk.nvme0c0n1.in‚31505.77734375& disk.rbd0.in‚1.759218604441423e+13( proc.memory.availableMB‚ 993.16015625* proc.memory.percent‚17.119526863098145 cpu‚25.569983377471246' disk.nvme0n1.out‚133056.99365234375 disk.rbd0.out‚0ã¥c‡4#/ _wandbruntime‚13943  _runtime‚13943"Ü:\ ž#š{ 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.2488829493522644 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.42129629850387573 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.27339157462120056 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 trainer/global_step‚8399 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5905246734619141 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5624464154243469 8"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.618677020072937 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.48379626870155334 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 particle/accuracy_1_of_4‚1 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 *jet/accuracy_1_of_3‚0.9025636315345764 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9285714030265808 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.07976430654525757 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.7141793370246887 +particle/specificity‚0.9042304158210754 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8089191913604736 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9918577075004578 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 +jet/accuracy_4_of_4‚0.01908157207071781 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.5327793955802917 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2680276930332184 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 _runtime‚13943.462716279 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7926682233810425 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.3038938343524933 _step‚692 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9908525347709656 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.04040495306253433 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 :"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.46188274025917053 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.33039116859436035 +jet/accuracy_3_of_3‚0.07282537966966629 (particle/accuracy‚0.8441529870033264 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2726091146469116 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.24494239687919617 8"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.691179633140564 particle/accuracy_5_of_5‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6191112995147705 8 Purity/*FRt1bo’ED2SRqqt*FBt/FRt_purity‚0.30937299132347107 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.16134631633758545 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.4444029629230499 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.5233332514762878 , Purity/0FRt2SRqqt*FBt/FBt_purity‚0.96875 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 epoch‚20 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9994588494300842 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.04088903218507767 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9961460828781128 7 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.1560671031475067 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.24541738629341125 8"Purity/1FRt*SRqqt2FBt/event_purity‚0.722260057926178 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 8"Purity/*FRt0SRqqt*FBt/event_purity‚0.315530389547348 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.4392414093017578 , Purity/0FRt2SRqqt1FBt/FBt_purity‚0.96875 *jet/accuracy_1_of_4‚0.9663073420524597 particle/accuracy_3_of_4‚1 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9970306158065796 8"Purity/*FRt1SRqqt0FBt/event_purity‚0.260867178440094 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.5787336230278015 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.23945024609565735 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.4563721716403961 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9983477592468262 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2725961208343506 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.2679232358932495 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8379629254341125 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9964489936828613 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.5025135278701782 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.03846153989434242 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8042706251144409 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9904476404190063 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6506621241569519 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5608012080192566 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.2573586702346802 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5306048393249512 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7145792245864868 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 8"Purity/*FRt*SRqqt2FBt/event_purity‚0.874695897102356 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.5819856524467468 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.5295535326004028 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7451738119125366 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9987789392471313 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 particle/accuracy_3_of_5‚1 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8651745319366455 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6128591895103455 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.7926682233810425 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.16544388234615326 +jet/accuracy_3_of_4‚0.09781728684902191 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.26757702231407166 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.49873778223991394 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.03125450015068054 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4296875 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 *jet/accuracy_1_of_2‚0.7382515668869019 7validation_average_jet_accuracy‚0.32560381293296814 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5613114833831787 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7973166704177856 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9988564848899841 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8379629254341125 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.10915730893611908 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.5419754981994629 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.03846153989434242 jet/accuracy_2_of_5‚1 particle/accuracy_1_of_5‚1 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6130648851394653 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.5985130667686462 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 particle/accuracy_2_of_2‚1 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.4471617639064789 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6068289875984192 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4950869083404541 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.25750356912612915 particle/accuracy_2_of_3‚1 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9966978430747986 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9722222089767456 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.6238827705383301 /particle/accuracy_4_of_4‚0.9410010576248169 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.2618166506290436 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.0555555559694767 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.5434678196907043 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9361435174942017 6 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.996579647064209 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.7141793370246887 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.02866722084581852 'particle/f_score‚0.6325077414512634 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.8571428656578064 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.3401963412761688 particle/accuracy_3_of_3‚1 # _timestamp‚1.78430375924485e+09 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.4756218194961548 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.2573586702346802 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 particle/accuracy_1_of_2‚1 particle/accuracy_1_of_3‚1 particle/accuracy_2_of_4‚1 )loss/val_total_loss‚8.033823854793825 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.4195899963378906 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9992673993110657 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.5464407205581665 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.2940044701099396 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5762499570846558 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 :"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.49943292140960693 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9992673993110657 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 +jet/accuracy_2_of_3‚0.47816726565361023 +particle/sensitivity‚0.6245077848434448 +"Purity/2FRt*SRqqt2FBt/event_purity‚0.75 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.4392414093017578 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9807692170143127 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.2520720958709717 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.6013687252998352 6 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.261633038520813 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9245488047599792 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.2624058127403259 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 particle/accuracy_1_of_1‚1 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.0555555559694767 6 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.998015820980072 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 * Purity/2FRt*SRqqt2FBt/FRt_purity‚0.875 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 particle/accuracy_4_of_5‚1 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.5041474103927612 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 7 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.1581403762102127 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.29943498969078064 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.16004835069179535 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9912070035934448 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.4799553453922272 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.4833591878414154 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9824140667915344 *jet/accuracy_1_of_1‚0.2787664830684662 )jet/accuracy_2_of_4‚0.776356041431427 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.4010416865348816 particle/accuracy_2_of_5‚1 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.25598883628845215 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.20252764225006104 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 jet/accuracy_1_of_5‚1 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.19265174865722656 jet/accuracy_3_of_5‚1 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9814814329147339 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 9"Purity/2FRt*SRqqt1FBt/event_purity‚0.1691666692495346 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.16544388234615326 jet/accuracy_4_of_5‚0 jet/accuracy_5_of_5‚0 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.06107129901647568 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9814814329147339 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6927083730697632 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 +jet/accuracy_2_of_2‚0.12812016904354095 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.28503385186195374 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4466616213321686 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.11119740456342697 5"Purity/0FRt0SRqqt*FBt/event_purity‚0.995849609375 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.29974859952926636 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8969459533691406 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6259812116622925 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.26940256357192993 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.5271845459938049õÌò¥=Ÿ#Ÿ{ 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.4471617639064789 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6068289875984192 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4950869083404541 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.25750356912612915 particle/accuracy_2_of_3‚1 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9966978430747986 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9722222089767456 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.6238827705383301 /particle/accuracy_4_of_4‚0.9410010576248169 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.2618166506290436 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.0555555559694767 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.5434678196907043 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9361435174942017 6 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.996579647064209 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.7141793370246887 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.02866722084581852 'particle/f_score‚0.6325077414512634 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.8571428656578064 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.3401963412761688 particle/accuracy_3_of_3‚1 # _timestamp‚1.78430375924485e+09 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.4756218194961548 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.2573586702346802 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 particle/accuracy_1_of_2‚1 particle/accuracy_1_of_3‚1 particle/accuracy_2_of_4‚1 )loss/val_total_loss‚8.033823854793825 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.4195899963378906 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9992673993110657 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.5464407205581665 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.2940044701099396 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5762499570846558 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 :"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.49943292140960693 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9992673993110657 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 +jet/accuracy_2_of_3‚0.47816726565361023 +particle/sensitivity‚0.6245077848434448 +"Purity/2FRt*SRqqt2FBt/event_purity‚0.75 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.4392414093017578 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9807692170143127 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.2520720958709717 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.6013687252998352 6 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.261633038520813 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9245488047599792 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.2624058127403259 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 particle/accuracy_1_of_1‚1 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.0555555559694767 6 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.998015820980072 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 * Purity/2FRt*SRqqt2FBt/FRt_purity‚0.875 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 particle/accuracy_4_of_5‚1 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.5041474103927612 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 7 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.1581403762102127 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.29943498969078064 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.16004835069179535 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9912070035934448 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.4799553453922272 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.4833591878414154 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9824140667915344 *jet/accuracy_1_of_1‚0.2787664830684662 )jet/accuracy_2_of_4‚0.776356041431427 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.4010416865348816 particle/accuracy_2_of_5‚1 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.25598883628845215 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.20252764225006104 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 jet/accuracy_1_of_5‚1 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.19265174865722656 jet/accuracy_3_of_5‚1 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9814814329147339 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 9"Purity/2FRt*SRqqt1FBt/event_purity‚0.1691666692495346 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.16544388234615326 jet/accuracy_4_of_5‚0 jet/accuracy_5_of_5‚0 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.06107129901647568 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9814814329147339 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6927083730697632 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 +jet/accuracy_2_of_2‚0.12812016904354095 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.28503385186195374 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4466616213321686 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.11119740456342697 5"Purity/0FRt0SRqqt*FBt/event_purity‚0.995849609375 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.29974859952926636 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8969459533691406 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6259812116622925 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.26940256357192993 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.5271845459938049 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.2488829493522644 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.42129629850387573 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.27339157462120056 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 trainer/global_step‚8399 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5905246734619141 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5624464154243469 8"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.618677020072937 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.48379626870155334 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 particle/accuracy_1_of_4‚1 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 *jet/accuracy_1_of_3‚0.9025636315345764 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9285714030265808 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.07976430654525757 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.7141793370246887 +particle/specificity‚0.9042304158210754 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8089191913604736 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9918577075004578 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 +jet/accuracy_4_of_4‚0.01908157207071781 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.5327793955802917 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2680276930332184 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 _runtime‚13943.462716279 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7926682233810425 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.3038938343524933 _step‚692 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9908525347709656 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.04040495306253433 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 :"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.46188274025917053 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.33039116859436035 +jet/accuracy_3_of_3‚0.07282537966966629 (particle/accuracy‚0.8441529870033264 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2726091146469116 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.24494239687919617 8"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.691179633140564 particle/accuracy_5_of_5‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6191112995147705 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.30937299132347107 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.16134631633758545 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.4444029629230499 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.5233332514762878 , Purity/0FRt2SRqqt*FBt/FBt_purity‚0.96875 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 epoch‚20 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9994588494300842 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.04088903218507767 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9961460828781128 7 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.1560671031475067 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.24541738629341125 8"Purity/1FRt*SRqqt2FBt/event_purity‚0.722260057926178 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 8"Purity/*FRt0SRqqt*FBt/event_purity‚0.315530389547348 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.4392414093017578 , Purity/0FRt2SRqqt1FBt/FBt_purity‚0.96875 *jet/accuracy_1_of_4‚0.9663073420524597 particle/accuracy_3_of_4‚1 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9970306158065796 8"Purity/*FRt1SRqqt0FBt/event_purity‚0.260867178440094 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.5787336230278015 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.23945024609565735 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.4563721716403961 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9983477592468262 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2725961208343506 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.2679232358932495 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8379629254341125 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9964489936828613 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.5025135278701782 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.03846153989434242 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8042706251144409 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9904476404190063 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6506621241569519 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5608012080192566 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.2573586702346802 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5306048393249512 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7145792245864868 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 8"Purity/*FRt*SRqqt2FBt/event_purity‚0.874695897102356 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.5819856524467468 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.5295535326004028 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7451738119125366 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9987789392471313 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 particle/accuracy_3_of_5‚1 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8651745319366455 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6128591895103455 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.7926682233810425 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.16544388234615326 +jet/accuracy_3_of_4‚0.09781728684902191 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.26757702231407166 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.49873778223991394 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.03125450015068054 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4296875 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 *jet/accuracy_1_of_2‚0.7382515668869019 7validation_average_jet_accuracy‚0.32560381293296814 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5613114833831787 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7973166704177856 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9988564848899841 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8379629254341125 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.10915730893611908 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.5419754981994629 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.03846153989434242 jet/accuracy_2_of_5‚1 particle/accuracy_1_of_5‚1 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6130648851394653 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.5985130667686462 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 particle/accuracy_2_of_2‚1´ƒ°’«ž #‚(:“ •¡éÒ¾–Ðö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚272.83' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚54.565999999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¹Uü7¡#‚(:¬ •¡éÒ¦›Ùú network.recv‚ 3.474741e+06) disk./.usagePercent‚37.62741213370852& disk.rbd0.in‚1.759218604441423e+13 disk.sda1.in‚0$ disk.nvme0c0n1.in‚31505.99609375 disk.sda.out‚ 742.6796875 disk.dm-1.out‚ 742.6796875" disk.dm-0.out‚133068.787109375 disk.sda1.out‚0 disk.rbd3.out‚0 disk.sda2.in‚0 disk.sda.in‚ 75.2734375* proc.memory.percent‚17.119526863098145$ disk./.usageGB‚560.6756324768066 disk.rbd3.in‚0" disk.nvme0n1.in‚31505.99609375# proc.memory.rssMB‚ 2804.86328125 disk.drbd2165.out‚0 network.sent‚ 1.6060034e+07 disk.rbd0.out‚0 disk.sda3.in‚ 75.2734375 disk.sda2.out‚0 disk.rbd1.out‚ 36.646484375$ memory_percent‚93.94659996032715 cpu‚25.57160294790328 disk.dm-0.in‚31505.99609375 disk.sda3.out‚ 742.6796875 disk.dm-1.in‚ 75.2734375' proc.memory.availableMB‚ 991.7890625 disk.drbd2165.in‚0' disk.nvme0c0n1.out‚133068.787109375% disk.nvme0n1.out‚133068.787109375 disk.rbd1.in‚ 0.50830078125 proc.cpu.threads‚17ÕL„~”¢#‚(:‰ ¤¡éÒÌÕö gpu.0.gpu‚92.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚274.718' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.9436 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0#-ž?£#‚(:´ ¤¡éÒ™ôåù* proc.memory.percent‚17.119526863098145" disk.nvme0n1.in‚31507.28515625 disk.sda.in‚ 75.2734375& disk.rbd0.in‚1.759218604441423e+13 disk.dm-0.in‚31507.28515625 network.recv‚ 3.479004e+06' disk.nvme0n1.out‚133083.12060546875 disk.dm-1.in‚ 75.2734375* disk./.usagePercent‚37.627900340076316$ disk./.usageGB‚560.6829071044922) disk.nvme0c0n1.out‚133083.12060546875 disk.sda1.out‚0 disk.sda3.in‚ 75.2734375 disk.sda3.out‚ 742.82421875 disk.drbd2165.out‚0 cpu‚25.57135088974873 disk.rbd3.out‚0 disk.rbd1.out‚ 36.646484375 disk.rbd0.out‚0 network.sent‚ 1.6118267e+07 disk.rbd3.in‚0$ disk.nvme0c0n1.in‚31507.28515625 disk.sda2.in‚0 disk.sda.out‚ 742.82421875$ memory_percent‚93.94702911376953 disk.sda2.out‚0 disk.rbd1.in‚ 0.50830078125$ disk.dm-0.out‚133083.12060546875 disk.sda1.in‚0 disk.drbd2165.in‚0% proc.memory.availableMB‚ 991.71875 proc.cpu.threads‚17 disk.dm-1.out‚ 742.82421875# proc.memory.rssMB‚ 2804.863281251´£ž¤#‚(:“ ³¡éÒìžõø gpu.0.gpu‚92.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚335.114' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚67.02279999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ðð M™¥#‚(:¬ ³¡éÒùËú& disk.rbd0.in‚1.759218604441423e+13 disk.sda1.in‚0 disk.sda3.out‚ 743.06640625 disk.drbd2165.out‚0' proc.memory.availableMB‚ 991.5234375 proc.cpu.threads‚17 disk.sda1.out‚0 network.recv‚ 3.482397e+06 disk.sda.out‚ 743.06640625$ memory_percent‚93.94822120666504 disk.dm-1.in‚ 75.2734375" disk.nvme0n1.in‚31507.48828125 cpu‚25.571182187359895$ disk.nvme0c0n1.in‚31507.48828125$ disk.nvme0n1.out‚133093.60546875 disk.sda2.in‚0* proc.memory.percent‚17.119526863098145 disk.sda2.out‚0 disk.sda.in‚ 75.2734375 disk.dm-1.out‚ 743.06640625 disk.rbd0.out‚0n×óž! disk.dm-0.out‚133093.60546875 disk.drbd2165.in‚0 network.sent‚ 1.6123329e+07# disk./.usageGB‚560.686954498291& disk.nvme0c0n1.out‚133093.60546875 disk.rbd1.out‚ 36.708984375) disk./.usagePercent‚37.62817196407014 disk.dm-0.in‚31507.48828125 disk.sda3.in‚ 75.2734375 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.rbd1.in‚ 0.50830078125# proc.memory.rssMB‚ 2804.86328125÷L•óŸ¦#‚(:” ¡éÒŒúö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚262.342' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚52.468399999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0X± <§#‚(:± ¡éÒÛšÃù) disk.nvme0c0n1.out‚133101.62939453125 disk.sda.out‚ 743.23828125 disk.sda1.in‚0 disk.sda3.in‚ 75.2734375 disk.drbd2165.in‚0 disk.sda2.in‚0 disk.sda1.out‚0 network.sent‚ 1.612787e+07 disk.drbd2165.out‚0* proc.memory.percent‚17.119526863098145 disk.rbd1.in‚ 0.50830078125 disk.dm-1.in‚ 75.2734375 disk.dm-0.in‚ 31631.703125 disk.sda3.out‚ 743.23828125 disk.rbd3.in‚0 network.recv‚ 3.485532e+06* disk./.usagePercent‚37.628677322942735$ disk./.usageGB‚560.6944847106934' disk.nvme0n1.out‚133101.62939453125 disk.rbd1.out‚ 36.708984375 disk.rbd3.out‚0 cpu‚25.570949552141958" disk.nvme0c0n1.in‚ 31631.703125 disk.sda2.out‚0& disk.rbd0.in‚1.759218604441423e+13 disk.nvme0n1.in‚ 31631.703125 disk.sda.in‚ 75.2734375 disk.rbd0.out‚0$ memory_percent‚93.94891262054443( proc.memory.availableMB‚ 991.41015625 proc.cpu.threads‚17 disk.dm-1.out‚ 743.23828125$ disk.dm-0.out‚133101.62939453125# proc.memory.rssMB‚ 2804.86328125Ñãwž¨#‚(:“ Ñ¡éÒÚ½‡ø gpu.0.gpu‚27.0 gpu.0.memory‚9, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚37.0 gpu.0.powerWatts‚109.997' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.999399999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0?áôö©#‚(:” Ñ¡éÒöÑœù disk.sda3.out‚ 743.84375 disk.dm-1.in‚ 75.2734375 disk.sda1.out‚0# proc.memory.rssMB‚ 2804.86328125# disk./.usageGB‚560.692138671875 disk.rbd3.in‚0' disk.nvme0c0n1.out‚133114.423828125 disk.sda.in‚ 75.2734375 disk.nvme0c0n1.in‚ 31632.125 disk.sda2.out‚0$ memory_percent‚93.94893646240234 network.recv‚ 3.489159e+06% disk.nvme0n1.out‚133114.423828125 disk.dm-1.out‚ 743.84375 disk.nvme0n1.in‚ 31632.125 disk.sda2.in‚0 disk.sda.out‚ 743.84375 disk.dm-0.in‚ 31632.125" disk.dm-0.out‚133114.423828125% proc.memory.availableMB‚ 991.40625 disk.rbd1.out‚ 36.771484375 disk.rbd0.out‚0 disk.sda1.in‚0 proc.cpu.threads‚17) disk./.usagePercent‚37.62851987830918 disk.drbd2165.out‚0 disk.rbd3.out‚0 disk.rbd1.in‚ 0.50830078125& disk.rbd0.in‚1.759218604441423e+13 cpu‚25.57076795598103 network.sent‚ 1.6133529e+07 disk.sda3.in‚ 75.2734375 disk.drbd2165.in‚0* proc.memory.percent‚17.1195268630981458Ä<\4ª#/ _wandbruntime‚14020  _runtime‚14020=Å·²Æ«#À # _timestamp‚1.78430383676442e+09 _runtime‚14020.981596192 _step‚693 & lr-AdamW/pg1‚0.0004417416142963871 & lr-AdamW/pg2‚0.0004417416142963871 trainer/global_step‚8449§ÉeÄˬ#Å trainer/global_step‚8449 # _timestamp‚1.78430383676442e+09 _runtime‚14020.981596192 _step‚693 & lr-AdamW/pg1‚0.0004417416142963871 & lr-AdamW/pg2‚0.0004417416142963871µG4­#/ _wandbruntime‚14020  _runtime‚140205wœH(®#ÿO P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2428531601575936 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚72117191 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2429581255647232 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7950727661568 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚72117191 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚73197023 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚73197023 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚21679827 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2437530637919232 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 145314220 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚72117197 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 _runtime‚14020.983013698 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2429581281820672 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7937123255303 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚655962253438976 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2437532009482240 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7950727661568 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 trainer/global_step‚8449 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2429581255647232 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 _step‚694 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚234 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚666220694451200 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2436467380230917 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7949382272000 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7949382272000 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚73187050 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10258072921088 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2436468750259207 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2437532009482240 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚21679832 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚23978142 $ _timestamp‚1.784303836765678e+09 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2428531627003904 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0ì~¦m (¯#„P KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 _runtime‚14020.983013698 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2429581281820672 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7937123255303 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚655962253438976 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2437532009482240 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7950727661568 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 trainer/global_step‚8449 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2429581255647232 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 _step‚694 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚234 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚666220694451200 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2436467380230917 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7949382272000 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7949382272000 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚73187050 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10258072921088 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2436468750259207 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2437532009482240 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚21679832 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚23978142 $ _timestamp‚1.784303836765678e+09 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2428531627003904 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2428531601575936 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚72117191 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2429581255647232 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7950727661568 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚72117191 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚73197023 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚73197023 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚21679827 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2437530637919232 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 145314220 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚72117197 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979¶(ÇLë4°#/ _wandbruntime‚14022  _runtime‚14022ÿ 3 ±#ŠN T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2437809563684864 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2437810935255040 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10259285979136 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2429859282550784 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚21682375 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2436746185475339 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 145330918 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚656039739846656 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚73205515 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚73195528 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚72125397 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚217 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2428809508782080 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚72125403 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚73195528 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚45663223 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚656040107937792 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚72125403 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23980853 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 _runtime‚14022.369066335 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7950307307520 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 _step‚695 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚72125397 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7951652704256 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 145330918 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 145320925 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚23981070 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 145320925 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚666299393916928 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚32432128 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚45663445 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocatm½J3Ýions.peak‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2429859282550784 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚72125397 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7951652704256 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 145330918 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 145320925 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚23981070 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 145320925 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚666299393916928 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚32432128 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚45663445 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2429859282550784 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10259285979136 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2429859282550784 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚21682375 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2436746185475339 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 145330918 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚656039739846656 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚73205515 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚73195528 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚72125397 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚217 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2428809508782080 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚72125403 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚73195528 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚45663223 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚656040107937792 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚72125403 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23980853 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 _runtime‚14022.369066335 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7950307307520 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 _step‚695 FDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚45923159 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚659836941914112 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2443481292152832 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚45923396 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _step‚698 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10318767908864 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2442425715912704 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2442425690484736 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚670155709822976 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 400530432 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚32439296 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2443481318326272 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2450409076281115 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚73622775 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚24116414 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 trainer/global_step‚8499 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚21806977 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2451476990355456 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚72527497 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2451478361918464 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2450407706252825 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚72527497 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚237 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚670155309292544 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7997043592192 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚232 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2451476990355456 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚45923159 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚659836941914112 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2443481292152832 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚45923396 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _step‚698 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10318767908864 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2442425715912704 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2442425690484736 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚670155709822976 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 400530432 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚32439296 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2443481318326272 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2450409076281115 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚73622775 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚24116414 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 trainer/global_step‚8499 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚21806977 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2451476990355456 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚72527497 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2451478361918464 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2450407706252825 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚72527497 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚237 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚670155309292544 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7997043592192 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚232 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2451476990355456 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7996623238144 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚73631267 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 400523264 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7997968634880 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚72535703 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚670234005939200 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7997968634880 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2451757287691264 QDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2443759292882944 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10319978140672 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚32432128 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7996623238144 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 146166970 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚225 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚72535703 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚220 _runtime‚14101.108280585 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2442703597690880 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚21809520 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚45928640 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚72535697 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚659914428321792 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚45928415 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚670234406462464 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 146156977 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2451755916121088 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚73621280 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚21809525 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10319945708544 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚659914060230656 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _step‚699 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7982939234335 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚72535697 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175±¥¢c'Ê#N H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚45928415 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚670234406462464 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 146156977 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2451755916121088 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚73621280 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚21809525 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10319945708544 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚659914060230656 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _step‚699 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7982939234335 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚72535697 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7996623238144 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚73631267 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 400523264 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7997968634880 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚72535703 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚670234005939200 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7997968634880 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2451757287691264 QDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2443759292882944 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10319978140672 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚32432128 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7996623238144 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 146166970 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚225 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚72535703 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚220 _runtime‚14101.108280585 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2442703597690880 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚21809520 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚45928640 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚72535697 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚659914428321792 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚74048527 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚32439296 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚46188198 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚72937791 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚14178.654939807 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚21934127 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2457381328658432 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8043359522816 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2465424714354688 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8028252881197 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8042014133248 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2457381328658432 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚24253844 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2465423342791680 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚663710894206976 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 % _timestamp‚1.7843039944340777e+09 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 146986324 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8029597481519 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚227 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 trainer/global_step‚8549 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚74038554 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚663711262298112 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8042014133248 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10379233762304 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 400530432 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2465423342791680 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2457381354831872 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚74048527 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚32439296 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚46188198 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚72937791 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚14178.654939807 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚21934127 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2457381328658432 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8043359522816 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2465424714354688 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8028252881197 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8042014133248 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2457381328658432 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚24253844 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2465423342791680 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚663710894206976 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 % _timestamp‚1.7843039944340777e+09 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 146986324 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8029597481519 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚227 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 trainer/global_step‚8549 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚74038554 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚663711262298112 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8042014133248 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10379233762304 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 400530432 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2465423342791680 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2457381354831872 JDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7843039958192234e+09 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10380417746944 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 146993029 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _step‚703 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2457659355561984 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚674168798361600 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚24256563 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 trainer/global_step‚8549 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚21936675 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚74047032 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8042939168768 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 802917634Ïåu:7443 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2465703640127488 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚46193233 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2456597661171712 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚74057019 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚218 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10380448081920 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 147003022 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2456597686599680 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8030520947877 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 _runtime‚14180.03659143 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚74057019 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚223 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2456597661171712 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚74057019 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚218 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10380448081920 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 147003022 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2456597686599680 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8030520947877 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 _runtime‚14180.03659143 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚74057019 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚223 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7843039958192234e+09 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10380417746944 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 146993029 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _step‚703 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2457659355561984 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚674168798361600 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚24256563 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 trainer/global_step‚8549 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚21936675 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚74047032 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8042939168768 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8029176347443 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2465703640127488 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚46193233 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 FDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚73348091 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚667585214590976 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10439553217536 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2470213868302336 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2478288358296641 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8089675453440 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2471281365164032 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚74474279 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2479371066790912 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚46453141 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚24392112 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2471281365164032 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚73348097 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚678024733272064 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚74464306 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _step‚706 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2479371066790912 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7843040733248801e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚73348097 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚247936¢ˆÕ 9695227904 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚73348091 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2479369695227904 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2471281391337472 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8089675453440 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚24391864 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10439518681088 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2470213893730304 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8074489994305 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚46453394 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚253 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚22061277 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚19958:ë½ (ó#…P G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10439553217536 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2470213868302336 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2478288358296641 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8089675453440 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2471281365164032 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚74474279 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2479371066790912 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚46453141 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚24392112 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2471281365164032 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚73348097 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚678024733272064 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚74464306 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _step‚706 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2479371066790912 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7843040733248801e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚73348097 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2479369695227904 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚73348091 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2479369695227904 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2471281391337472 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8089675453440 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚24391864 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10439518681088 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2470213893730304 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8074489994305 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚46453394 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚253 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚22061277 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚74474279 trainer/global_step‚8599 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2471281391337472 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 147822376 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 147812397 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚74464306 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8075834594627 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8088330063872 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚678025135899648 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚14257.542241022 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚22061282 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚73348091 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚667585214590976ÂLìÇ4ô#/ _wandbruntime‚14258  _runtime‚14258ÛP'õ#ŠN RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10440711651328 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 _step‚707 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8075413460551 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2478567163541063 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚24394604 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8076758060985 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚74472784 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2479649992563712 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 147829081 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚14258.928369415 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8089255099392 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚46458424 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24394845 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚73356297 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2479648620993536 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚667663069089792 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2478568533569465 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2470491750080512 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2479649992563712 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚74482771 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚73356303 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚22063825 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8090600496128 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 147839074 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚22063820 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8090600496128 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚73356303 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚678103412649984 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0^Æ{&ö#N E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚22063820 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8090600496128 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚73356303 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚678103412649984 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10440711651328 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 _step‚707 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8075413460551 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2478567163541063 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚24394604 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8076758060985 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚74472784 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2479649992563712 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 147829081 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚14258.928369415 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8089255099392 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚46458424 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24394845 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚73356297 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2479648620993536 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚667663069089792 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2478568533569465 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2470491750080512 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2479649992563712 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚74482771 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚73356303 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚22063825 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocateJMø1šd‚ 8090600496128 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 147839074 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0Ããíª4÷#/ _wandbruntime‚14259  _runtime‚14259¿žãð ø#† 1loss/SRqqt1/detection_loss‚0.6889528132022851 /loss/FBt1/detection_loss‚0.7960366601037094 trainer/global_step‚8599 _step‚708 4loss/FBt1/assignment_loss‚1.2840169244965338e-05 2loss/FBt2/assignment_loss‚0.010409756243612494 0loss/SRqqt2/detection_loss‚0.650247845832564 /loss/FBt2/detection_loss‚0.5456515023335898 %loss/total_loss‚7.464521534671961 $ _timestamp‚1.784304074796926e+09 0loss/FRt1/assignment_loss‚1.4414592933313273 2loss/SRqqt2/assignment_loss‚0.4040902339042349 /loss/FRt1/detection_loss‚0.5286144295175894 epoch‚21 _runtime‚14259.013878954 0loss/FRt2/assignment_loss‚0.5364135388553757 3loss/SRqqt1/assignment_loss‚0.44019261989463737 /loss/FRt2/detection_loss‚1.4224400012837914iä©ù#‹ 1loss/SRqqt1/detection_loss‚0.6889528132022851 /loss/FBt1/detection_loss‚0.7960366601037094 trainer/global_step‚8599 _step‚708 4loss/FBt1/assignment_loss‚1.2840169244965338e-05 2loss/FBt2/assignment_loss‚0.010409756243612494 0loss/SRqqt2/detection_loss‚0.650247845832564 /loss/FBt2/detection_loss‚0.5456515023335898 %loss/total_loss‚7.464521534671961 $ _timestamp‚1.784304074796926e+09 0loss/FRt1/assignment_loss‚1.4414592933313273 2loss/SRqqt2/assignment_loss‚0.4040902339042349 /loss/FRt1/detection_loss‚0.5286144295175894 epoch‚21 _runtime‚14259.013878954 0loss/FRt2/assignment_loss‚0.5364135388553757 3loss/SRqqt1/assignment_loss‚0.44019261989463737 /loss/FRt2/detection_loss‚1.4224400012837914ÄF”6ÁŸú#‚(:” УéÒ€»òö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚284.579' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚56.915800000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0îø–?û#‚(:´ УéÒâüø disk.dm-1.in‚ 75.2734375& disk.rbd0.in‚1.759218604441423e+13 disk.rbd0.out‚0 disk.rbd3.out‚0 disk.sda1.in‚0 disk.drbd2165.in‚0$ disk.nvme0c0n1.in‚31961.57421875 disk.rbd1.in‚ 0.50830078125 disk.rbd1.out‚ 37.396484375$ disk.dm-0.out‚133312.27490234375 disk.dm-1.out‚ 747.73828125" disk.nvme0n1.in‚31961.57421875' disk.nvme0n1.out‚133312.27490234375 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda1.out‚0) disk.nvme0c0n1.out‚133312.27490234375 disk.sda.out‚ 747.73828125# proc.memory.rssMB‚ 2804.86328125* proc.memory.percent‚17.119526863098145 cpu‚25.56738879270889 network.sent‚ 1.6430111e+07) disk./.usagePercent‚37.63757409677265 disk.sda.in‚ 75.2734375' proc.memory.availableMB‚ 991.6953125 disk.drbd2165.out‚0 disk.sda3.in‚ 75.2734375 disk.rbd3.in‚0 proc.cpu.threads‚17 disk.dm-0.in‚31961.57421875$ memory_percent‚93.94717216491699 disk.sda3.out‚ 747.73828125 network.recv‚ 3.55113e+06$ disk./.usageGB‚560.8270530700684T<ü#‚(:± ߣéÒ‡òÉù) disk.nvme0c0n1.out‚133326.39501953125 disk.sda3.out‚ 747.984375 proc.cpu.threads‚17 disk.rbd3.in‚0' disk.nvme0n1.out‚133326.39501953125 disk.dm-1.out‚ 747.984375 disk.rbd0.out‚0$ disk.dm-0.out‚133326.39501953125( proc.memory.availableMB‚ 991.62109375 disk.sda2.out‚0$ memory_percent‚93.94762516021729# proc.memory.rssMB‚ 2804.86328125* proc.memory.percent‚17.119526863098145" disk.nvme0n1.in‚31961.99609375 disk.rbd1.in‚ 0.50830078125 disk.sda.out‚ 747.984375 disk.dm-1.in‚ 75.2734375 disk.sda3.in‚ 75.2734375 network.recv‚ 3.555865e+06 disk.drbd2165.in‚0 network.sent‚ 1.6493321e+07* disk./.usagePercent‚37.637970908450725$ disk.nvme0c0n1.in‚31961.99609375 disk.sda2.in‚0 disk.rbd1.out‚ 37.458984375 disk.dm-0.in‚31961.99609375 disk.sda1.in‚0 cpu‚25.56721104418497 disk.sda1.out‚0$ disk./.usageGB‚560.8329658508301 disk.sda.in‚ 75.2734375& disk.rbd0.in‚1.759218604441423e+13 disk.drbd2165.out‚0 disk.rbd3.out‚0Qà6”ý#‚(:‰ ߣéÒ»üΉ gpu.0.gpu‚57.0 gpu.0.memory‚18, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚333.378' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚66.6756 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ìà&†’þ#‚(:‡ î£éÒ²ŠÝö gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚276.07' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.214 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0öê>/ÿ#‚(:¤ î£éÒ¬îø network.sent‚ 1.6497863e+07# disk.nvme0c0n1.in‚ 32020.8984375 disk.sda2.in‚0 disk.rbd1.in‚ 0.50830078125 disk.sda1.in‚0# disk.dm-0.out‚133336.8974609375( proc.memory.availableMB‚ 991.62890625 network.recv‚ 3.55894e+06 disk.sda.in‚ 75.2734375) disk./.usagePercent‚37.63828758977058 disk.rbd3.out‚0 disk.sda.out‚748.1875$ disk./.usageGB‚560.8376846313477 disk.sda3.in‚ 75.2734375 disk.sda3.out‚748.1875 disk.rbd0.out‚0# proc.memory.rssMB‚ 2804.86328125 disk.rbd1.out‚ 37.458984375& disk.rbd0.in‚1.759218604441423e+13$ memory_percent‚93.94757747650146* proc.memory.percent‚17.119526863098145 cpu‚25.567003581075685 disk.rbd3.in‚0 disk.sda2.out‚0 disk.dm-1.out‚748.1875 disk.sda1.out‚0 disk.dm-0.in‚ 32020.8984375 disk.drbd2165.in‚0 disk.drbd2165.out‚0( disk.nvme0c0n1.out‚133336.8974609375! disk.nvme0n1.in‚ 32020.8984375& disk.nvme0n1.out‚133336.8974609375 disk.dm-1.in‚ 75.2734375 proc.cpu.threads‚17u’¾™•€$‚(:Š ý£éÒ¹±Š÷ gpu.0.gpu‚100.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚346.522' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚69.3044 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0“ö:$‚(:¯ ý£éÒׂÉù disk.rbd1.in‚ 0.50830078125 disk.dm-0.in‚ 32021.3203125 disk.sda1.out‚0 disk.sda2.in‚0 disk.dm-1.in‚ 75.2734375 disk.sda1.in‚0 disk.sda3.in‚ 75.2734375# disk.nvme0c0n1.in‚ 32021.3203125 disk.sda.out‚ 748.89453125) disk./.usagePercent‚37.63891276016919 disk.drbd2165.in‚0 disk.rbd3.in‚0! disk.nvme0n1.in‚ 32021.3203125& disk.nvme0n1.out‚133344.5556640625 disk.sda.in‚ 75.2734375$ memory_percent‚93.94917488098145# proc.memory.rssMB‚ 2804.86328125 network.recv‚ 3.562367e+06$ disk./.usageGB‚560.8470001220703 disk.drbd2165.out‚0 disk.dm-1.out‚ 748.89453125 disk.rbd0.out‚0# disk.dm-0.out‚133344.5556640625 disk.sda3.out‚ 748.89453125 proc.cpu.threads‚17 disk.rbd3.out‚0( disk.nvme0c0n1.out‚133344.5556640625 disk.sda2.out‚0 disk.rbd1.out‚ 37.521484375' proc.memory.availableMB‚ 991.3671875* proc.memory.percent‚17.119526863098145& disk.rbd0.in‚1.759218604441423e+13 cpu‚25.56682613681036 network.sent‚ 1.6503563e+07¼lÞ”‚$‚(:‰ Œ¤éÒ¹ô…÷ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚251.566' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚50.3132 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0æ"<¡?ƒ$‚(:´ Œ¤éÒ¢íâù disk.sda.in‚ 75.2734375 disk.rbd3.in‚0' disk.nvme0n1.out‚133358.83740234375 disk.sda2.in‚0 disk.sda2.out‚0& disk.rbd0.in‚1.759218604441423e+13 disk.drbd2165.out‚0) disk.nvme0c0n1.out‚133358.83740234375 disk.rbd1.in‚ 0.50830078125 disk.rbd0.out‚0 network.sent‚ 1.6508764e+07 disk.dm-1.out‚ 749.11328125 proc.cpu.threads‚17* disk./.usagePercent‚37.639197952562334 disk.sda1.out‚0! disk.nvme0n1.in‚ 32021.5234375$ memory_percent‚93.94688606262207 cpu‚25.566598719986633 disk.sda.out‚ 749.11328125 disk.dm-0.in‚ 32021.5234375 disk.drbd2165.in‚0 disk.rbd3.out‚0 network.recv‚ 3.566216e+06 disk.dm-1.in‚ 75.2734375$ disk.dm-0.out‚133358.83740234375 disk.sda3.in‚ 75.2734375 disk.rbd1.out‚ 37.521484375' proc.memory.availableMB‚ 991.7421875* proc.memory.percent‚17.119526863098145$ disk./.usageGB‚560.8512496948242 disk.sda1.in‚0 disk.sda3.out‚ 749.11328125# disk.nvme0c0n1.in‚ 32021.5234375# proc.memory.rssMB‚ 2804.86328125ùT84„$/ _wandbruntime‚14336  _runtime‚14336^ÁÖ*È…$ _runtime‚14336.288035656 _step‚709 & lr-AdamW/pg1‚0.0004310387857834388 & lr-AdamW/pg2‚0.0004310387857834388 trainer/global_step‚8649 % _timestamp‚1.7843041520708551e+09òQ03͆$Ç _runtime‚14336.288035656 _step‚709 & lr-AdamW/pg1‚0.0004310387857834388 & lr-AdamW/pg2‚0.0004310387857834388 trainer/global_step‚8649 % _timestamp‚1.7843041520708551e+09Å F ‹4‡$/ _wandbruntime‚14336  _runtime‚14336j§Á(ˆ$€P E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚671459534974976 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚74890058 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2484107957211136 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2485181401669632 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚24530288 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 148658428 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚22188432 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2492230054346839 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8134645994496 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚671459903066112 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚250 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚73758397 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚74900031 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚46718465 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2493316047664128 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8134645994496 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2485181427843072 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8120727107413 trainer/global_step‚8649 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2485181401669632 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚681959089475584 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2484107982639104 _runtime‚14336.289288994 _step‚710 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2493316047664128 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2492228684318549 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2493317419227136 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10499589037056 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚73758391 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚73758391 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚74890058 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚46718720 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2485181427843072 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚73758397 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚22188427 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8135991384064 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚74890058 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚46718720 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2485181427843072 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚73758397 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚22188427 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8135991384064 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚74890058 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2484107957211136 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2485181401669632 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚24530288 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 148658428 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚22188432 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2492230054346839 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8134645994496 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚671459903066112 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚250 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚73758397 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚74900031 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚46718465 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2493316047664128 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8134645994496 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2485181427843072 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8120727107413 trainer/global_step‚8649 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627Æ…G²Ð4Š$/ _wandbruntime‚14337  _runtime‚14337‘Q ~( ‹$ˆN UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2485459402399744 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚74908523 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚73766597 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚671537021382656 _runtime‚14337.67567445 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚22190975 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 148675126 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚682038176569344 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚46723759 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8122995174093 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚682037773948928 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10500787095552 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2484385838989312 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚247 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2485459402399744 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8135571030016 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚73766597 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚46724006 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2492508859591373 trainer/global_step‚8649 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚73766603 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚711 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2493594973429760 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8136916426752 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚711 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2493594973429760 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8136916426752 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 148675126 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚682038176569344 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚46723759 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8122995174093 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚682037773948928 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10500787095552 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2484385838989312 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚247 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2485459402399744 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8135571030016 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚73766597 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0Ç¿—™Ë4$/ _wandbruntime‚14337  _runtime‚14337:?eŽ$‰ 0loss/FRt1/assignment_loss‚1.4398101481593724 3loss/FBt2/assignment_loss‚0.0032543731031876924 1loss/SRqqt1/detection_loss‚0.7221093097206174 _runtime‚14337.761035023 0loss/FRt2/assignment_loss‚0.5126323531602175 2loss/SRqqt2/assignment_loss‚0.3523717260509392 3loss/FBt1/assignment_loss‚0.0019553487122252024 /loss/FRt1/detection_loss‚0.4773116530720374 1loss/SRqqt2/detection_loss‚0.6114230176677382 trainer/global_step‚8649 2loss/SRqqt1/assignment_loss‚0.4389278184535036 /loss/FRt2/detection_loss‚1.4147876114300133 0loss/FBt2/detection_loss‚0.33386969018238977 _step‚712 /loss/FBt1/detection_loss‚0.5048872463637963 &loss/total_loss‚6.8133402960760385 epoch‚21 % _timestamp‚1.7843041535440857e+09Ït($Ž 1loss/SRqqt2/detection_loss‚0.6114230176677382 trainer/global_step‚8649 2loss/SRqqt1/assignment_loss‚0.4389278184535036 /loss/FRt2/detection_loss‚1.4147876114300133 0loss/FBt2/detection_loss‚0.33386969018238977 _step‚712 /loss/FBt1/detection_loss‚0.5048872463637963 &loss/total_loss‚6.8133402960760385 epoch‚21 % _timestamp‚1.7843041535440857e+09 0loss/FRt1/assignment_loss‚1.4398101481593724 3loss/FBt2/assignment_loss‚0.0032543731031876924 1loss/SRqqt1/detection_loss‚0.7221093097206174 _runtime‚14337.761035023 0loss/FRt2/assignment_loss‚0.5126323531602175 2loss/SRqqt2/assignment_loss‚0.3523717260509392 3loss/FBt1/assignment_loss‚0.0019553487122252024 /loss/FRt1/detection_loss‚0.4773116530720374È”ÐÁŸ$‚(:” ›¤éÒö›À÷ gpu.0.gpu‚100.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚46.0 gpu.0.powerWatts‚345.878' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚69.17559999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚05µiŽ:‘$‚(:¯ ›¤éÒÚ£åú' disk.nvme0n1.out‚133367.68603515625 disk.sda.in‚ 75.2734375 network.sent‚ 1.6513829e+07 network.recv‚ 3.570058e+06) disk./.usagePercent‚37.63805743899729 disk.sda1.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-0.in‚ 32021.7421875# disk.nvme0c0n1.in‚ 32021.7421875! disk.nvme0n1.in‚ 32021.7421875& proc.memory.availableMB‚ 991.953125 disk.sda1.out‚0 disk.rbd1.out‚ 37.646484375 cpu‚25.566438433203057 proc.cpu.threads‚17$ disk./.usageGB‚560.8342552185059& disk.rbd0.in‚1.759218604441423e+13 disk.rbd0.out‚0 disk.rbd3.in‚0 disk.sda.out‚ 749.7734375$ disk.dm-0.out‚133367.68603515625 disk.drbd2165.out‚0 disk.rbd3.out‚0 disk.dm-1.out‚ 749.7734375 disk.sda3.in‚ 75.2734375 disk.drbd2165.in‚0 disk.dm-1.in‚ 75.2734375# proc.memory.rssMB‚ 2804.86328125* proc.memory.percent‚17.119526863098145 disk.sda3.out‚ 749.7734375) disk.nvme0c0n1.out‚133367.68603515625 disk.rbd1.in‚ 0.50830078125$ memory_percent‚93.94559860229492mÔž’$‚(:“ ª¤éÒ÷ûö gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚294.335' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚58.86699999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0˜ha‰5“$‚(:ª ª¤éÒŠ¬¨ù disk.dm-1.out‚ 750.33984375 disk.drbd2165.in‚0 cpu‚25.56623542827779 disk.sda.out‚ 750.33984375 disk.rbd3.out‚0$ memory_percent‚93.94562244415283$ disk./.usageGB‚560.8413543701172 disk.sda1.out‚0 disk.rbd1.in‚ 0.50830078125 disk.rbd1.out‚ 37.646484375# proc.memory.rssMB‚ 2804.86328125 disk.dm-0.in‚ 32199.6015625 disk.sda3.out‚ 750.33984375 disk.drbd2165.out‚0! disk.nvme0n1.in‚ 32199.6015625$ disk.nvme0n1.out‚133379.78515625 disk.sda2.in‚0& disk.rbd0.in‚1.759218604441423e+13# disk.nvme0c0n1.in‚ 32199.6015625( proc.memory.availableMB‚ 991.94921875* proc.memory.percent‚17.119526863098145 proc.cpu.threads‚17 network.sent‚ 1.6576784e+07) disk./.usagePercent‚37.63853386901852 disk.rbd0.out‚0! disk.dm-0.out‚133379.78515625 disk.sda3.in‚ 75.2734375 disk.rbd3.in‚0& disk.nvme0c0n1.out‚133379.78515625 network.recv‚ 3.574085e+06 disk.sda2.out‚0 disk.sda.in‚ 75.2734375 disk.sda1.in‚0 disk.dm-1.in‚ 75.2734375Êw»””$‚(:‰ ¹¤éÒƒÆøö gpu.0.gpu‚90.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚213.059' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚42.6118 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‚8•$‚(:­ ¹¤éÒÇ­õù disk.rbd1.in‚ 0.50830078125 disk.sda.out‚ 750.5390625$ disk.dm-0.out‚133391.89404296875 disk.sda1.out‚0 disk.drbd2165.in‚0$ disk./.usageGB‚560.8455963134766$ memory_percent‚93.94714832305908# proc.memory.rssMB‚ 2804.86328125 disk.sda.in‚ 75.2734375 disk.dm-1.in‚ 75.2734375 disk.dm-1.out‚ 750.5390625 disk.sda1.in‚0 disk.sda3.in‚ 75.2734375 disk.rbd3.in‚0( proc.memory.availableMB‚ 991.69921875* proc.memory.percent‚17.119526863098145 disk.dm-0.in‚ 32254.515625 disk.drbd2165.out‚0 disk.nvme0n1.in‚ 32254.515625 disk.rbd1.out‚ 37.708984375 network.recv‚ 3.577775e+06& disk.rbd0.in‚1.759218604441423e+13 disk.rbd0.out‚0 disk.sda3.out‚ 750.5390625) disk.nvme0c0n1.out‚133391.89404296875 disk.sda2.out‚0 cpu‚25.56605997711868 network.sent‚ 1.6581112e+07) disk./.usagePercent‚37.63881854939659 disk.rbd3.out‚0" disk.nvme0c0n1.in‚ 32254.515625 disk.sda2.in‚0' disk.nvme0n1.out‚133391.89404296875 proc.cpu.threads‚17K«¦yž–$‚(:“ ȤéÒ–ÿö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚289.84' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚57.967999999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0®VûÚ6—$‚(:« ȤéÒ¹ÞÓù disk.sda2.out‚0 disk.drbd2165.in‚0 disk.drbd2165.out‚0 proc.cpu.threads‚17) disk./.usagePercent‚37.63943859964452 disk.nvme0n1.in‚ 32254.9375 disk.rbd1.out‚ 37.708984375 disk.sda.in‚ 75.2734375$ memory_percent‚93.94714832305908 disk.sda3.in‚ 75.2734375 disk.rbd3.out‚0( proc.memory.availableMB‚ 991.69921875 network.sent‚ 1.6586216e+07 disk.dm-1.in‚ 75.2734375 disk.rbd0.out‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd3.in‚0# proc.memory.rssMB‚ 2804.86328125* proc.memory.percent‚17.119526863098145$ disk./.usageGB‚560.8548355102539 disk.sda2.in‚0& disk.rbd0.in‚1.759218604441423e+13 disk.sda3.out‚ 750.93359375 disk.sda.out‚ 750.93359375 disk.nvme0c0n1.in‚ 32254.9375 disk.dm-0.in‚ 32254.9375$ disk.dm-0.out‚133400.93701171875 disk.dm-1.out‚ 750.93359375 cpu‚25.565835981913825) disk.nvme0c0n1.out‚133400.93701171875 disk.rbd1.in‚ 0.50830078125 network.recv‚ 3.581277e+06' disk.nvme0n1.out‚133400.93701171875½«hðŸ˜$‚(:” פéÒåÄ÷ gpu.0.gpu‚89.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚288.151' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚57.630199999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‡À­|7™$‚(:¬ פéÒ·Ô·ù network.sent‚ 1.6591293e+07) disk./.usagePercent‚37.63905330830549 disk.dm-0.in‚ 32255.171875 disk.drbd2165.out‚0 disk.rbd3.out‚0* proc.memory.percent‚17.119526863098145 network.recv‚ 3.584701e+06 disk.sda.out‚ 751.56640625& disk.rbd0.in‚1.759218604441423e+13 disk.sda3.in‚ 75.29296875' disk.nvme0c0n1.out‚133414.818359375 disk.rbd1.out‚ 37.771484375 disk.sda3.out‚ 751.56640625& proc.memory.availableMB‚ 991.703125" disk.dm-0.out‚133414.818359375 disk.rbd3.in‚0 disk.nvme0n1.in‚ 32255.171875% disk.nvme0n1.out‚133414.818359375 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda2.out‚0 cpu‚25.565664333035425 proc.cpu.threads‚17$ disk./.usageGB‚560.8490943908691 disk.sda.in‚ 75.29296875 disk.rbd1.in‚ 0.50830078125 disk.dm-1.in‚ 75.29296875 disk.sda1.in‚0 disk.drbd2165.in‚0" disk.nvme0c0n1.in‚ 32255.171875$ memory_percent‚93.94712448120117# proc.memory.rssMB‚ 2804.86328125 disk.dm-1.out‚ 751.56640625 disk.rbd0.out‚0œˆŸš$‚(:” æ¤éÒÒÅŽ÷ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚269.062' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚53.812400000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ºæ|Y ›$‚(:• æ¤éÒä’Êù* disk./.usagePercent‚37.639538186575344* proc.memory.percent‚17.119526863098145 proc.cpu.threads‚17 network.sent‚ 1.6600067e+07 disk.sda.out‚ 751.8203125 disk.sda1.out‚0 disk.sda3.out‚ 751.8203125 disk.nvme0n1.in‚ 32255.375& disk.rbd0.in‚1.759218604441423e+13# proc.memory.rssMB‚ 2804.86328125 cpu‚25.565457059024933 disk.drbd2165.in‚0 disk.rbd1.in‚ 0.50830078125$ disk./.usageGB‚560.8563194274902 disk.rbd0.out‚0 disk.dm-0.out‚ 133428.03125 disk.sda3.in‚ 75.29296875 disk.rbd3.out‚0# disk.nvme0c0n1.out‚ 133428.03125 disk.rbd1.out‚ 37.771484375 network.recv‚ 3.594693e+06 disk.nvme0c0n1.in‚ 32255.375! disk.nvme0n1.out‚ 133428.03125 disk.sda1.in‚0 disk.drbd2165.out‚0 disk.rbd3.in‚0 disk.sda2.out‚0 disk.sda.in‚ 75.29296875 disk.dm-1.in‚ 75.29296875 disk.dm-1.out‚ 751.8203125 disk.dm-0.in‚ 32255.375 disk.sda2.in‚0$ memory_percent‚93.94712448120117& proc.memory.availableMB‚ 991.703125¯ q4œ$/ _wandbruntime‚14415  _runtime‚14415°Ó­È$ _step‚713 & lr-AdamW/pg1‚0.0004283489812128771 & lr-AdamW/pg2‚0.0004283489812128771 trainer/global_step‚8699 % _timestamp‚1.7843042309605587e+09 _runtime‚14415.177692983"ž³2Íž$Ç trainer/global_step‚8699 % _timestamp‚1.7843042309605587e+09 _runtime‚14415.177692983 _step‚713 & lr-AdamW/pg1‚0.0004283489812128771 & lr-AdamW/pg2‚0.0004283489812128771É[2;ô4Ÿ$/ _wandbruntime‚14415  _runtime‚144152Z± 8 $€P % _timestamp‚1.7843042309616637e+09 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚74168697 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚251 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2506170380368747 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚22315582 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8166964220521 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚46984780 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 149494480 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚256 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2499081464348672 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2506169010340457 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2498002046119936 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚74168697 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚22315577 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 _runtime‚14415.178983971 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2507263771663360 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2507262400100352 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8180961925120 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10559625113600 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚75315810 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚24669198 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚685893445936128 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚75325783 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚75315810 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175ãæ … (¡$…P L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚74168691 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2507262400100352 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚75325783 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚675333855358976 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚714 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8182307314688 trainer/global_step‚8699 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2498002071547904 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚24668947 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚685893848563712 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10559590577152 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8182307314688 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 149484501 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚74168691 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 _runtime‚14415.178983971 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2507263771663360 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2507262400100352 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8180961925120 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10559625113600 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚75315810 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚24669198 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚685893445936128 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚75325783 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚75315810 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 % _timestamp‚1.7843042309616637e+09 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚74168697 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚251 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2506170380368747 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚22315582 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8166964220521 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚46984780 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 149494480 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚256 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2499081464348672 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2506169010340457 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2498002046119936 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚74168697 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚22315577 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 149511178 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚675411709857792 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2506449185613281 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚248 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚685972534905856 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7843042323487372e+09 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2499359465078784 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 149501185 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2498279927898112 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8183232357376 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8169232287201 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚22318125 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚685972132285440 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8181886960640 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚75324288 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚75324288 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8181886960640 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 trainer/global_step‚8699 _step‚715 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 149501185 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚75334275 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2499359465078784 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8183232357376 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 149511178 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2507542697436160 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚46989850 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚24671730 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚22318120 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2499359438905344 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2499359438905344 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚74176897 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚74176903 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24671973 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 149511178 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚675411709857792 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2506449185613281 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚248 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚685972534905856 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7843042323487372e+09 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2499359465078784 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 149501185 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.largeÞlU_pool.freed‚2498279927898112 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8183232357376 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8169232287201 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚22318125 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚685972132285440 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8181886960640 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚75324288 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚75324288 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8181886960640 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 trainer/global_step‚8699 _step‚715 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 149501185 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚75334275 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2499359465078784 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8183232357376 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 149511178 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2507542697436160 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚46989850 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚24671730˵~€4¥$/ _wandbruntime‚14416  _runtime‚14416ø‚Ù| ¦$ƒ 0loss/FRt1/assignment_loss‚1.3571318981174312 2loss/FBt2/assignment_loss‚0.017083210706609418 /loss/FRt2/detection_loss‚1.3938165553441575 %loss/total_loss‚6.958379411062145 epoch‚21 _runtime‚14416.650945674 /loss/FBt2/detection_loss‚0.3924214542950604 0loss/FRt2/assignment_loss‚0.5049538379725145 2loss/SRqqt1/assignment_loss‚0.4127779791329519 1loss/FBt1/assignment_loss‚0.01755374283298779 /loss/FRt1/detection_loss‚0.5245691337586637 0loss/SRqqt1/detection_loss‚0.709502030275237 1loss/SRqqt2/detection_loss‚0.5226391395992177 trainer/global_step‚8699 _step‚716 2loss/SRqqt2/assignment_loss‚0.4198301759583936 /loss/FBt1/detection_loss‚0.6861002530689196 % _timestamp‚1.7843042324339712e+09Àt;Ч$ˆ 0loss/FRt1/assignment_loss‚1.3571318981174312 2loss/FBt2/assignment_loss‚0.017083210706609418 /loss/FRt2/detection_loss‚1.3938165553441575 %loss/total_loss‚6.958379411062145 epoch‚21 _runtime‚14416.650945674 /loss/FBt2/detection_loss‚0.3924214542950604 0loss/FRt2/assignment_loss‚0.5049538379725145 2loss/SRqqt1/assignment_loss‚0.4127779791329519 1loss/FBt1/assignment_loss‚0.01755374283298779 /loss/FRt1/detection_loss‚0.5245691337586637 0loss/SRqqt1/detection_loss‚0.709502030275237 1loss/SRqqt2/detection_loss‚0.5226391395992177 trainer/global_step‚8699 _step‚716 2loss/SRqqt2/assignment_loss‚0.4198301759583936 /loss/FBt1/detection_loss‚0.6861002530689196 % _timestamp‚1.7843042324339712e+09ÌœÆ4]”¨$‚(:‰ õ¤éÒó†êö gpu.0.gpu‚96.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚242.671' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚48.5342 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0/Xˆ¾>©$‚(:³ õ¤éÒê–ù network.recv‚ 3.599269e+06 disk.rbd1.in‚ 0.50830078125 disk.rbd1.out‚ 37.833984375$ memory_percent‚93.94903182983398# proc.memory.rssMB‚ 2804.86328125 cpu‚25.565286863989567 network.sent‚ 1.6653916e+07 disk.sda.out‚ 752.53515625& disk.rbd0.in‚1.759218604441423e+13 disk.sda1.out‚0 disk.drbd2165.out‚0# disk.nvme0c0n1.in‚ 32355.6015625* disk./.usagePercent‚37.640017176671904# disk.dm-0.out‚133438.1611328125 disk.sda1.in‚0( disk.nvme0c0n1.out‚133438.1611328125& disk.nvme0n1.out‚133438.1611328125 disk.sda.in‚ 75.29296875 disk.dm-0.in‚ 32355.6015625 disk.drbd2165.in‚0! disk.nvme0n1.in‚ 32355.6015625 disk.sda2.in‚0 disk.dm-1.out‚ 752.53515625 disk.sda3.out‚ 752.53515625 disk.rbd3.out‚0 proc.cpu.threads‚17 disk.dm-1.in‚ 75.29296875 disk.rbd0.out‚0 disk.sda3.in‚ 75.29296875& proc.memory.availableMB‚ 991.390625$ disk./.usageGB‚560.8634567260742 disk.rbd3.in‚0 disk.sda2.out‚0* proc.memory.percent‚17.119526863098145‚𠔪$‚(:‰ „¥éÒ¶„àö gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚40.0 gpu.0.powerWatts‚299.456' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚59.8912 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚06Ù4n:«$‚(:¯ „¥éÒØçÇø proc.cpu.threads‚17 network.sent‚ 1.6706593e+07 disk.sda3.out‚ 755.20703125 disk.rbd1.out‚ 37.833984375 disk.sda.out‚ 755.20703125 disk.rbd0.out‚0 disk.sda1.in‚0 disk.sda1.out‚0 cpu‚25.565083193286817& disk.nvme0c0n1.out‚133465.60546875 disk.sda3.in‚ 75.29296875# disk.nvme0c0n1.in‚ 32355.8203125$ disk.nvme0n1.out‚133465.60546875 disk.rbd1.in‚ 0.50830078125 disk.dm-1.out‚ 755.20703125! disk.dm-0.out‚133465.60546875* proc.memory.percent‚17.119526863098145* disk./.usagePercent‚37.640400419950666 disk.rbd3.out‚0 disk.sda2.out‚0! disk.nvme0n1.in‚ 32355.8203125( proc.memory.availableMB‚ 991.10546875 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.sda2.in‚0 disk.dm-1.in‚ 75.29296875 disk.rbd3.in‚0 disk.dm-0.in‚ 32355.8203125$ memory_percent‚93.95077228546143# proc.memory.rssMB‚ 2804.86328125 network.recv‚ 3.603858e+06$ disk./.usageGB‚560.8691673278809 disk.sda.in‚ 75.29296875& disk.rbd0.in‚1.759218604441423e+13@¹¼€”¬$‚(:‰ “¥éÒ°ð÷ gpu.0.gpu‚90.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚272.007' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.4014 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0(EåD4­$‚(:© “¥éÒÄ©Åø" disk.dm-0.out‚133478.572265625 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda3.in‚ 75.29296875 disk.sda3.out‚ 757.828125% disk.nvme0n1.out‚133478.572265625# memory_percent‚93.9495325088501) disk./.usagePercent‚37.64093803577257 disk.sda.in‚ 75.29296875# proc.memory.rssMB‚ 2804.86328125* proc.memory.percent‚17.119526863098145 disk.dm-0.in‚ 32356.2421875# disk.nvme0c0n1.in‚ 32356.2421875' disk.nvme0c0n1.out‚133478.572265625 disk.dm-1.in‚ 75.29296875( proc.memory.availableMB‚ 991.30859375 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd1.in‚ 0.50830078125 disk.sda.out‚ 757.828125& disk.rbd0.in‚1.759218604441423e+13 disk.rbd0.out‚0 proc.cpu.threads‚17$ disk./.usageGB‚560.8771781921387 disk.dm-1.out‚ 757.828125 disk.rbd3.in‚0 disk.rbd3.out‚0 cpu‚25.56491229746507 disk.drbd2165.out‚0 disk.rbd1.out‚ 37.896484375 network.recv‚ 3.608319e+06 disk.drbd2165.in‚0! disk.nvme0n1.in‚ 32356.2421875 network.sent‚ 1.6711931e+07új“®$‚(:ˆ ¢¥éÒßÓøö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚287.615' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚57.523 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0´¾D>¯$‚(:³ ¢¥éÒŸâÜû disk.rbd3.in‚0 disk.sda.out‚ 758.41796875 disk.dm-0.in‚ 32356.4453125# proc.memory.rssMB‚ 2804.86328125 network.sent‚ 1.6715523e+07$ disk./.usageGB‚560.8812446594238( disk.nvme0c0n1.out‚133495.8798828125 disk.drbd2165.out‚0# disk.nvme0c0n1.in‚ 32356.4453125 disk.dm-1.in‚ 75.29296875 disk.dm-1.out‚ 758.41796875 disk.sda3.in‚ 75.29296875 disk.sda3.out‚ 758.41796875& disk.nvme0n1.out‚133495.8798828125 disk.rbd1.in‚ 0.50830078125 disk.sda.in‚ 75.29296875* proc.memory.percent‚17.119526863098145) disk./.usagePercent‚37.64121093980407 disk.sda2.in‚0 disk.rbd1.out‚ 37.896484375' proc.memory.availableMB‚ 991.3046875 disk.sda2.out‚0 disk.rbd0.out‚0# disk.dm-0.out‚133495.8798828125 network.recv‚ 3.610502e+06 disk.rbd3.out‚0! disk.nvme0n1.in‚ 32356.4453125 cpu‚25.564729933251716 proc.cpu.threads‚17& disk.rbd0.in‚1.759218604441423e+13 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd2165.in‚0$ memory_percent‚93.94955635070801ćwóŸ°$‚(:” ±¥éÒ˜ñö gpu.0.gpu‚99.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚302.407' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚60.481399999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0áµÐD±$‚(:¹ ±¥éÒÌéÈù disk.dm-1.in‚ 75.29296875 disk.dm-1.out‚ 759.02734375 disk.sda1.in‚0 disk.sda3.out‚ 759.02734375 disk.drbd2165.in‚0 disk.drbd2165.out‚0$ disk.nvme0c0n1.in‚32454.13671875$ disk./.usageGB‚560.8880500793457 disk.sda3.in‚ 75.29296875 disk.rbd1.in‚ 0.50830078125 disk.rbd0.out‚0* proc.memory.percent‚17.119526863098145 cpu‚25.564533170857306 disk.sda2.in‚0 disk.rbd1.out‚ 37.958984375 disk.sda.in‚ 75.29296875 disk.sda1.out‚0" disk.nvme0n1.in‚32454.13671875 network.sent‚ 1.6725906e+07 disk.rbd3.out‚0 disk.sda.out‚ 759.02734375' proc.memory.availableMB‚ 991.2734375# proc.memory.rssMB‚ 2804.86328125 proc.cpu.threads‚17 network.recv‚ 3.622158e+06' disk.nvme0n1.out‚133503.89599609375) disk./.usagePercent‚37.64166765724515) disk.nvme0c0n1.out‚133503.89599609375 disk.sda2.out‚0$ disk.dm-0.out‚133503.89599609375$ memory_percent‚93.94974708557129 disk.rbd3.in‚0& disk.rbd0.in‚1.759218604441423e+13 disk.dm-0.in‚32454.13671875Ör4²$/ _wandbruntime‚14493  _runtime‚14493£ ‘>dz$Á & lr-AdamW/pg2‚0.0004256538826543076 trainer/global_step‚8749 $ _timestamp‚1.784304309701107e+09 _runtime‚14493.918395306 _step‚717 & lr-AdamW/pg1‚0.0004256538826543076È&µÌ´$Æ & lr-AdamW/pg1‚0.0004256538826543076 & lr-AdamW/pg2‚0.0004256538826543076 trainer/global_step‚8749 $ _timestamp‚1.784304309701107e+09 _runtime‚14493.918395306 _step‚717Íì_Y4µ$/ _wandbruntime‚14493  _runtime‚14493 ‰)›(¶$€P P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2521210124099584 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚24808073 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2512981474680832 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚679208175742976 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 _runtime‚14493.923016122 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8228623245312 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2521208752536576 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚47250559 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2511896135028736 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8227277855744 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚246 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2520110706390655 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2511896160456704 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 trainer/global_step‚8749 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2512981474680832 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 150330532 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚74578991 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚24807832 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0[n†·"·$…P QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚241 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8214545933951 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚74578997 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚74578997 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚679208543834112 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 150320553 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2520109336362365 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2512981500854272 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚47250805 _step‚718 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚689828103759872 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚22442727 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 % _timestamp‚1.7843043097055392e+09 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8227277855744 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2521210124099584 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10619559925760 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚75751535 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2512981500854272 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8228623245312 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚75741562 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚75741562 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚689827701132288 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚24808073 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2512981474680832 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚679208175742976 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 _runtime‚14493.923016122 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8228623245312 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2521208752536576 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚47250559 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2511896135028736 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8227277855744 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚246 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2520110706390655 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2511896160456704 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 trainer/global_step‚8749 G>Devic,V(DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2512981474680832 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 150330532 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚74578991 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚24807832 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2521210124099584 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚233 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2521489049872384 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚74587203 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚22445270 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚75760027 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2512174042234880 _step‚719 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 150347230 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2513259475410944 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚47255847 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8229548288000 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 150347230 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚75760027 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24810810 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚679286030241792 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚32432128 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2521489049872384 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2512174016806912 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2513259475410944 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 150337237 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚689906795148288 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8228202891264 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚47256085 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚679285662150656 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8228202891264 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10620732474368 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2521487678302208 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚24810577 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10620764906496 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 % _timestamp‚1.7843043110918007e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8228202891264 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10620732474368 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2521487678302208 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚24810577 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10620764906496 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 % _timestamp‚1.7843043110918007e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚233 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2521489049872384 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚74587203 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚22445270 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚75760027 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2512174042234880 _step‚719 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 150347230 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2513259475410944 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚47255847 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8229548288000 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 150347230 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚75760027 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24810810 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚679286030241792 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚32432128 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097Ïk_P04»$/ _wandbruntime‚14495  _runtime‚14495Gpª ¼$‡ 2loss/FBt1/assignment_loss‚0.008853415849964548 1loss/SRqqt1/detection_loss‚0.7864857086713776 % _timestamp‚1.7843043111767027e+09 2loss/SRqqt2/assignment_loss‚0.4431887608251404 /loss/FBt1/detection_loss‚0.6749562131063569 0loss/FBt2/detection_loss‚0.36431944877563033 0loss/FRt1/assignment_loss‚1.4370262723322778 3loss/FBt2/assignment_loss‚0.0031982565005092656 1loss/SRqqt2/detection_loss‚0.5937803110547643 %loss/total_loss‚7.277692467332892 2loss/SRqqt1/assignment_loss‚0.4708167404588442 /loss/FRt1/detection_loss‚0.5050751651568394 /loss/FRt2/detection_loss‚1.4526889639519118 epoch‚21 trainer/global_step‚8749 _runtime‚14495.393718866 _step‚720 0loss/FRt2/assignment_loss‚0.5373032106492752â¬2½$Œ 2loss/SRqqt2/assignment_loss‚0.4431887608251404 /loss/FBt1/detection_loss‚0.6749562131063569 0loss/FBt2/detection_loss‚0.36431944877563033 0loss/FRt1/assignment_loss‚1.4370262723322778 3loss/FBt2/assignment_loss‚0.0031982565005092656 1loss/SRqqt2/detection_loss‚0.5937803110547643 %loss/total_loss‚7.277692467332892 2loss/SRqqt1/assignment_loss‚0.4708167404588442 /loss/FRt1/detection_loss‚0.5050751651568394 /loss/FRt2/detection_loss‚1.4526889639519118 epoch‚21 trainer/global_step‚8749 _runtime‚14495.393718866 _step‚720 0loss/FRt2/assignment_loss‚0.5373032106492752 2loss/FBt1/assignment_loss‚0.008853415849964548 1loss/SRqqt1/detection_loss‚0.7864857086713776 % _timestamp‚1.7843043111767027e+09ÐÃò=”¾$‚(:‰ À¥éÒ§ï‚û gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚263.646' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.7292 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0á7qqB¿$‚(:· À¥éÒ¤‡Ñü network.recv‚ 3.625346e+06 disk.sda3.in‚ 75.29296875 disk.sda2.out‚0) disk./.usagePercent‚37.64246998485746 disk.dm-0.in‚32454.55859375 disk.drbd2165.out‚0 disk.sda.out‚ 761.26171875 disk.dm-1.out‚ 761.26171875& disk.rbd0.in‚1.759218604441423e+13 disk.rbd0.out‚0" disk.nvme0n1.in‚32454.55859375 disk.rbd1.out‚ 37.958984375 disk.sda.in‚ 75.29296875# memory_percent‚93.9516544342041* proc.memory.percent‚17.119526863098145 cpu‚25.56454141384578 proc.cpu.threads‚17 disk.dm-1.in‚ 75.29296875 disk.sda1.in‚0 disk.drbd2165.in‚0 disk.rbd3.in‚0$ disk.nvme0c0n1.in‚32454.55859375 disk.sda2.in‚0 disk.rbd3.out‚0 disk.rbd1.in‚ 0.50830078125$ disk./.usageGB‚560.9000053405762$ disk.dm-0.out‚133518.62548828125 disk.sda3.out‚ 761.26171875 network.sent‚ 1.6729638e+07 disk.sda1.out‚0) disk.nvme0c0n1.out‚133518.62548828125' disk.nvme0n1.out‚133518.62548828125' proc.memory.availableMB‚ 990.9609375# proc.memory.rssMB‚ 2804.86328125ó\©žÀ$‚(:“ Ï¥éÒСËö gpu.0.gpu‚84.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚333.524' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚66.70479999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¸§þ,Á$‚(:¡ Ï¥éÒ–ü…ú$ disk./.usageGB‚560.9040870666504 disk.sda3.out‚762.3125( disk.nvme0c0n1.out‚133537.3486328125 disk.sda2.in‚0 disk.sda.out‚762.3125" memory_percent‚93.951416015625# disk.dm-0.out‚133537.3486328125 disk.rbd1.out‚ 38.083984375* disk./.usagePercent‚37.642743912919094" disk.nvme0n1.in‚32454.76171875 proc.memory.availableMB‚991& disk.rbd0.in‚1.759218604441423e+13& disk.nvme0n1.out‚133537.3486328125# proc.memory.rssMB‚ 2804.86328125* proc.memory.percent‚17.119526863098145 disk.rbd0.out‚0 disk.dm-0.in‚32454.76171875 disk.sda3.in‚ 75.29296875 disk.rbd3.out‚0 disk.rbd1.in‚ 0.50830078125 cpu‚25.564776577461757 disk.dm-1.in‚ 75.29296875 disk.drbd2165.in‚0$ disk.nvme0c0n1.in‚32454.76171875 disk.sda2.out‚0 proc.cpu.threads‚17 network.sent‚ 1.6792367e+07 disk.dm-1.out‚762.3125 disk.sda1.in‚0 disk.drbd2165.out‚0 disk.rbd3.in‚0 disk.sda.in‚ 75.29296875 network.recv‚ 3.630012e+06 disk.sda1.out‚0Ã7:¯”Â$‚(:‰ Þ¥éÒå¥þö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚263.646' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.7292 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ƒÞ"U>Ã$‚(:³ Þ¥éÒ¶Ò˜ù disk.rbd1.out‚ 38.083984375( proc.memory.availableMB‚ 990.94140625 proc.cpu.threads‚17 cpu‚25.56488467029009 network.recv‚ 3.633194e+06$ disk./.usageGB‚560.9101524353027 disk.sda1.in‚0 disk.drbd2165.in‚0# proc.memory.rssMB‚ 2804.86328125 disk.dm-1.in‚ 75.29296875 disk.sda3.in‚ 75.29296875 disk.sda3.out‚ 762.671875 disk.drbd2165.out‚0 disk.rbd1.in‚ 0.50830078125$ memory_percent‚93.95177364349365 disk.sda.in‚ 75.29296875 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.sda2.in‚0 disk.sda.out‚ 762.671875 disk.rbd0.out‚0 disk.sda1.out‚0$ disk.nvme0c0n1.in‚32454.98046875) disk.nvme0c0n1.out‚133552.40966796875" disk.nvme0n1.in‚32454.98046875* proc.memory.percent‚17.119526863098145 network.sent‚ 1.6797984e+07) disk./.usagePercent‚37.64315096489854 disk.dm-1.out‚ 762.671875& disk.rbd0.in‚1.759218604441423e+13' disk.nvme0n1.out‚133552.40966796875$ disk.dm-0.out‚133552.40966796875 disk.sda2.out‚0 disk.dm-0.in‚32454.98046875Ujk’Ä$‚(:‡ í¥éÒ盂÷ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚41.0 gpu.0.powerWatts‚334.052' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚66.8104 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ç¢Þ+Å$‚(:  í¥éÒŠõÙú disk.sda3.in‚ 75.29296875$ disk./.usageGB‚560.9170455932617 disk.drbd2165.out‚0 disk.rbd0.out‚0 disk.rbd3.out‚0% disk.nvme0c0n1.out‚133565.8828125 disk.rbd1.in‚ 0.50830078125 disk.rbd1.out‚ 38.146484375$ memory_percent‚93.95203590393066 disk.drbd2165.in‚0 disk.sda.out‚ 762.984375' proc.memory.availableMB‚ 990.8984375* proc.memory.percent‚17.119526863098145 disk.dm-1.in‚ 75.29296875 disk.dm-0.in‚ 32587.171875 disk.sda1.out‚0) disk./.usagePercent‚37.64361357051291 disk.rbd3.in‚0" disk.nvme0c0n1.in‚ 32587.171875 disk.sda2.out‚0 disk.sda.in‚ 75.29296875 network.sent‚ 1.6803038e+07 disk.dm-0.out‚133565.8828125# proc.memory.rssMB‚ 2804.86328125 cpu‚25.56512112349217 disk.sda3.out‚ 762.984375 disk.sda2.in‚0 proc.cpu.threads‚17 disk.nvme0n1.in‚ 32587.171875 disk.dm-1.out‚ 762.984375& disk.rbd0.in‚1.759218604441423e+13 disk.sda1.in‚0# disk.nvme0n1.out‚133565.8828125 network.recv‚ 3.636712e+06ëûqa”Æ$‚(:‰ ü¥éÒ±Öö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚251.859' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚50.3718 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ÿéf2Ç$‚(:§ ü¥éÒÉåàø& disk.nvme0n1.out‚133575.7685546875 disk.sda2.in‚0 disk.sda.out‚ 763.27734375 disk.sda1.in‚0 disk.sda3.in‚ 75.29296875 disk.nvme0n1.in‚ 32587.375$ memory_percent‚93.95098686218262* proc.memory.percent‚17.119526863098145 proc.cpu.threads‚17$ disk./.usageGB‚560.9233779907227 disk.rbd0.out‚0# disk.dm-0.out‚133575.7685546875 disk.rbd1.out‚ 38.146484375 disk.dm-1.out‚ 763.27734375 disk.drbd2165.in‚0 disk.rbd3.in‚0 disk.rbd1.in‚ 0.50830078125 cpu‚25.56528408968306 disk.drbd2165.out‚0 disk.rbd3.out‚0 disk.nvme0c0n1.in‚ 32587.375 network.recv‚ 3.639232e+06* disk./.usagePercent‚37.644038543019754 disk.dm-1.in‚ 75.29296875 disk.sda3.out‚ 763.27734375# proc.memory.rssMB‚ 2804.86328125 network.sent‚ 1.6807191e+07 disk.sda.in‚ 75.29296875& disk.rbd0.in‚1.759218604441423e+13 disk.dm-0.in‚ 32587.375 disk.sda1.out‚0( disk.nvme0c0n1.out‚133575.7685546875 disk.sda2.out‚0' proc.memory.availableMB‚ 991.0703125`9p 4È$/ _wandbruntime‚14573  _runtime‚14573DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚74989291 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚47516281 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36633600 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚22569877 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 $ _timestamp‚1.784304389371954e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚76177287 _runtime‚14573.589319144 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚74989297 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 _step‚722 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2526881537359872 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404724736 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2525790249365504 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8273593786368 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 151166584 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚76177287 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚693761682761728 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚74989291 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚24946399 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2535155104972800 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2534051032412563 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 trainer/global_step‚8799 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚24946137 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2526881537359872 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2525790223937536 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2526881511186432 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚74989297 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚76167314 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10679223268352 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2534049662384273 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2535155104972800 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2526881511186432 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚693762087486464 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚683082496126976 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚76167314 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚47516014 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648Kã¸8 (Í$„P TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚22569877 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 $ _timestamp‚1.784304389371954e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚76177287 _runtime‚14573.589319144 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚74989297 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 _step‚722 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2526881537359872 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404724736 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2525790249365504 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8273593786368 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 151166584 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚76177287 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚693761682761728 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚74989291 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚24946399 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2535155104972800 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2534051032412563 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 trainer/global_step‚8799 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚24946137 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2526881537359872 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2525790223937536 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2526881511186432 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚74989297 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚76167314 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10679223268352 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2534049662384273 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2535155104972800 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2526881511186432 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚693762087486464 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚683082496126976 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚76167314 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚47516014 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚262 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 151156605 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚267 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2535156476535808 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚683082864218112 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10679186634752 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚74989291 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚47516281 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36633600 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640Ò2 æ*4Î$/ _wandbruntime‚14574  _runtime‚14574ƒ©A&'Ï$ŠN RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2527159511916544 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 151173289 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8261706513417 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2527159511916544 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 151183282 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36626432 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 % _timestamp‚1.7843043907582963e+09 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚22572420 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 _step‚723 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚74997503 _runtime‚14574.975830597 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 151173289 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2534328467628695 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 trainer/global_step‚8799 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚74997497 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚74997497 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚76175792 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2526068131143680 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚47521589 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2535434030738432 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚693840358517760 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚76185779 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 151183282 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10680412609536 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚47521326 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚683159982534656 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2527159538089984 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8274518821888 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8275864218624 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8274518821888 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24949164 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚74997503 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚263 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10680375983104 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8275864218624 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚693840763235328 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚74997497 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚74997497 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚76175792 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2526068131143680 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚47521589 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2535434030738432 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚693840358517760 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚76185779 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 151183282 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10680412609536 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚47521326 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚683159982534656 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2527159538089984 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8274518821888 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8275864218624 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8274518821888 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24949164 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚74997503 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚263 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10680375983104 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8275864218624 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚693840763235328 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚22572420 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 _step‚723 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚74997503 _runtime‚14574.975830597 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 151173289 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2534328467628695 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 trainer/global_step‚8799Ó&è¦+4Ñ$/ _wandbruntime‚14575  _runtime‚14575  "Ò$Œ 0loss/FRt1/assignment_loss‚1.4110529188676313 4loss/FBt2/assignment_loss‚0.00047507633238689796 /loss/FRt2/detection_loss‚1.4265740654060926 1loss/SRqqt2/detection_loss‚0.6156568811558576 trainer/global_step‚8799 %loss/total_loss‚6.979923908269869 _runtime‚14575.060233003 _step‚724 /loss/FRt1/detection_loss‚0.4885774208398255 1loss/SRqqt1/detection_loss‚0.7332659581473145 epoch‚21 0loss/FRt2/assignment_loss‚0.6067739910412329 3loss/SRqqt1/assignment_loss‚0.38093763155206684 3loss/SRqqt2/assignment_loss‚0.45950115331498176 3loss/FBt1/assignment_loss‚0.0009974257646745187 0loss/FBt1/detection_loss‚0.39617404836769166 0loss/FBt2/detection_loss‚0.45993733748011373 % _timestamp‚1.7843043908431652e+09ëÖ“»Ó$‘ 3loss/SRqqt2/assignment_loss‚0.45950115331498176 3loss/FBt1/assignment_loss‚0.0009974257646745187 0loss/FBt1/detection_loss‚0.39617404836769166 0loss/FBt2/detection_loss‚0.45993733748011373 % _timestamp‚1.7843043908431652e+09 0loss/FRt1/assignment_loss‚1.4110529188676313 4loss/FBt2/assignment_loss‚0.00047507633238689796 /loss/FRt2/detection_loss‚1.4265740654060926 1loss/SRqqt2/detection_loss‚0.6156568811558576 trainer/global_step‚8799 %loss/total_loss‚6.979923908269869 _runtime‚14575.060233003 _step‚724 /loss/FRt1/detection_loss‚0.4885774208398255 1loss/SRqqt1/detection_loss‚0.7332659581473145 epoch‚21 0loss/FRt2/assignment_loss‚0.6067739910412329 3loss/SRqqt1/assignment_loss‚0.38093763155206684Ô«có”Ô$‚(:‰ ‹¦éÒ¡ÆÏö gpu.0.gpu‚98.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚340.764' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚68.1528 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0úìº%)Õ$‚(:ž ‹¦éÒêóø disk.nvme0n1.in‚ 32587.796875$ disk./.usageGB‚560.9312133789062 disk.sda.in‚ 75.29296875 disk.dm-1.out‚763.5( proc.memory.availableMB‚ 990.56640625 network.recv‚ 3.641896e+06) disk./.usagePercent‚37.64456438249508 disk.sda.out‚763.5& disk.rbd0.in‚1.759218604441423e+13 disk.sda3.in‚ 75.29296875" disk.nvme0c0n1.in‚ 32587.796875$ memory_percent‚93.95406246185303* proc.memory.percent‚17.119526863098145 disk.dm-0.in‚ 32587.796875$ disk.dm-0.out‚133584.29931640625 disk.drbd2165.out‚0 disk.sda2.out‚0 disk.rbd1.in‚ 0.50830078125 cpu‚25.56612379165184 proc.cpu.threads‚17 disk.rbd0.out‚0 disk.sda3.out‚763.5 disk.sda2.in‚0 disk.rbd1.out‚ 38.208984375) disk.nvme0c0n1.out‚133584.29931640625 network.sent‚ 1.6812112e+07 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.rbd3.out‚0 disk.dm-1.in‚ 75.29296875 disk.rbd3.in‚0# proc.memory.rssMB‚ 2804.86328125' disk.nvme0n1.out‚133584.29931640625sT®,Ö$‚(:„ š¦éÒ¿Öóö gpu.0.gpu‚1.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚41.0 gpu.0.powerWatts‚338.35' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚67.67 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0óÀ:3×$‚(:¨ š¦éÒó¼¿ù disk.dm-1.out‚763.6875 disk.sda3.in‚ 75.29296875 disk.drbd2165.in‚0) disk.nvme0c0n1.out‚133599.32080078125$ disk./.usageGB‚560.9346008300781 disk.drbd2165.out‚0 disk.rbd3.out‚0( proc.memory.availableMB‚ 990.55859375 network.sent‚ 1.6876721e+07 disk.sda.in‚ 75.29296875 disk.sda1.in‚0 disk.rbd1.out‚ 38.208984375 disk.sda.out‚763.6875& disk.rbd0.in‚1.759218604441423e+13$ disk.dm-0.out‚133599.32080078125 disk.sda2.out‚0 disk.rbd1.in‚ 0.50830078125 disk.dm-1.in‚ 75.29296875 disk.sda3.out‚763.6875# proc.memory.rssMB‚ 2804.86328125) disk./.usagePercent‚37.64479171718549 disk.rbd0.out‚0 disk.dm-0.in‚ 32588.015625 disk.nvme0n1.in‚ 32588.015625 proc.cpu.threads‚17 network.recv‚ 3.647145e+06 disk.sda2.in‚0 disk.rbd3.in‚0" disk.nvme0c0n1.in‚ 32588.015625 disk.sda1.out‚0' disk.nvme0n1.out‚133599.32080078125$ memory_percent‚93.95411014556885* proc.memory.percent‚17.119526863098145 cpu‚25.569012379793946p¤b¯4Ø$/ _wandbruntime‚14602  _runtime‚14602œ$‘ä=Ù$Þ{ 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5995661020278931 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6155828833580017 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6735625863075256 6"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6755411028862 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.10185185074806213 9"Purity/*FRt2SRqqt*FBt/event_purity‚0.2020145207643509 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.06499029695987701 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8711639642715454 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.014348335564136505 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9951818585395813 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9867116808891296 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.18218374252319336 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.24447476863861084 particle/accuracy_1_of_3‚1 jet/accuracy_1_of_5‚1 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9911719560623169 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 6 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.942307710647583 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.10185185074806213 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.4027317464351654 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 *jet/accuracy_1_of_1‚0.3206828832626343 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6173346638679504 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_3_of_5‚1 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9989176988601685 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 +jet/accuracy_4_of_4‚0.01670061983168125 +particle/sensitivity‚0.6245066523551941 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7118870615959167 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9814814329147339 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5714889764785767 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 particle/accuracy_5_of_5‚0 _step‚725 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.044750090688467026 *jet/accuracy_2_of_3‚0.5273633003234863 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8017635345458984 7validation_average_jet_accuracy‚0.36135831475257874 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.20329393446445465 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9931880235671997 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9490740299224854 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 particle/accuracy_2_of_3‚1 +particle/specificity‚0.9056400060653687 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6155828833580017 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6261848211288452 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.21777257323265076 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.18535132706165314 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6402665376663208 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7431409358978271 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7291699647903442 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 *jet/accuracy_1_of_2‚0.7727466821670532 *jet/accuracy_1_of_4‚0.9664904475212097 particle/accuracy_1_of_4‚1 (particle/accuracy‚0.8452651500701904 particle/accuracy_4_of_5‚1 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6042338609695435 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.11538461595773697 epoch‚21 trainer/global_step‚8799 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.7079840898513794 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5666400194168091 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 +jet/accuracy_3_of_4‚0.08119279146194458 /particle/accuracy_3_of_3‚0.9994120597839355 _runtime‚14602.353943725 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.1775549203157425 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.20560325682163239 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.20383323729038239 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 particle/accuracy_1_of_1‚1 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.4615486264228821 particle/accuracy_1_of_2‚1 particle/accuracy_2_of_2‚1 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.21612809598445892 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6420204639434814 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.22462427616119385 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8557633757591248 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9946339130401611 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5512499809265137 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 )loss/val_total_loss‚8.004491643224068 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6735196113586426 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.48161065578460693 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7363138794898987 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.21917088329792023 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4652777910232544 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.23167158663272858 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.14000001549720764 +jet/accuracy_2_of_2‚0.13757748901844025 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9285714030265808 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6900684833526611 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.36245089769363403 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9955247044563293 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9893755316734314 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 , Purity/0FRt2SRqqt1FBt/FBt_purity‚0.96875 +"Purity/2FRt*SRqqt2FBt/event_purity‚0.75 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.1967507302761078 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.028048869222402573 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2609141767024994 *jet/accuracy_2_of_4‚0.7538572549819946 jet/accuracy_2_of_5‚1 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9972258806228638 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.11538461595773697 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6755783557891846 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8498110175132751 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.06499029695987701 *jet/accuracy_1_of_3‚0.9201108813285828 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3541666865348816 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.23459941148757935 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6396564841270447 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9787510633468628 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6381890177726746 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.17110155522823334 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.052484165877103806 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.33614152669906616 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.44826722145080566 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.18839791417121887 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3100064992904663 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6570641994476318 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.037124957889318466 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 particle/accuracy_2_of_5‚1 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.4004629850387573 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6892880201339722 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 +jet/accuracy_3_of_3‚0.06032029166817665 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.6772167682647705 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.22462427616119385 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.2772207260131836 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.05000000074505806 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6726952195167542 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 jet/accuracy_4_of_5‚0 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.36458051204681396 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8287036418914795 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6700835824012756 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9211474657058716 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9985233545303345 , Purity/0FRt2SRqqt*FBt/FBt_purity‚0.96875 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6570989489555359 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8406273126602173 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9860765337944031 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6653123497962952 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.28202298283576965 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8466614484786987 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.058235783129930496 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9939970374107361 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.32504117488861084 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.17183218896389008 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9985233545303345 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7642977833747864 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6607828140258789 * Purity/2FRt*SRqqt2FBt/FRt_purity‚0.875 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 jet/accuracy_5_of_5‚0 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.05799569562077522 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.18594226241111755 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9980775117874146 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8791576623916626 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6386057734489441 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8287036418914795 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7180794477462769 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 8"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.738322913646698 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 'particle/f_score‚0.6341538429260254 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9961844086647034 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.48143476247787476 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8017635345458984 particle/accuracy_2_of_4‚1 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9962515830993652 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.21614287793636322 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9904571771621704 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6258655786514282 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.4285328686237335 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8486016392707825 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9407699108123779 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.23684072494506836 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.023737803101539612 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.20584510266780853 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 ."Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.40625 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.21164482831954956 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9814814329147339 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 particle/accuracy_1_of_5‚1 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9977869391441345 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.05328382924199104 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8307958245277405 particle/accuracy_3_of_4‚1 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 /particle/accuracy_4_of_4‚0.9529063105583191 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6614583730697632 jet/accuracy_3_of_5‚1 % _timestamp‚1.7843044181360598e+09 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.22151587903499603 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7727135419845581 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.05000000074505806 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7318123579025269 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.046403687447309494 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.8571428656578064 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6441377401351929 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5Á!Õµ¨Ú$ã{ 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.22151587903499603 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7727135419845581 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.05000000074505806 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7318123579025269 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.046403687447309494 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.8571428656578064 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6441377401351929 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5995661020278931 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6155828833580017 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6735625863075256 6"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6755411028862 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.10185185074806213 9"Purity/*FRt2SRqqt*FBt/event_purity‚0.2020145207643509 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.06499029695987701 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8711639642715454 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.014348335564136505 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9951818585395813 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9867116808891296 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.18218374252319336 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.24447476863861084 particle/accuracy_1_of_3‚1 jet/accuracy_1_of_5‚1 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9911719560623169 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 6 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.942307710647583 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.10185185074806213 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.4027317464351654 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 *jet/accuracy_1_of_1‚0.3206828832626343 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6173346638679504 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_3_of_5‚1 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9989176988601685 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 +jet/accuracy_4_of_4‚0.01670061983168125 +particle/sensitivity‚0.6245066523551941 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7118870615959167 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9814814329147339 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5714889764785767 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 particle/accuracy_5_of_5‚0 _step‚725 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.044750090688467026 *jet/accuracy_2_of_3‚0.5273633003234863 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8017635345458984 7validation_average_jet_accuracy‚0.36135831475257874 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.20329393446445465 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9931880235671997 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9490740299224854 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 particle/accuracy_2_of_3‚1 +particle/specificity‚0.9056400060653687 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6155828833580017 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6261848211288452 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.21777257323265076 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.18535132706165314 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6402665376663208 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7431409358978271 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7291699647903442 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 *jet/accuracy_1_of_2‚0.7727466821670532 *jet/accuracy_1_of_4‚0.9664904475212097 particle/accuracy_1_of_4‚1 (particle/accuracy‚0.8452651500701904 particle/accuracy_4_of_5‚1 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6042338609695435 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.11538461595773697 epoch‚21 trainer/global_step‚8799 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.7079840898513794 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5666400194168091 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 +jet/accuracy_3_of_4‚0.08119279146194458 /particle/accuracy_3_of_3‚0.9994120597839355 _runtime‚14602.353943725 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.1775549203157425 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.20560325682163239 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.20383323729038239 @pÅVA&&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 particle/accuracy_1_of_1‚1 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.4615486264228821 particle/accuracy_1_of_2‚1 particle/accuracy_2_of_2‚1 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.21612809598445892 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6420204639434814 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.22462427616119385 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8557633757591248 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9946339130401611 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5512499809265137 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 )loss/val_total_loss‚8.004491643224068 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6735196113586426 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.48161065578460693 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7363138794898987 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.21917088329792023 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4652777910232544 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.23167158663272858 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.14000001549720764 +jet/accuracy_2_of_2‚0.13757748901844025 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9285714030265808 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6900684833526611 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.36245089769363403 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9955247044563293 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9893755316734314 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 , Purity/0FRt2SRqqt1FBt/FBt_purity‚0.96875 +"Purity/2FRt*SRqqt2FBt/event_purity‚0.75 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.1967507302761078 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.028048869222402573 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2609141767024994 *jet/accuracy_2_of_4‚0.7538572549819946 jet/accuracy_2_of_5‚1 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9972258806228638 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.11538461595773697 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6755783557891846 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8498110175132751 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.06499029695987701 *jet/accuracy_1_of_3‚0.9201108813285828 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3541666865348816 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.23459941148757935 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6396564841270447 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9787510633468628 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6381890177726746 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.17110155522823334 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.052484165877103806 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.33614152669906616 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.44826722145080566 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.18839791417121887 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3100064992904663 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6570641994476318 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.037124957889318466 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 particle/accuracy_2_of_5‚1 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.4004629850387573 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6892880201339722 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 +jet/accuracy_3_of_3‚0.06032029166817665 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.6772167682647705 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.22462427616119385 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.2772207260131836 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.05000000074505806 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6726952195167542 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 jet/accuracy_4_of_5‚0 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.36458051204681396 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8287036418914795 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6700835824012756 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9211474657058716 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9985233545303345 , Purity/0FRt2SRqqt*FBt/FBt_purity‚0.96875 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6570989489555359 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8406273126602173 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9860765337944031 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6653123497962952 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.28202298283576965 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8466614484786987 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.058235783129930496 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9939970374107361 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.32504117488861084 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.17183218896389008 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9985233545303345 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7642977833747864 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6607828140258789 * Purity/2FRt*SRqqt2FBt/FRt_purity‚0.875 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 jet/accuracy_5_of_5‚0 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.05799569562077522 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.18594226241111755 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9980775117874146 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8791576623916626 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6386057734489441 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8287036418914795 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7180794477462769 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 8"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.738322913646698 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 'particle/f_score‚0.6341538429260254 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9961844086647034 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.48143476247787476 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8017635345458984 particle/accuracy_2_of_4‚1 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9962515830993652 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.21614287793636322 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9904571771621704 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6258655786514282 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.4285328686237335 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8486016392707825 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9407699108123779 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.23684072494506836 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.023737803101539612 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.20584510266780853 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 ."Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.40625 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.21164482831954956 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9814814329147339 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 particle/accuracy_1_of_5‚1 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9977869391441345 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.05328382924199104 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8307958245277405 particle/accuracy_3_of_4‚1 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 /particle/accuracy_4_of_4‚0.9529063105583191 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6614583730697632 jet/accuracy_3_of_5‚1 % _timestamp‚1.7843044181360598e+09 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 ("Purity/1FRt2SRqqt*FBt/event_purity‚0Õ‹HœYŸÛ$‚(:” ©¦éÒÕžÔö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚284.382' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚56.876400000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ÿ ¼19Ü$‚(:® ©¦éÒ¾Áúø disk.rbd3.out‚0 network.recv‚ 3.650792e+06 disk.sda3.out‚ 764.09375 disk.rbd1.in‚ 0.50830078125) disk./.usagePercent‚37.64524229044575$ disk./.usageGB‚560.9413146972656 disk.dm-1.in‚ 75.29296875 disk.dm-1.out‚ 764.09375 disk.dm-0.in‚ 32716.6640625 disk.sda3.in‚ 75.29296875 disk.rbd3.in‚0 disk.sda2.in‚0 network.sent‚ 1.6880477e+07$ disk.dm-0.out‚133606.98193359375! disk.nvme0n1.in‚ 32716.6640625 disk.sda.in‚ 75.29296875& disk.rbd0.in‚1.759218604441423e+13 disk.sda1.out‚0 proc.cpu.threads‚17 disk.sda1.in‚0 disk.drbd2165.in‚0# disk.nvme0c0n1.in‚ 32716.6640625$ memory_percent‚93.95277500152588# proc.memory.rssMB‚ 2804.86328125* proc.memory.percent‚17.119526863098145 disk.sda2.out‚0 disk.rbd1.out‚ 38.271484375) disk.nvme0c0n1.out‚133606.98193359375' disk.nvme0n1.out‚133606.98193359375 cpu‚25.570464133233262 disk.sda.out‚ 764.09375 disk.rbd0.out‚0 disk.drbd2165.out‚0( proc.memory.availableMB‚ 990.77734375§"3¯“Ý$‚(:ˆ ¸¦éÒÐÙæö gpu.0.gpu‚79.0 gpu.0.memory‚24, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚37.0 gpu.0.powerWatts‚110.125' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.025 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0tîò-Þ$‚(:¢ ¸¦éÒ£óÈù network.sent‚ 1.6938303e+07) disk./.usagePercent‚37.64569311971355 disk.dm-1.in‚ 75.29296875# disk.nvme0c0n1.in‚ 32717.0859375! disk.nvme0n1.in‚ 32717.0859375 disk.sda1.in‚0 disk.rbd3.out‚0 disk.sda2.out‚0 disk.rbd1.out‚ 38.271484375 cpu‚25.570679854582174 disk.sda.in‚ 75.29296875 disk.dm-0.in‚ 32717.0859375 disk.drbd2165.in‚0 disk.rbd3.in‚0# proc.memory.rssMB‚ 2804.86328125$ disk./.usageGB‚560.9480323791504 disk.sda.out‚ 764.2109375& disk.rbd0.in‚1.759218604441423e+13 disk.dm-0.out‚ 133621.65625 disk.sda3.in‚ 75.29296875! disk.nvme0n1.out‚ 133621.65625( proc.memory.availableMB‚ 990.79296875 disk.dm-1.out‚ 764.2109375 disk.rbd1.in‚ 0.50830078125$ memory_percent‚93.95267963409424 proc.cpu.threads‚17 network.recv‚ 3.655449e+06 disk.sda1.out‚0 disk.drbd2165.out‚0 disk.sda2.in‚0* proc.memory.percent‚17.119526863098145 disk.rbd0.out‚0 disk.sda3.out‚ 764.2109375# disk.nvme0c0n1.out‚ 133621.65625– nçžß$‚(:“ ǦéÒ¨þ÷ö gpu.0.gpu‚89.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚269.376' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚53.87519999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0I„»–>à$‚(:³ ǦéÒ¿ÔÀø( disk.nvme0c0n1.out‚133629.8251953125) disk./.usagePercent‚37.64614830110943# proc.memory.rssMB‚ 2804.86328125 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd1.in‚ 0.50830078125 disk.sda.in‚ 75.29296875 disk.sda.out‚ 764.55078125 disk.rbd0.out‚0 disk.sda3.in‚ 75.29296875 disk.drbd2165.out‚0 disk.rbd1.out‚ 38.333984375& disk.rbd0.in‚1.759218604441423e+13 proc.cpu.threads‚17 network.sent‚ 1.6943412e+07$ disk./.usageGB‚560.9548149108887# disk.nvme0c0n1.in‚ 32717.3046875 disk.dm-0.in‚ 32717.3046875# disk.dm-0.out‚133629.8251953125 disk.drbd2165.in‚0& disk.nvme0n1.out‚133629.8251953125( proc.memory.availableMB‚ 990.53515625 disk.sda3.out‚ 764.55078125! disk.nvme0n1.in‚ 32717.3046875 disk.sda2.in‚0 disk.sda2.out‚0$ memory_percent‚93.95425319671631* proc.memory.percent‚17.119526863098145 disk.dm-1.in‚ 75.29296875 disk.dm-1.out‚ 764.55078125 cpu‚25.57079897558233 network.recv‚ 3.659167e+06 disk.rbd3.in‚0 disk.rbd3.out‚0ºié”á$‚(:‰ Ö¦éÒâºñö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚267.733' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.5466 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÅpÞžAâ$‚(:¶ Ö¦éÒ êþø network.sent‚ 1.6948307e+07$ disk./.usageGB‚560.9520606994629 disk.drbd2165.out‚0' disk.nvme0n1.out‚133639.75146484375& disk.rbd0.in‚1.759218604441423e+13( proc.memory.availableMB‚ 990.77734375* proc.memory.percent‚17.119526863098145) disk./.usagePercent‚37.64596346366971) disk.nvme0c0n1.out‚133639.75146484375 disk.rbd1.out‚ 38.333984375 disk.sda.in‚ 75.29296875 disk.dm-0.in‚ 32717.7265625$ disk.dm-0.out‚133639.75146484375 disk.sda3.in‚ 75.29296875 disk.drbd2165.in‚0 disk.sda.out‚ 764.75390625 disk.dm-1.in‚ 75.29296875 cpu‚25.57096330488077 disk.sda3.out‚ 764.75390625 disk.sda2.out‚0$ memory_percent‚93.95277500152588 disk.rbd3.out‚0 disk.sda1.out‚0 disk.rbd1.in‚ 0.50830078125 disk.rbd3.in‚0 disk.dm-1.out‚ 764.75390625# proc.memory.rssMB‚ 2804.86328125# disk.nvme0c0n1.in‚ 32717.7265625 network.recv‚ 3.661553e+06 disk.sda1.in‚0 disk.sda2.in‚0! disk.nvme0n1.in‚ 32717.7265625 disk.rbd0.out‚0 proc.cpu.threads‚17J®8’ã$‚(:‡ å¦éÒ¼ ãö gpu.0.gpu‚99.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚41.0 gpu.0.powerWatts‚251.33' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚50.266 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0û·-d@ä$‚(:µ å¦éÒ…¯—ú disk.dm-0.in‚ 33114.6015625) disk./.usagePercent‚37.64626376050737 disk.sda1.in‚0 disk.rbd1.out‚ 38.458984375 disk.sda.in‚ 75.29296875 disk.dm-1.out‚ 765.07421875$ memory_percent‚93.95308494567871* proc.memory.percent‚17.119526863098145) disk.nvme0c0n1.out‚133652.28564453125! disk.nvme0n1.in‚ 33114.6015625' disk.nvme0n1.out‚133652.28564453125 disk.dm-1.in‚ 75.29296875 proc.cpu.threads‚17 disk.sda3.in‚ 75.29296875 disk.sda3.out‚ 765.07421875 disk.sda2.in‚0& disk.rbd0.in‚1.759218604441423e+13 disk.drbd2165.out‚0 network.sent‚ 1.6951166e+07 disk.drbd2165.in‚0 disk.sda2.out‚0 disk.rbd0.out‚0$ disk.dm-0.out‚133652.28564453125' proc.memory.availableMB‚ 990.7265625# proc.memory.rssMB‚ 2804.86328125 cpu‚25.57115877412504 network.recv‚ 3.664188e+06 disk.sda1.out‚0# disk.nvme0c0n1.in‚ 33114.6015625 disk.rbd1.in‚ 0.50830078125 disk.sda.out‚ 765.07421875$ disk./.usageGB‚560.9565353393555 disk.rbd3.in‚0 disk.rbd3.out‚0¤ØKY4å$/ _wandbruntime‚14680  _runtime‚14680©×–ÌÅæ$¿ _step‚726 % lr-AdamW/pg1‚0.000420248496306627 % lr-AdamW/pg2‚0.000420248496306627 trainer/global_step‚8849 % _timestamp‚1.7843044963428657e+09 _runtime‚14680.56001185gleÊç$Ä trainer/global_step‚8849 % _timestamp‚1.7843044963428657e+09 _runtime‚14680.56001185 _step‚726 % lr-AdamW/pg1‚0.000420248496306627 % lr-AdamW/pg2‚0.000420248496306627Ö$j.ê4è$/ _wandbruntime‚14680  _runtime‚14680c%>(é$€P JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25115896 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚25116144 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10746754282496 % _timestamp‚1.7843044963436384e+09 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2544609555300352 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2551821726470226 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 _step‚727 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚248 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2543510221778944 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2543510247206912 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2544609555300352 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8325752328704 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10746786721792 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚75531603 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 trainer/global_step‚8849 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚22705950 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8311504691282 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _runtime‚14680.561108701 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚75531603 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2552936653018624 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚32439296 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚76652048 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚687004928757760 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2544609529126912 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 400530432 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2551823096498516 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚76652048 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8327097718272 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚253 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚697751683040256 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚687005296848896 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚47821841 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2552935281455616 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8327097718272 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 152183645 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8325752328704 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚75531597 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2544609529126912 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚697752083570688 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚47822094 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚76662021 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚75531597 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2544609529126912 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚697752083570688 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚47822094 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚76662021 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 trainer/global_step‚8849 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚22705950 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8311504691282 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _runtime‚14680.561108701 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚75531603 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2552936653018624 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚32439296 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚76652048 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚687004928757760 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2544609529126912 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 400530432 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2551823096498516 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚76652048 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8327097718272 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚253 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚697751683040256 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚687005296848896 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚47821841 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2552935281455616 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8327097718272 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 152183645 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8325752328704 JDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚76670513 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 152200329 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚76660526 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8312428157528 _runtime‚14681.947296763 _step‚728 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚75539809 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 152210322 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2553215578791424 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 trainer/global_step‚8849 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚75539803 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10747994396672 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8328022760960 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚697830777653248 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚76660526 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚47827134 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 % _timestamp‚1.7843044977298522e+09 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2544887556030464 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8326677364224 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚47827382 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚22708493 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25118646 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2553215578791424 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2553214207221248 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚22708488 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175Œçÿ}'í$N FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚697830777653248 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚76660526 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚47827134 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 % _timestamp‚1.7843044977298522e+09 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2544887556030464 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8326677364224 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚47827382 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚22708493 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25118646 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2553215578791424 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2553214207221248 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚22708488 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚76670513 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 152200329 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚76660526 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8312428157528 _runtime‚14681.947296763 _step‚728 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚75539809 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 152210322 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2553215578791424 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 trainer/global_step‚8849 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚75539803 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10747994396672 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8328022760960 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0ØÜ)ñ14î$/ _wandbruntime‚14682  _runtime‚14682Ö9jÙ ï$… _step‚729 1loss/FRt2/assignment_loss‚0.42482559623010935 2loss/SRqqt2/assignment_loss‚0.3745166860781477 1loss/SRqqt2/detection_loss‚0.6004830433329958 /loss/FBt1/detection_loss‚0.3415922357178316 /loss/FBt2/detection_loss‚0.4629153374646889 %loss/total_loss‚6.522980806387301 epoch‚22 $ _timestamp‚1.784304497815572e+09 0loss/FRt1/assignment_loss‚1.3620677892858168 2loss/SRqqt1/assignment_loss‚0.4071867425779668 3loss/FBt1/assignment_loss‚0.0026295344970500324 2loss/FBt2/assignment_loss‚0.006098882029646564 /loss/FRt1/detection_loss‚0.4984184602749269 1loss/SRqqt1/detection_loss‚0.7409628274358276 trainer/global_step‚8849 .loss/FRt2/detection_loss‚1.301283671462294 _runtime‚14682.032546165Žèþ9ð$Š 2loss/FBt2/assignment_loss‚0.006098882029646564 /loss/FRt1/detection_loss‚0.4984184602749269 1loss/SRqqt1/detection_loss‚0.7409628274358276 trainer/global_step‚8849 .loss/FRt2/detection_loss‚1.301283671462294 _runtime‚14682.032546165 _step‚729 1loss/FRt2/assignment_loss‚0.42482559623010935 2loss/SRqqt2/assignment_loss‚0.3745166860781477 1loss/SRqqt2/detection_loss‚0.6004830433329958 /loss/FBt1/detection_loss‚0.3415922357178316 /loss/FBt2/detection_loss‚0.4629153374646889 %loss/total_loss‚6.522980806387301 epoch‚22 $ _timestamp‚1.784304497815572e+09 0loss/FRt1/assignment_loss‚1.3620677892858168 2loss/SRqqt1/assignment_loss‚0.4071867425779668 3loss/FBt1/assignment_loss‚0.0026295344970500324Ù¼žüh”ñ$‚(:‰ ô¦éÒ§Ù÷ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚263.118' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.6236 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0a3²Ì=ò$‚(:² ô¦éÒÎã¬ú disk.sda3.in‚ 75.29296875 disk.drbd2165.in‚0) disk.nvme0c0n1.out‚133664.43896484375 disk.sda1.out‚0# disk.nvme0c0n1.in‚ 33161.1171875' disk.nvme0n1.out‚133664.43896484375( proc.memory.availableMB‚ 990.20703125 cpu‚25.571289566935572 disk.dm-1.out‚ 765.453125 disk.sda1.in‚0 disk.sda2.in‚0 disk.rbd1.out‚ 38.458984375 network.recv‚ 3.668321e+06 disk.dm-1.in‚ 75.29296875 disk.rbd3.out‚0 disk.rbd1.in‚ 0.50830078125 proc.cpu.threads‚17 network.sent‚ 1.6960021e+07* disk./.usagePercent‚37.646728414182014 disk.rbd0.out‚0$ disk.dm-0.out‚133664.43896484375$ memory_percent‚93.95625591278076# proc.memory.rssMB‚ 2804.86328125& disk.rbd0.in‚1.759218604441423e+13 disk.sda.out‚ 765.453125 disk.sda3.out‚ 765.453125$ disk./.usageGB‚560.9634590148926 disk.dm-0.in‚ 33161.1171875 disk.drbd2165.out‚0 disk.rbd3.in‚0 disk.sda.in‚ 75.29296875* proc.memory.percent‚17.119526863098145! disk.nvme0n1.in‚ 33161.1171875 disk.sda2.out‚0lzw”ó$‚(:‰ ƒ§éÒ‡…Ž÷ gpu.0.gpu‚85.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚338.056' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚67.6112 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0=iÚ=ô$‚(:² ƒ§éÒó»Ãú# proc.memory.rssMB‚ 2804.86328125 network.recv‚ 3.672542e+06& disk.rbd0.in‚1.759218604441423e+13 disk.rbd1.out‚ 38.521484375 cpu‚25.57142004805421$ disk./.usageGB‚560.9822463989258 disk.sda3.in‚ 75.29296875 disk.drbd2165.in‚0 disk.rbd3.in‚0 proc.cpu.threads‚17) disk./.usagePercent‚37.64798925128815 disk.sda1.in‚0 disk.drbd2165.out‚0 disk.sda.in‚ 75.29296875 disk.sda.out‚ 765.76171875 disk.dm-1.in‚ 75.29296875 disk.dm-0.in‚ 33161.5390625 disk.rbd1.in‚ 0.50830078125 network.sent‚ 1.6999872e+07 disk.rbd0.out‚0 disk.sda3.out‚ 765.76171875 disk.rbd3.out‚0$ memory_percent‚93.95642280578613* proc.memory.percent‚17.119526863098145! disk.nvme0n1.in‚ 33161.5390625' proc.memory.availableMB‚ 990.1796875 disk.sda1.out‚0# disk.nvme0c0n1.in‚ 33161.5390625& disk.nvme0n1.out‚133677.2490234375 disk.sda2.in‚0 disk.dm-1.out‚ 765.76171875# disk.dm-0.out‚133677.2490234375( disk.nvme0c0n1.out‚133677.2490234375 disk.sda2.out‚0uÖ#“õ$‚(:ˆ ’§éÒ…÷ž÷ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚265.005' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ŒEQ?ö$‚(:´ ’§éÒýÖùù disk.rbd0.out‚0! disk.nvme0n1.in‚ 33161.7421875' proc.memory.availableMB‚ 990.2421875 proc.cpu.threads‚17' disk.nvme0n1.out‚133687.19091796875 network.sent‚ 1.7062908e+07* disk./.usagePercent‚37.648301324472385$ disk.dm-0.out‚133687.19091796875 disk.drbd2165.out‚0 cpu‚25.571502113858806 disk.dm-0.in‚ 33161.7421875 disk.sda1.out‚0 disk.rbd3.in‚0 disk.rbd3.out‚0) disk.nvme0c0n1.out‚133687.19091796875 disk.sda.in‚ 75.29296875$ memory_percent‚93.95604133605957 disk.sda1.in‚0 disk.sda3.in‚ 75.29296875# disk.nvme0c0n1.in‚ 33161.7421875 disk.sda2.in‚0# proc.memory.rssMB‚ 2804.86328125* proc.memory.percent‚17.119526863098145$ disk./.usageGB‚560.9868965148926 disk.drbd2165.in‚0 disk.rbd1.in‚ 0.50830078125 network.recv‚ 3.677147e+06& disk.rbd0.in‚1.759218604441423e+13 disk.dm-1.in‚ 75.29296875 disk.dm-1.out‚ 765.8828125 disk.sda3.out‚ 765.8828125 disk.sda2.out‚0 disk.rbd1.out‚ 38.521484375 disk.sda.out‚ 765.8828125Ñ_•Œ“÷$‚(:ˆ ¡§éÒãºöö gpu.0.gpu‚88.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚330.385' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚66.077 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‘~Ÿ ø$‚(:¥ ¡§éÒ¸ÿ–ú disk.sda3.out‚ 766.09375$ disk.nvme0n1.out‚133696.41796875 disk.sda1.in‚0 disk.sda1.out‚0$ memory_percent‚93.95534992218018* proc.memory.percent‚17.119526863098145 disk.dm-0.in‚ 33162.1640625$ disk./.usageGB‚560.9961128234863 disk.rbd3.in‚0 disk.sda.out‚ 766.09375 disk.dm-1.out‚ 766.09375 network.recv‚ 3.680625e+06) disk./.usagePercent‚37.64891983867511 disk.rbd1.in‚ 0.50830078125 disk.sda3.in‚ 75.29296875 disk.drbd2165.out‚0ЯHA# disk.rbd0.out‚0# proc.memory.rssMB‚ 2804.86328125( proc.memory.availableMB‚ 990.35546875! disk.nvme0n1.in‚ 33162.1640625 disk.drbd2165.in‚0 disk.rbd3.out‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda.in‚ 75.29296875 disk.dm-1.in‚ 75.29296875& disk.rbd0.in‚1.759218604441423e+13 disk.rbd1.out‚ 38.583984375 cpu‚25.571699767254625 proc.cpu.threads‚17& disk.nvme0c0n1.out‚133696.41796875 network.sent‚ 1.7068117e+07# disk.nvme0c0n1.in‚ 33162.1640625! disk.dm-0.out‚133696.417968752û‰±”ù$‚(:‰ °§éÒåçêö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚255.581' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚51.1162 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0£œ E:ú$‚(:¯ °§éÒ¿Âù$ disk./.usageGB‚561.0005073547363$ disk.nvme0c0n1.in‚33356.27734375$ disk.nvme0n1.out‚133713.37109375 disk.sda2.in‚0 disk.sda2.out‚0! disk.dm-0.out‚133713.37109375# memory_percent‚93.9551591873169" proc.memory.rssMB‚ 2804.8671875" disk.nvme0n1.in‚33356.27734375 disk.rbd3.in‚0 cpu‚25.571815522238523 disk.sda.in‚ 75.29296875 disk.dm-1.in‚ 75.29296875 disk.dm-1.out‚ 766.32421875 disk.sda3.out‚ 766.32421875 disk.rbd3.out‚0 network.recv‚ 3.682969e+06 disk.sda.out‚ 766.32421875 disk.sda1.out‚0) disk./.usagePercent‚37.64921475935456 disk.rbd1.out‚ 38.583984375 disk.sda3.in‚ 75.29296875& disk.rbd0.in‚1.759218604441423e+13 disk.rbd0.out‚0 disk.dm-0.in‚33356.27734375 disk.drbd2165.out‚0 network.sent‚ 1.7070934e+07& disk.nvme0c0n1.out‚133713.37109375 disk.sda1.in‚0 disk.drbd2165.in‚0 disk.rbd1.in‚ 0.50830078125( proc.memory.availableMB‚ 990.38671875* proc.memory.percent‚17.119550704956055 proc.cpu.threads‚17ê!Sžû$‚(:“ ¿§éÒËëÓö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚38.0 gpu.0.powerWatts‚272.542' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚54.50839999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚01–£?ü$‚(:´ ¿§éÒä¼Õú$ disk.dm-0.out‚133721.95263671875$ memory_percent‚93.95537376403809 network.sent‚ 1.7080401e+07 disk.sda3.in‚ 75.29296875' proc.memory.availableMB‚ 990.3515625$ disk./.usageGB‚561.0070610046387 disk.drbd2165.in‚0' disk.nvme0n1.out‚133721.95263671875 network.recv‚ 3.688593e+06 disk.rbd3.in‚0 disk.rbd1.out‚ 38.646484375" proc.memory.rssMB‚ 2804.8671875 disk.rbd3.out‚0 disk.sda2.in‚0 disk.sda.out‚ 766.5859375 disk.sda1.in‚0 disk.sda3.out‚ 766.5859375 cpu‚25.57199213301814) disk./.usagePercent‚37.64965458029838 disk.drbd2165.out‚0$ disk.nvme0c0n1.in‚33356.48046875) disk.nvme0c0n1.out‚133721.95263671875" disk.nvme0n1.in‚33356.48046875& disk.rbd0.in‚1.759218604441423e+13 disk.rbd0.out‚0* proc.memory.percent‚17.119550704956055 disk.sda1.out‚0 proc.cpu.threads‚17 disk.sda2.out‚0 disk.rbd1.in‚ 0.50830078125 disk.sda.in‚ 75.29296875 disk.dm-1.in‚ 75.29296875 disk.dm-1.out‚ 766.5859375 disk.dm-0.in‚33356.48046875o‹}d4ý$/ _wandbruntime‚14760  _runtime‚14760á¢GÊþ$Ä ' lr-AdamW/pg1‚0.00041753855569538024 ' lr-AdamW/pg2‚0.00041753855569538024 trainer/global_step‚8899 % _timestamp‚1.7843045758947058e+09 _runtime‚14760.111905212 _step‚730ÐÝÏÿ$É ' lr-AdamW/pg1‚0.00041753855569538024 ' lr-AdamW/pg2‚0.00041753855569538024 trainer/global_step‚8899 % _timestamp‚1.7843045758947058e+09 _runtime‚14760.111905212 _step‚730Úÿ%å:4€%/ _wandbruntime‚14760  _runtime‚14760€/ä(%€P JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2566883005454848 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 153029676 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚75941903 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚77087773 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10806782693376 trainer/global_step‚8899 _step‚731 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚287 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚22833095 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚25254298 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10806746059776 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25254016 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404724736 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2566883005454848 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚77077800 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 % _timestamp‚1.7843045758957982e+09 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2566881633891840 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚282 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 _runtime‚14760.113163547 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚22833100 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8359086404712 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2565763422520424 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2558509591805952 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8372068259328 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚48087111 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚701686399926272 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8373413648896 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2558509565632512 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚75941903 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36633600 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚77077800 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚77087773 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2558509591805952 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚48087398 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚701685995201536 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚690879249141760 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2566881633891840 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 153019697 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8372068259328 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚75941897 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2558509591805952 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8372068259328 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚48087111 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚701686399926272 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8373413648896 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2558509565632512 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚75941903 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36633600 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚77077800 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚77087773 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2558509591805952 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚48087398 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚701685995201536 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚690879249141760 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2566881633891840 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 153019697 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8372068259328 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚75941897 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2566883005454848 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚77077800 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 % _timestamp‚1.7843045758957982e+09 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2566881633891840 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚282 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 _runtime‚14760.113163547 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚22833100 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8359086404712 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2565763422520424 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995Û„ W4ƒ%/ _wandbruntime‚14761  _runtime‚14761ùiV/‘„%ŠN \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8374338691584 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8374338691584 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2567161931227648 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2558787566362624 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8360009871070 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2557682217893888 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚77086278 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚22835643 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10807968979968 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 153046374 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚701765072620544 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚75950109 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚48092267 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚25587875925¬›M36064 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2558787592536064 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚75950103 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2558787566362624 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10807932353536 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2567160559657472 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 153046374 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0û€'…%N OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚48092541 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 _runtime‚14761.498878421 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚75950109 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 trainer/global_step‚8899 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚690956735549440 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8372993294848 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚690957103640576 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2566040857736556 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2566042227764958 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 153036381 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36626432 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 _step‚732 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2557682217893888 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚77086278 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚22835643 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10807968979968 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 153046374 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚701765072620544 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚75950109 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚48092267 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2558787592536064 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2558787592536064 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚75950103 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2558787566362624 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10807932353536 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2567160559657472 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 153046374 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8374338691584 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8374338691584 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2567161931227648 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2558787566362624 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8360009871070 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 Bƒ5Œ%‚(:ª ݧéÒâ¼÷ø disk.dm-1.out‚767.0625& disk.rbd0.in‚1.759218604441423e+13" disk.nvme0n1.in‚33357.12109375' disk.nvme0n1.out‚133746.73486328125 disk.sda.out‚767.0625 disk.dm-1.in‚ 75.29296875 disk.dm-0.in‚33357.12109375 disk.sda1.in‚0 disk.sda1.out‚0$ memory_percent‚93.95856857299805* proc.memory.percent‚17.119550704956055 network.recv‚ 3.696225e+06 disk.drbd2165.out‚0 disk.rbd1.in‚ 0.50830078125 disk.rbd0.out‚0" proc.memory.rssMB‚ 2804.8671875 proc.cpu.threads‚17 disk.sda3.in‚ 75.29296875 disk.sda2.in‚0 disk.rbd1.out‚ 38.708984375 network.sent‚ 1.7146653e+07$ disk.nvme0c0n1.in‚33357.12109375 disk.sda2.out‚0 disk.sda.in‚ 75.29296875& proc.memory.availableMB‚ 989.828125$ disk.dm-0.out‚133746.73486328125 cpu‚25.57232512598129 disk.sda3.out‚767.0625 disk.rbd3.out‚0) disk./.usagePercent‚37.65041671472781$ disk./.usageGB‚561.0184173583984 disk.drbd2165.in‚0 disk.rbd3.in‚0) disk.nvme0c0n1.out‚133746.73486328125ûÌêu”%‚(:‰ ì§éÒ¯çÔö gpu.0.gpu‚67.0 gpu.0.memory‚21, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚334.874' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚66.9748 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÈnýI=Ž%‚(:² ì§éÒ„†²ù disk.dm-0.in‚ 33643.0546875* proc.memory.percent‚17.119550704956055 network.sent‚ 1.715152e+07 disk.sda2.out‚0 proc.cpu.threads‚17 disk.dm-1.out‚ 767.17578125 disk.drbd2165.in‚0& disk.rbd0.in‚1.759218604441423e+13 disk.rbd0.out‚0 disk.sda3.in‚ 75.29296875 disk.rbd3.out‚0# disk.nvme0c0n1.in‚ 33643.0546875& disk.nvme0n1.out‚133757.4189453125 disk.sda1.out‚0 disk.sda3.out‚ 767.17578125 disk.rbd3.in‚0 disk.sda2.in‚0$ memory_percent‚93.96054744720459# disk.dm-0.out‚133757.4189453125 disk.sda.in‚ 75.29296875 disk.sda1.in‚0 disk.drbd2165.out‚0( disk.nvme0c0n1.out‚133757.4189453125! disk.nvme0n1.in‚ 33643.0546875 disk.rbd1.in‚ 0.50830078125 network.recv‚ 3.699542e+06) disk./.usagePercent‚37.65104572523944$ disk./.usageGB‚561.0277900695801 disk.sda.out‚ 767.17578125( proc.memory.availableMB‚ 989.50390625" proc.memory.rssMB‚ 2804.8671875 disk.rbd1.out‚ 38.708984375 disk.dm-1.in‚ 75.29296875 cpu‚25.572536940724866?÷ß'ž%‚(:“ û§éÒÄ‘§÷ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚272.542' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚54.50839999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‡ªHZ@%‚(:µ û§éÒËáÂú disk.sda.out‚ 768.66796875 network.sent‚ 1.7156727e+07# disk./.usageGB‚561.028938293457& disk.rbd0.in‚1.759218604441423e+13 disk.rbd1.out‚ 38.771484375$ memory_percent‚93.96054744720459 proc.cpu.threads‚17 disk.sda1.out‚0 disk.drbd2165.out‚0) disk.nvme0c0n1.out‚133771.19873046875 disk.sda2.out‚0* proc.memory.percent‚17.119550704956055 disk.sda3.in‚ 75.29296875! disk.nvme0n1.in‚ 33643.2578125 disk.dm-1.out‚ 768.66796875" proc.memory.rssMB‚ 2804.8671875* disk./.usagePercent‚37.651122783507255# disk.nvme0c0n1.in‚ 33643.2578125 disk.rbd1.in‚ 0.50830078125( proc.memory.availableMB‚ 989.50390625 network.recv‚ 3.703367e+06 disk.rbd3.in‚0' disk.nvme0n1.out‚133771.19873046875 disk.sda2.in‚0 cpu‚25.57264487554943 disk.sda1.in‚0 disk.sda3.out‚ 768.66796875 disk.drbd2165.in‚0 disk.sda.in‚ 75.29296875 disk.dm-1.in‚ 75.29296875 disk.rbd0.out‚0 disk.dm-0.in‚ 33643.2578125$ disk.dm-0.out‚133771.19873046875 disk.rbd3.out‚0`l ”‘%‚(:‰ Š¨éÒ䂈÷ gpu.0.gpu‚70.0 gpu.0.memory‚23, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚340.229' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚68.0458 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Á¡Õ.’%‚(:£ Š¨éÒÖüÓù network.sent‚ 1.7161909e+07 network.recv‚ 3.70651e+06 disk.rbd1.out‚ 38.771484375 disk.sda.out‚ 770.203125 disk.dm-1.in‚ 75.29296875 disk.sda1.in‚0 disk.dm-0.out‚133788.6640625# disk.nvme0n1.out‚133788.6640625 disk.sda2.in‚0$ memory_percent‚93.96061897277832' proc.memory.availableMB‚ 989.4921875 disk.rbd3.in‚0! disk.nvme0n1.in‚ 33643.4765625" proc.memory.rssMB‚ 2804.8671875* disk./.usagePercent‚37.651583597068885$ disk./.usageGB‚561.0358047485352 disk.rbd1.in‚ 0.50830078125 disk.dm-1.out‚ 770.203125 disk.sda1.out‚0 disk.sda3.out‚ 770.203125 cpu‚25.572843804947457& disk.rbd0.in‚1.759218604441423e+13 disk.rbd0.out‚0* proc.memory.percent‚17.119550704956055 proc.cpu.threads‚17 disk.drbd2165.out‚0# disk.nvme0c0n1.in‚ 33643.4765625 disk.sda2.out‚0 disk.dm-0.in‚ 33643.4765625 disk.sda.in‚ 75.29296875 disk.drbd2165.in‚0 disk.rbd3.out‚0% disk.nvme0c0n1.out‚133788.6640625 disk.sda3.in‚ 75.29296875Œ›u"4“%/ _wandbruntime‚14839  _runtime‚14839¼”dÇ”%Á _step‚734 & lr-AdamW/pg1‚0.0004148240154295558 & lr-AdamW/pg2‚0.0004148240154295558 trainer/global_step‚8949 $ _timestamp‚1.784304655189783e+09 _runtime‚14839.406948944ìÓÜéÌ•%Æ & lr-AdamW/pg1‚0.0004148240154295558 & lr-AdamW/pg2‚0.0004148240154295558 trainer/global_step‚8949 $ _timestamp‚1.784304655189783e+09 _runtime‚14839.406948944 _step‚734ÞѪRv4–%/ _wandbruntime‚14839  _runtime‚14839?õû(—%€P RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25389890 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2580827986328064 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚77513525 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚278 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚25390168 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚77503552 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚22960245 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚76352197 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚705620088008704 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8405323517820 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 % _timestamp‚1.7843046551908805e+09 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8418384189952 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚694753937616896 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2571298399596544 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚48350135 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2571298425024512 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚77503552 _step‚735 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8418384189952 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚76352197 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36633600 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2579703748542332 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8419729579520 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚76352203 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2572409602138112 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 trainer/global_step‚8949 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚283 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2572409628311552 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2572409602138112 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚22960250 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8419729579520 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚76352203 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2572409602138112 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 trainer/global_step‚8949 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚283 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2572409628311552 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2572409602138112 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚22960250 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2580827986328064 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚77513525 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚278 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚25390168 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚77503552 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚22960245 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚76352197 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚705620088008704 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8405323517820 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 % _timestamp‚1.7843046551908805e+09 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8418384189952 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚694753937616896ßýcWc4™%/ _wandbruntime‚14840  _runtime‚14840I²q²'š%ŠN ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚705698754123776 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 153872433 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2581108283663872 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚278 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8404902383744 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 153882426 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2571576281374720 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚76360409 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚76360403 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚22962793 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚705698349406208 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2579982553786866 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 _runtime‚14840.796463862 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2581108283663872 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚76360409 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 % _timestamp‚1.7843046565790498e+09 _step‚736 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8419309225472 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8419309225472 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚76360403 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚22962788 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25392855 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚77522017 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2572687629041664 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 153882426 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10867293472768 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8406246984178 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10867330099200 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2579981183758464 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8420654622208 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2572687602868224 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 153872433 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚48355370 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚273 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚77522017 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2571576306802688 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25392582 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2572687602868224 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚694831424024576 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36626432 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2581106912093696 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚694831055933440 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚8949 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2572687629041664 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚48355648±–Û¤'›%N I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚705698754123776 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 153872433 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2581108283663872 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚278 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8404902383744 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 153882426 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2571576281374720 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚76360409 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚76360403 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚22962793 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚705698349406208 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2579982553786866 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 _runtime‚14840.796463862 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2581108283663872 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚76360409 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 % _timestamp‚1.7843046565790498e+09 _step‚736 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8419309225472 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8419309225472 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚76360403 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚22962788 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25392855 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚77522017 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2572687629041664 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 153882426 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10867293472768 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8406246984178 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10867330099200 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2579981183758464 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8420654622208 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2572687602868224 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 153872433 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚48355370 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚273 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚77522017 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2571576306802688 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25392582 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2572687602868224 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚694831424024576 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36626432 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2581106912093696 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚694831055933440 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚8949 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2572687629041664 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚48355648 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736à Rp74œ%/ _wandbruntime‚14840  _runtime‚14840 ^Øb %„ epoch‚22 3loss/FBt1/assignment_loss‚0.0011562633603942273 1loss/FBt2/assignment_loss‚0.00332030173616135 1loss/SRqqt2/detection_loss‚0.5222106272207252 trainer/global_step‚8949 _runtime‚14840.879825032 2loss/SRqqt1/assignment_loss‚0.4351990391729239 % _timestamp‚1.7843046566628666e+09 _step‚737 /loss/FBt2/detection_loss‚0.5944755963011024 %loss/total_loss‚6.700228997230722 /loss/FRt1/assignment_loss‚1.352765055643412 0loss/FRt1/detection_loss‚0.48394948635851215 .loss/FRt2/detection_loss‚1.456999500458532 1loss/SRqqt1/detection_loss‚0.7474990851325177 .loss/FBt1/detection_loss‚0.289011403277982 0loss/FRt2/assignment_loss‚0.4493674128940425 3loss/SRqqt2/assignment_loss‚0.36427522567441656Ë•®•ž%‰ /loss/FBt2/detection_loss‚0.5944755963011024 %loss/total_loss‚6.700228997230722 /loss/FRt1/assignment_loss‚1.352765055643412 0loss/FRt1/detection_loss‚0.48394948635851215 .loss/FRt2/detection_loss‚1.456999500458532 1loss/SRqqt1/detection_loss‚0.7474990851325177 .loss/FBt1/detection_loss‚0.289011403277982 0loss/FRt2/assignment_loss‚0.4493674128940425 3loss/SRqqt2/assignment_loss‚0.36427522567441656 epoch‚22 3loss/FBt1/assignment_loss‚0.0011562633603942273 1loss/FBt2/assignment_loss‚0.00332030173616135 1loss/SRqqt2/detection_loss‚0.5222106272207252 trainer/global_step‚8949 _runtime‚14840.879825032 2loss/SRqqt1/assignment_loss‚0.4351990391729239 % _timestamp‚1.7843046566628666e+09 _step‚737á"¤kc”Ÿ%‚(:‰ ™¨éÒ§šáö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚270.943' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.1886 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0h9+Z: %‚(:¯ ™¨éÒÓĪù# disk.nvme0c0n1.in‚ 33643.8984375' disk.nvme0c0n1.out‚133802.326171875 disk.rbd1.in‚ 0.50830078125$ memory_percent‚93.95909309387207" proc.memory.rssMB‚ 2804.8671875* proc.memory.percent‚17.119550704956055 network.sent‚ 1.7166984e+07 disk.sda.in‚ 75.29296875 disk.dm-1.in‚ 75.29296875 disk.dm-0.in‚ 33643.8984375 disk.rbd3.out‚0! disk.nvme0n1.in‚ 33643.8984375' proc.memory.availableMB‚ 989.7421875& disk.rbd0.in‚1.759218604441423e+13 disk.rbd1.out‚ 38.833984375 disk.sda.out‚ 770.70703125 disk.sda1.out‚0 disk.sda3.in‚ 75.29296875 disk.sda3.out‚ 770.70703125 disk.drbd2165.out‚0 disk.sda2.in‚0 cpu‚25.572939644614014 network.recv‚ 3.709786e+06$ disk./.usageGB‚561.0426177978516 proc.cpu.threads‚17 disk.drbd2165.in‚0% disk.nvme0n1.out‚133802.326171875) disk./.usagePercent‚37.65204082652504 disk.dm-1.out‚ 770.70703125" disk.dm-0.out‚133802.326171875 disk.rbd3.in‚0 disk.sda2.out‚0 disk.rbd0.out‚0 disk.sda1.in‚0îç=«¡%‚(:‰ ¨¨éÒûúÌö gpu.0.gpu‚99.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚¡€Þ=é44.0 gpu.0.powerWatts‚345.994' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚69.1988 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚05æÂ)¢%‚(:ž ¨¨éÒÁ¡ú network.recv‚ 3.714749e+06& disk.rbd0.in‚1.759218604441423e+13 disk.sda3.in‚ 75.29296875 disk.sda3.out‚ 771.2109375 disk.sda2.out‚0$ disk./.usageGB‚561.0488166809082 disk.dm-0.out‚133814.3515625 proc.cpu.threads‚17 network.sent‚ 1.722965e+07 disk.rbd0.out‚0 disk.sda1.in‚0 disk.drbd2165.in‚0 disk.drbd2165.out‚0$ memory_percent‚93.95990371704102) disk./.usagePercent‚37.65245683876818% disk.nvme0c0n1.out‚133814.3515625# disk.nvme0n1.out‚133814.3515625& proc.memory.availableMB‚ 989.609375" proc.memory.rssMB‚ 2804.8671875* proc.memory.percent‚17.119550704956055 cpu‚25.573131454504843 disk.sda.in‚ 75.29296875 disk.sda.out‚ 771.2109375 disk.dm-1.in‚ 75.29296875 disk.dm-0.in‚ 33931.21875 disk.rbd3.in‚0 disk.rbd1.in‚ 0.50830078125 disk.rbd1.out‚ 38.833984375 disk.rbd3.out‚0 disk.sda2.in‚0 disk.sda1.out‚0 disk.dm-1.out‚ 771.2109375! disk.nvme0c0n1.in‚ 33931.21875 disk.nvme0n1.in‚ 33931.21875תI¾ž£%‚(:“ ·¨éÒßš›ø gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚281.143' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚56.22859999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Lºèˆ9¤%‚(:® ·¨éÒ‡ïüø disk.sda3.in‚ 75.29296875 disk.drbd2165.out‚0 network.recv‚ 3.718288e+06 disk.dm-1.out‚ 771.9453125' disk.nvme0n1.out‚133822.75537109375 disk.rbd0.out‚0 cpu‚25.573213437281552& disk.rbd0.in‚1.759218604441423e+13 disk.rbd3.out‚0 disk.sda2.out‚0 disk.rbd1.out‚ 38.896484375& proc.memory.availableMB‚ 989.796875$ disk.dm-0.out‚133822.75537109375 disk.sda2.in‚0 disk.rbd1.in‚ 0.50830078125 proc.cpu.threads‚17 network.sent‚ 1.7235251e+07 disk.sda3.out‚ 771.9453125) disk.nvme0c0n1.out‚133822.75537109375 disk.nvme0n1.in‚ 33931.640625" proc.memory.rssMB‚ 2804.8671875$ disk./.usageGB‚561.0544738769531 disk.sda.in‚ 75.29296875 disk.sda1.in‚0 disk.rbd3.in‚0$ memory_percent‚93.95875930786133* proc.memory.percent‚17.119550704956055 disk.dm-1.in‚ 75.29296875 disk.sda1.out‚0 disk.drbd2165.in‚0" disk.nvme0c0n1.in‚ 33931.640625) disk./.usagePercent‚37.65283649794146 disk.sda.out‚ 771.9453125 disk.dm-0.in‚ 33931.640625Èl§Šž¥%‚(:“ ƨéÒæùçö gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚292.743' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚58.54859999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Õ½f7¦%‚(:¬ ƨéÒ²¾ºù disk.sda.out‚ 772.234375$ memory_percent‚93.96028518676758 cpu‚25.573422953883547$ disk./.usageGB‚561.0584716796875& disk.rbd0.in‚1.759218604441423e+13 disk.rbd0.out‚0 disk.dm-1.out‚ 772.234375* disk./.usagePercent‚37.653104793837336 disk.sda2.in‚0& proc.memory.availableMB‚ 989.546875* proc.memory.percent‚17.119550704956055 disk.sda1.in‚0 disk.sda3.out‚ 772.234375 disk.drbd2165.in‚0 disk.rbd3.in‚0 disk.nvme0n1.in‚ 33931.859375' disk.nvme0n1.out‚133838.39013671875 disk.dm-1.in‚ 75.29296875 network.sent‚ 1.7239813e+07$ disk.dm-0.out‚133838.39013671875" disk.nvme0c0n1.in‚ 33931.859375 disk.rbd1.out‚ 38.896484375 proc.cpu.threads‚17 disk.drbd2165.out‚0 disk.rbd3.out‚0 disk.rbd1.in‚ 0.50830078125 disk.sda.in‚ 75.29296875 network.recv‚ 3.721585e+06 disk.sda1.out‚0) disk.nvme0c0n1.out‚133838.39013671875" proc.memory.rssMB‚ 2804.8671875 disk.dm-0.in‚ 33931.859375 disk.sda3.in‚ 75.29296875 disk.sda2.out‚0?…!KŸ§%‚(:” Õ¨éÒœ±îö gpu.0.gpu‚90.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚275.038' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚55.007600000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0„Ö/k.¨%‚(:£ Õ¨éÒ—¿Ãù) disk./.usagePercent‚37.65357175157979! disk./.usageGB‚561.0654296875 disk.rbd3.out‚0 disk.drbd2165.in‚0# disk.nvme0n1.out‚133846.1171875 disk.sda2.out‚0 disk.dm-0.out‚133846.1171875$ memory_percent‚93.96028518676758" proc.memory.rssMB‚ 2804.8671875 cpu‚25.57358333313168 disk.sda1.in‚0 disk.rbd1.in‚ 0.50830078125& disk.rbd0.in‚1.759218604441423e+13 disk.dm-0.in‚33932.07421875* proc.memory.percent‚17.119550704956055 disk.drbd2165.out‚0$ disk.nvme0c0n1.in‚33932.07421875" disk.nvme0n1.in‚33932.07421875 disk.rbd1.out‚ 38.958984375 disk.dm-1.out‚ 772.7265625 network.recv‚ 3.72514e+06 disk.sda3.out‚ 772.7265625 disk.sda.out‚ 772.7265625 disk.dm-1.in‚ 75.29296875 proc.cpu.threads‚17 network.sent‚ 1.7244889e+07 disk.sda3.in‚ 75.29296875 disk.rbd3.in‚0 disk.sda2.in‚0 disk.rbd0.out‚0% disk.nvme0c0n1.out‚133846.1171875 disk.sda.in‚ 75.29296875& proc.memory.availableMB‚ 989.546875 disk.sda1.out‚0Òc-Ú4©%/ _wandbruntime‚14918  _runtime‚14918M¨¦Êª%Ä _runtime‚14918.892045284 _step‚738 ' lr-AdamW/pg1‚0.00041210504985900236 ' lr-AdamW/pg2‚0.00041210504985900236 trainer/global_step‚8999 % _timestamp‚1.7843047346748116e+09A‹+†Ï«%É ' lr-AdamW/pg2‚0.00041210504985900236 trainer/global_step‚8999 % _timestamp‚1.7843047346748116e+09 _runtime‚14918.892045284 _step‚738 ' lr-AdamW/pg1‚0.00041210504985900236âR Ž4¬%/ _wandbruntime‚14918  _runtime‚14918wÜ$J(­%þO @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2593642704535950 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚76762503 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚698627889909760 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10925494247424 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚25526428 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚77939277 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚23087400 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2594775710327296 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚241 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2594775710327296 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2586309638643712 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 154701780 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 154691801 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8451560630928 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚76762497 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚76762503 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚76762497 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2594774338764288 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2593644074564240 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2585192488505344 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚77929304 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2586309664817152 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25526192 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 _step‚739 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 $ _timestamp‚1.784304734676019e+09 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2594774338764288 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚48613587 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚23087395 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚709553752248320 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚236 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _runtime‚14918.89354178 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚77929304 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚48613828 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2586309638643712 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2586309664817152 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10925461808128 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8466045510144 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚32439296Ñ^½ (®%ƒP VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2594774338764288 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2593644074564240 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2585192488505344 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚77929304 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2586309664817152 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25526192 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 _step‚739 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 $ _timestamp‚1.784304734676019e+09 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2594774338764288 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚48613587 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚23087395 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚709553752248320 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚236 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _runtime‚14918.89354178 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚77929304 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚48613828 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2586309638643712 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2586309664817152 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10925461808128 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8466045510144 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚32439296 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2593642704535950 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚76762503 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚698627889909760 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10925494247424 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚25526428 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚77939277 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚23087400 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2594775710327296 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚241 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2594775710327296 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2586309638643712 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 154701780 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 154691801 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8451560630928 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚76762497 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚76762503 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚76762497 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290ã¼?D24¯%/ _wandbruntime‚14920  _runtime‚14920VåCßð%ŠN XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8465625156096 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚23089938 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10926664674304 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 trainer/global_step‚8999 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚77937782 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚240 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚76770703 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8465625156096 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 _runtime‚14920.281416457 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 154718478 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 154708485 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 154708485 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚77937782 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2585470395711488 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 154718478 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25528935 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2586587639373824 C9DeviceSt¾›×˜MatsMonitor.on_train_batch_end/allocation.all.peak‚12154 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2586587665547264 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚709632040991744 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚48618873 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚698705744408576 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8451139496852 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2586587639373824 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8466970552832 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚77947769 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10926697106432 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2593922879808774 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2595054636100096 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25529175 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚76770703 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚245 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚23089938 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10926664674304 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 trainer/global_step‚8999 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚77937782 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚240 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚76770703 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8465625156096 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 _runtime‚14920.281416457 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 154718478 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 154708485 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 154708485 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚77937782 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2585470395711488 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 154718478 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25528935 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2586587639373824 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2586587665547264 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚709632040991744 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚48618873 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚698705744408576 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8451139496852 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2586587639373824 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8466970552832 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚77947769 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10926697106432 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2593922879808774 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2595054636100096 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25529175 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚76770703 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚245 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2607584400586148 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚272 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2607583030557858 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚23214550 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2599086577414144 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚9049 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚702502578384896 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10985661085696 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚78355056 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10985695622144 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 155537832 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚25663295 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2600209675149312 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8497797744036 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚713487871379456 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 _runtime‚14998.143180893 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8511016051200 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2608720691200512 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2600209701322752 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25663028 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2608722062763520 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚77172797 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚23214545 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2600209675149312 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 E?DeviceStatsMonitor.on_train_batch_start/oversize_segmentDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2599086602842112 % _timestamp‚1.7843048139258242e+09Lèa1 (Ä%…P JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚77172797 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚23214545 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2600209675149312 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚77172797 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8511016051200 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚78355056 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 155527853 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚713488274007040 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8496453143714 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2599086602842112 % _timestamp‚1.7843048139258242e+09 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2600209701322752 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚48877845 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚78365029 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8512361440768 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚78365029 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚702502210293760 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚48877573 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2607584400586148 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚272 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2607583030557858 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚23214550 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2599086577414144 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚9049 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚702502578384896 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10985661085696 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚78355056 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10985695622144 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 155537832 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚25663295 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2600209675149312 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8497797744036 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚713487871379456 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 _runtime‚14998.143180893 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8511016051200 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2608720691200512 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2600209701322752 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25663028 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2608722062763520 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584çÛ9 F4Å%/ _wandbruntime‚14999  _runtime‚14999ÁŒ¥'Æ%ŠN D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚78373521 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2609000988536320 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚23217088 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚252 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚702580064792576 _runtime‚14999.530912009 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8513286483456 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 155544537 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8498721210394 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 155554530 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10986859265024 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8497376609960 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25665791 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚48882879 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚77181009 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚77181003 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2607863205830682 trainer/global_step‚9049 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 _step‚744 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚23217093 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25666043 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2600487702052864 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2600487675879424 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7843048153134258e+09 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚78363534 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2600487702052864 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚78363534 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚713566958586880 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚702579696701440 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2600487675879424 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚77181009 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2599364484620288 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8513286483456 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8511941086720 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154L- 'Ç%N OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚252 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚702580064792576 _runtime‚14999.530912009 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8513286483456 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 155544537 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8498721210394 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 155554530 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10986859265024 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8497376609960 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25665791 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚48882879 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚77181009 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚77181003 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2607863205830682 trainer/global_step‚9049 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 _step‚744 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚23217093 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25666043 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2600487702052864 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2600487675879424 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7843048153134258e+09 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚78363534 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2600487702052864 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚78363534 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚713566958586880 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚702579696701440 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2600487675879424 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚77181009 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2599364484620288 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8513286483456 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8511941086720 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚78373521 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2609000988536320 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚23217088è[‹lõ4È%/ _wandbruntime‚14999  _runtime‚14999ªŸ1)É%‰ %loss/total_loss‚6.749262086228513 0loss/FRt1/assignment_loss‚1.3933082280037807 /loss/FRt2/detection_loss‚1.4841806279168328 0loss/FBt1/detection_loss‚0.42080152574728463 0loss/FBt2/detection_loss‚0.44312381210523505 epoch‚22 2loss/SRqqt1/assignment_loss‚0.4098753536897096 1loss/FBt1/assignment_loss‚0.05853892970039377 3loss/FBt2/assignment_loss‚0.0009788973533504238 1loss/SRqqt1/detection_loss‚0.7291130195238537 trainer/global_step‚9049 0loss/FRt2/assignment_loss‚0.4116952188749046 1loss/SRqqt2/detection_loss‚0.5727819879762539 % _timestamp‚1.7843048153973582e+09 _runtime‚14999.614314613 _step‚745 3loss/SRqqt2/assignment_loss‚0.34184020125104514 0loss/FRt1/detection_loss‚0.48302428408586856°é,%ÊÊ%Ž epoch‚22 2loss/SRqqt1/assignment_loss‚0.4098753536897096 1loss/FBt1/assignment_loss‚0.05853892970039377 3loss/FBt2/assignment_loss‚0.0009788973533504238 1loss/SRqqt1/detection_mwý»Jloss‚0.7291130195238537 trainer/global_step‚9049 0loss/FRt2/assignment_loss‚0.4116952188749046 1loss/SRqqt2/detection_loss‚0.5727819879762539 % _timestamp‚1.7843048153973582e+09 _runtime‚14999.614314613 _step‚745 3loss/SRqqt2/assignment_loss‚0.34184020125104514 0loss/FRt1/detection_loss‚0.48302428408586856 %loss/total_loss‚6.749262086228513 0loss/FRt1/assignment_loss‚1.3933082280037807 /loss/FRt2/detection_loss‚1.4841806279168328 0loss/FBt1/detection_loss‚0.42080152574728463 0loss/FBt2/detection_loss‚0.44312381210523505é{5ØœË%‚(:‘ ¯©éÒœ±íö gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚337.288' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚67.45760000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0”E ‡?Ì%‚(:´ ¯©éÒõŸÊù disk.sda2.out‚0 disk.rbd1.in‚ 0.50830078125 disk.sda.in‚ 75.29296875$ disk.dm-0.out‚133911.00830078125 disk.sda1.in‚0( proc.memory.availableMB‚ 990.47265625 disk.sda3.out‚ 774.5859375 disk.drbd2165.out‚0) disk./.usagePercent‚37.65629541573657# disk./.usageGB‚561.106014251709" disk.nvme0n1.in‚34305.23046875 disk.sda.out‚ 774.5859375 disk.dm-1.in‚ 75.29296875 proc.cpu.threads‚17 network.recv‚ 3.74677e+06) disk.nvme0c0n1.out‚133911.00830078125 disk.sda2.in‚0$ memory_percent‚93.95463466644287* proc.memory.percent‚17.119550704956055 disk.rbd3.in‚0' disk.nvme0n1.out‚133911.00830078125 disk.sda1.out‚0" proc.memory.rssMB‚ 2804.8671875& disk.rbd0.in‚1.759218604441423e+13 disk.dm-0.in‚34305.23046875 disk.rbd3.out‚0$ disk.nvme0c0n1.in‚34305.23046875 disk.rbd1.out‚ 39.208984375 disk.dm-1.out‚ 774.5859375 cpu‚25.574440474026158 disk.rbd0.out‚0 disk.drbd2165.in‚0 network.sent‚ 1.7333609e+07 disk.sda3.in‚ 75.29296875½D ¥ŸÍ%‚(:” ¾©éÒÍíö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚284.382' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚56.876400000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0YÃÆ3<Î%‚(:± ¾©éÒÓÑœú cpu‚25.574546714065068 disk.sda.in‚ 75.29296875" disk.dm-0.out‚133922.708984375 disk.sda1.in‚0& proc.memory.availableMB‚ 990.515625 network.recv‚ 3.751038e+06 disk.rbd1.in‚ 0.50830078125 disk.rbd0.out‚0 disk.drbd2165.out‚0" proc.memory.rssMB‚ 2804.8671875* proc.memory.percent‚17.119550704956055 disk.dm-1.out‚ 774.99609375 disk.rbd3.in‚0$ memory_percent‚93.95437240600586* disk./.usagePercent‚37.656738820785876# disk./.usageGB‚561.112621307373 disk.sda2.out‚0 disk.sda2.in‚0 network.sent‚ 1.7396709e+07 disk.rbd1.out‚ 39.208984375 disk.sda1.out‚0 disk.sda3.in‚ 75.29296875 disk.rbd3.out‚0$ disk.nvme0c0n1.in‚34305.43359375 proc.cpu.threads‚17' disk.nvme0c0n1.out‚133922.708984375" disk.nvme0n1.in‚34305.43359375 disk.dm-1.in‚ 75.29296875 disk.dm-0.in‚34305.43359375 disk.sda3.out‚ 774.99609375 disk.sda.out‚ 774.99609375& disk.rbd0.in‚1.759218604441423e+13 disk.drbd2165.in‚0% disk.nvme0n1.out‚133922.708984375‘z䌞Ï%‚(:“ Í©éÒ ¬ù gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚300.808' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚60.16160000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ä” »BÐ%‚(:· Í©éÒÄâáú cpu‚25.574702997317146 disk.sda2.out‚0 disk.sda3.in‚ 75.29296875' proc.memory.availableMB‚ 990.7421875 proc.cpu.threads‚17$ disk.nvme0c0n1.in‚34305.85546875) disk.nvme0c0n1.out‚133936.64111328125 disk.rbd1.in‚ 0.50830078125 disk.sda.out‚ 775.23828125 network.sent‚ 1.740173e+07 network.recv‚ 3.75469e+06* disk./.usagePercent‚37.657218578905045$ disk./.usageGB‚561.1197700500488 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.dm-1.in‚ 75.29296875& disk.rbd0.in‚1.759218604441423e+13 disk.drbd2165.in‚0 disk.drbd2165.out‚0' disk.nvme0n1.out‚133936.64111328125 disk.dm-0.in‚34305.85546875$ disk.dm-0.out‚133936.64111328125 disk.sda1.in‚0 disk.sda1.out‚0* proc.memory.percent‚17.119550704956055" disk.nvme0n1.in‚34305.85546875" proc.memory.rssMB‚ 2804.8671875 disk.sda2.in‚0 disk.rbd0.out‚0 disk.rbd1.out‚ 39.271484375 disk.sda.in‚ 75.29296875 disk.dm-1.out‚ 775.23828125 disk.sda3.out‚ 775.23828125$ memory_percent‚93.95298957824707…'Ó¶“Ñ%‚(:ˆ Ü©éÒ‰…Á÷ gpu.0.gpu‚90.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚261.005' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.201 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0×½øÝ;Ò%‚(:° Ü©éÒŸŠù disk.sda1.out‚0 disk.rbd3.out‚0) disk.nvme0c0n1.out‚133947.06298828125* proc.memory.percent‚17.119550704956055$ disk.dm-0.out‚133947.06298828125 disk.sda3.out‚ 775.59375 disk.sda3.in‚ 75.29296875 disk.drbd2165.in‚0 disk.dm-1.out‚ 775.59375 network.recv‚ 3.757608e+06 disk.dm-0.in‚34306.07421875$ disk.nvme0c0n1.in‚34306.07421875 network.sent‚ 1.7406269e+07& disk.rbd0.in‚1.759218604441423e+13 disk.sda1.in‚0 disk.rbd3.in‚0 disk.sda.in‚ 75.29296875$ memory_percent‚93.95458698272705" proc.memory.rssMB‚ 2804.8671875) disk./.usagePercent‚37.65751708368997" disk.nvme0n1.in‚34306.07421875 disk.dm-1.in‚ 75.29296875 cpu‚25.574900019403483 disk.rbd0.out‚0 disk.drbd2165.out‚0' disk.nvme0n1.out‚133947.06298828125 disk.rbd1.in‚ 0.50830078125 disk.rbd1.out‚ 39.271484375 disk.sda.out‚ 775.59375( proc.memory.availableMB‚ 990.48046875 proc.cpu.threads‚17$ disk./.usageGB‚561.1242179870605 disk.sda2.in‚0 disk.sda2.out‚0Ñé &žÓ%‚(:“ ë©éÒŽûÚø gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚282.255' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚56.45099999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0$­{9Ô%‚(:® ë©éÒ«…Èù disk.dm-1.out‚ 775.94921875 disk.dm-0.in‚ 34436.96875 disk.sda1.in‚0 disk.sda3.in‚ 75.29296875 disk.drbd2165.in‚0 disk.rbd1.in‚ 0.50830078125 disk.sda1.out‚0 disk.rbd3.out‚0 disk.sda.out‚ 775.94921875$ disk./.usageGB‚561.1357955932617 disk.sda3.out‚ 775.94921875 disk.drbd2165.out‚0 disk.rbd0.out‚0 network.recv‚ 3.761133e+06) disk.nvme0c0n1.out‚133955.28564453125 disk.sda2.in‚0 disk.sda.in‚ 75.29296875 proc.cpu.threads‚17 network.sent‚ 1.7411882e+07' disk.nvme0n1.out‚133955.28564453125$ memory_percent‚93.95151138305664" proc.memory.rssMB‚ 2804.8671875 cpu‚25.575022753971794 disk.sda2.out‚0 disk.dm-1.in‚ 75.29296875$ disk.dm-0.out‚133955.28564453125* proc.memory.percent‚17.119550704956055 disk.rbd3.in‚0! disk.nvme0c0n1.in‚ 34436.96875 disk.nvme0n1.in‚ 34436.96875& disk.rbd0.in‚1.759218604441423e+13& proc.memory.availableMB‚ 990.984375) disk./.usagePercent‚37.65829406655639 disk.rbd1.out‚ 39.333984375@›ÀIŸÕ%‚(:” ú©éÒóûø gpu.0.gpu‚100.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚46.0 gpu.0.powerWatts‚334.977' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚66.99539999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¬6œ²;Ö%‚(:° ú©éÒî´€ú$ memory_percent‚93.95453929901123 disk.dm-1.in‚ 75.29296875 disk.rbd0.out‚0 disk.sda1.in‚0 disk.sda2.in‚0 disk.rbd1.in‚ 0.50830078125* proc.memory.percent‚17.119550704956055 disk.sda3.in‚ 75.29296875 disk.nvme0n1.in‚ 34437.171875 disk.sda2.out‚0 network.sent‚ 1.7416462e+07* disk./.usagePercent‚37.658572858761175 cpu‚25.575145315059743 disk.sda.in‚ 75.29296875 disk.dm-0.in‚ 34437.171875# disk.dm-0.out‚133972.5986328125" disk.nvme0c0n1.in‚ 34437.171875& disk.nvme0n1.out‚133972.5986328125 disk.rbd1.out‚ 39.333984375& disk.rbd0.in‚1.759218604441423e+13 disk.rbd3.out‚0 proc.cpu.threads‚17 network.recv‚ 3.764387e+06 disk.sda1.out‚0 disk.sda3.out‚ 776.12890625 disk.drbd2165.out‚0 disk.rbd3.in‚0( proc.memory.availableMB‚ 990.48828125" proc.memory.rssMB‚ 2804.8671875# disk./.usageGB‚561.139949798584 disk.dm-1.out‚ 776.12890625 disk.sda.out‚ 776.12890625 disk.drbd2165.in‚0( disk.nvme0c0n1.out‚133972.5986328125Á95Ð4×%/ _wandbruntime‚15077  _runtime‚15077°çõèÊØ%Ä _step‚746 ' lr-AdamW/pg1‚0.00040665454161303226 ' lr-AdamW/pg2‚0.00040665454161303226 trainer/global_step‚9099 % _timestamp‚1.7843048934627578e+09 _runtime‚15077.679914239HMÏÙ%É ' lr-AdamW/pg2‚0.00040665454161303226 trainer/global_step‚9099 % _timestamp‚1.7843048934627578e+09 _runtime‚15077.679914239 _step‚746 ' lr-AdamW/pg1‚0.00040665454161303226êA‹Pc4Ú%/ _wandbruntime‚15077  _runtime‚15077üø&Ô(Û%þO ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8544034857144 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8542690256822 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2614109711654912 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚77583103 # _timestamp‚1.78430489346384e+09 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2614109737828352 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 156373884 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚717421940724736 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2612980666322944 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚78790781 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚78780808 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚49140241 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚15077.681329072 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8558677371392 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25798546 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2612980691750912 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 156363905 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2621524726608056 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8558677371392 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚25798801 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2622667043636736 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8557331981824 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚23341695 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8557331981824 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2614109711654912 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚260 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚706376898768896 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2622667043636736 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11045444583424 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚77583103 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 trainer/global_step‚9099 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚78790781 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2621523356579766îLáf (Ü%ƒP PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚77583097 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 _step‚747 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8558677371392 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25798546 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2612980691750912 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 156363905 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2621524726608056 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8558677371392 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚25798801 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2622667043636736 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8557331981824 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚23341695 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8557331981824 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2614109711654912 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚260 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚706376898768896 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2622667043636736 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11045444583424 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚77583103 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 trainer/global_step‚9099 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚78790781 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2621523356579766 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8544034857144 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8542690256822 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2614109711654912 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚77583103 # _timestamp‚1.78430489346384e+09 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2614109737828352 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 156373884 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚717421940724736 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2612980666322944 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚78790781 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚78780808 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚49140241 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚15077.681329072 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚49145395 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚77591309 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 [MDeviceStatsMonitor.on_train_batch_end/inactive_splÞ›¼íDit_bytes.large_pool.current‚ 368091136 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 trainer/global_step‚9099 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 156390582 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚706454385176576 % _timestamp‚1.7843048948514218e+09 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚77591303 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2614387738558464 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚717501021273088 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚706454017085440 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2614387712385024 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚256 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25801157 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2614387738558464 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚251 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2613258548101120 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚78799273 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 156390582 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 156380589 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11046601567232 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2613258573529088 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8559602414080 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2621802161824188 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚748 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚78789286 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8543613723068 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚23344243 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚23344238 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008^Î'ß%N P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚78799273 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚77591309 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚717500618652672 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8558257017344 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚78789286 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 _runtime‚15079.068741907 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚49145395 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚77591309 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 trainer/global_step‚9099 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 156390582 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚706454385176576 % _timestamp‚1.7843048948514218e+09 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚77591303 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2614387738558464 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚717501021273088 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚706454017085440 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2614387712385024 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚256 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25801157 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2614387738558464 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚251 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2613258548101120 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚78799273 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 156390582 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 156380589 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11046601567232 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2613258573529088 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8559602414080 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2621802161824188 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚748 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚78789286 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8543613723068 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚23344243 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚23344238 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 156380589 FDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚49402874 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚49402610 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 trainer/global_step‚9149 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚79206560 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚25934024 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25933765 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚77993397 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚721356384524288 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2635463682601674 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚259 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8604993302016 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚264 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2626874780659712 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚23468850 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚79216533 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 =6DeviceStatsMonitor.on_train_batch_start/maxCzJ{n_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2636613396072960 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚77993403 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2636613396072960 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚77993403 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚49402874 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚49402610 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 trainer/global_step‚9149 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚79206560 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚25934024 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25933765 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚77993397 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚721356384524288 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2635463682601674 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚259 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8604993302016 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚264 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119ï§A¹?4ó%/ _wandbruntime‚15158  _runtime‚15158X½i'ô%ŠN I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2635742487846096 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2628287748890624 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8605918344704 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 _step‚752 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚49407927 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2627152662437888 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2636893693408768 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 % _timestamp‚1.7843049747795324e+09 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8605918344704 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2627152637009920 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8604572947968 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚78001603 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚721435057839104 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2628287775064064 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11106317749248 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚23471388 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2636892321838592 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25936539 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 157226634 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11106352278528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8604572947968 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2636893693408768 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚79215038 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚710328705560576 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 157226634 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚78001609 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2628287775064064 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 157216641 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25936789 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚721434655218688 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚78001609 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8591195436610 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8589850836176 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2628287748890624 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2635743857874498 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚255 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚250 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8589850836176 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2628287748890624 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2635743857874498 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚255 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚250 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚49407927 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2627152662437888 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2636893693408768 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 % _timestamp‚1.7843049747795324e+09 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8605918344704 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2627152637009920 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8604572947968 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚78001603 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚721435057839104 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2628287775064064 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11106317749248 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚23471388 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2636892321838592 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25936539 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 157226634 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11106352278528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8604572947968 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2636893693408768 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚79215038 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚710328705560576 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 157226634 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚78001609 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2628287775064064 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 157216641 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25936789 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚721434655218688 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820ðòÃCƒ4ö%/ _wandbruntime‚15159  _runtime‚15159œì¿ ÷%„ /loss/FRt1/detection_loss‚0.4774885581282536 /loss/FBt1/detection_loss‚0.5201652146144105 .loss/FBt2/detection_loss‚0.447539880870923 %loss/total_loss‚6.847585745794074 % _timestamp‚1.7843049748595288e+09 2loss/SRqqt2/assignment_loss‚0.3877735389353104 1loss/SRqqt1/detection_loss‚0.6984352089385403 1loss/SRqqt2/detection_loss‚0.5668386976288731 _step‚753 0loss/FRt2/assignment_loss‚0.4904457844098409 /loss/FRt2/detection_loss‚1.4134819925278426 epoch‚22 0loss/FRt1/assignment_loss‚1.4072177235016041 2loss/FBt2/assignment_loss‚0.029393314406507234 trainer/global_step‚9149 _runtime‚15159.076529688 2loss/SRqqt1/assignment_loss‚0.4036077782790654 2loss/FBt1/assignment_loss‚0.005198053552903608áö£ø%‰ % _timestamp‚1.7843049748595288e+09 2loss/SRqqt2/assignment_loss‚0.3877735389353104 1loss/SRqqt1/detection_loss‚0.6984352089385403 1loss/SRqqt2/detection_loss‚0.5668386976288731 _step‚753 0loss/FRt2/assignment_loss‚0.4904457844098409 /loss/FRt2/detection_loss‚1.4134819925278426 epoch‚22 0loss/FRt1/assignment_loss‚1.4072177235016041 2loss/FBt2/assignment_loss‚0.029393314406507234 trainer/global_step‚9149 _runtime‚15159.076529688 2loss/SRqqt1/assignment_loss‚0.4036077782790654 2loss/FBt1/assignment_loss‚0.005198053552903608 /loss/FRt1/detection_loss‚0.4774885581282536 /loss/FBt1/detection_loss‚0.5201652146144105 .loss/FBt2/detection_loss‚0.447539880870923 %loss/total_loss‚6.847585745794074ñæ_š9”ù%‚(:‰ ÔªéÒÕ¡Øö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚290.031' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚58.0062 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0žÄð­.ú%‚(:£ ÔªéÒ§”Éø network.recv‚ 3.786106e+06* disk./.usagePercent‚37.661155718774125$ disk.nvme0c0n1.out‚ 134040.734375 disk.rbd0.out‚0 disk.dm-0.out‚ 134040.734375$ disk./.usageGB‚561.1784362792969 proc.cpu.threads‚17 disk.drbd2165.out‚0 disk.sda2.in‚0 disk.dm-0.in‚34539.44140625 disk.sda1.in‚0 disk.sda1.out‚0 network.sent‚ 1.750521e+07 disk.rbd3.in‚0 disk.sda2.out‚0 disk.rbd1.out‚ 39.521484375' proc.memory.availableMB‚ 990.7578125" proc.memory.rssMB‚ 2804.8671875$ disk.nvme0c0n1.in‚34539.44140625 disk.dm-1.out‚ 778.046875* proc.memory.percent‚17.119550704956055 cpu‚25.574919061886416 disk.sda3.in‚ 75.29296875 disk.rbd3.out‚0" disk.nvme0n1.in‚34539.44140625" disk.nvme0n1.out‚ 134040.734375 disk.sda.in‚ 75.29296875 disk.sda.out‚ 778.046875 disk.dm-1.in‚ 75.29296875& disk.rbd0.in‚1.759218604441423e+13 disk.rbd1.in‚ 0.50830078125$ memory_percent‚93.95289421081543 disk.drbd2165.in‚0 disk.sda3.out‚ 778.046875¸úƒðŸû%‚(:” ãªéÒ“Þðö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚37.0 gpu.0.powerWatts‚109.997' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.999399999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÊÜ{;ü%‚(:° ãªéÒè¬Âù disk.dm-1.in‚ 75.29296875 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda3.out‚ 778.40625 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.rbd1.in‚ 0.50830078125) disk./.usagePercent‚37.66159093158233 disk.rbd0.out‚0 disk.dm-0.in‚34691.76953125 network.recv‚ 3.791178e+06$ disk./.usageGB‚561.1849212646484 disk.sda.in‚ 75.29296875$ disk.nvme0c0n1.in‚34691.76953125" disk.nvme0n1.in‚34691.76953125 disk.sda2.in‚0( proc.memory.availableMB‚ 990.58984375" proc.memory.rssMB‚ 2804.8671875 disk.sda3.in‚ 75.29296875 disk.drbd2165.in‚0 disk.sda2.out‚0$ memory_percent‚93.95391941070557* proc.memory.percent‚17.119550704956055 proc.cpu.threads‚17 disk.dm-1.out‚ 778.40625& disk.rbd0.in‚1.759218604441423e+13 cpu‚25.574841807050838$ disk.dm-0.out‚134053.51611328125' disk.nvme0n1.out‚134053.51611328125 disk.rbd1.out‚ 39.583984375 disk.drbd2165.out‚0) disk.nvme0c0n1.out‚134053.51611328125 network.sent‚ 1.7568491e+07 disk.sda.out‚ 778.40625„Å)Zžý%‚(:“ òªéÒÌ­êö gpu.0.gpu‚89.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚288.679' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚57.73579999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0_¤?þ%‚(:´ òªéÒÓ¨¯ù$ disk./.usageGB‚561.1896209716797 disk.dm-1.out‚ 778.73828125 disk.sda3.in‚ 75.29296875 proc.cpu.threads‚17 network.recv‚ 3.794359e+06) disk./.usagePercent‚37.66190633286452 disk.dm-1.in‚ 75.29296875 disk.dm-0.in‚34691.98828125 disk.sda1.out‚0 disk.sda.in‚ 75.29296875 disk.drbd2165.in‚0 disk.rbd3.in‚0$ disk.nvme0c0n1.in‚34691.98828125 disk.sda2.in‚0* proc.memory.percent‚17.119550704956055# disk.dm-0.out‚134063.7275390625& disk.nvme0n1.out‚134063.7275390625# memory_percent‚93.9540147781372& disk.rbd0.in‚1.759218604441423e+13 disk.drbd2165.out‚0 disk.rbd3.out‚0( disk.nvme0c0n1.out‚134063.7275390625" disk.nvme0n1.in‚34691.98828125 disk.rbd1.out‚ 39.583984375 cpu‚25.57478499322395 disk.sda1.in‚0 disk.sda3.out‚ 778.73828125 disk.sda2.out‚0( proc.memory.availableMB‚ 990.57421875" proc.memory.rssMB‚ 2804.8671875 network.sent‚ 1.7573589e+07 disk.sda.out‚ 778.73828125 disk.rbd0.out‚0 disk.rbd1.in‚ 0.50830078125’B žÿ%‚(:“ «éÒªÂôö gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚282.255' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚56.45099999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0£¦§üD€&‚(:¹ «éÒ¥áÓù disk.rbd3.out‚0* proc.memory.percent‚17.119550704956055 network.recv‚ 3.797672e+06$ disk./.usageGB‚561.1989707946777 disk.rbd1.in‚ 0.50830078125 disk.dm-1.in‚ 75.29296875 disk.sda3.in‚ 75.29296875 disk.dm-1.out‚ 779.44140625 disk.dm-0.in‚34692.41015625$ disk.dm-0.out‚134071.34033203125) disk./.usagePercent‚37.66253380733094 disk.drbd2165.out‚0 disk.rbd1.out‚ 39.646484375& disk.rbd0.in‚1.759218604441423e+13 disk.sda1.in‚0" proc.memory.rssMB‚ 2804.8671875 disk.sda2.in‚0 disk.sda2.out‚0 proc.cpu.threads‚17) disk.nvme0c0n1.out‚134071.34033203125" disk.nvme0n1.in‚34692.41015625 disk.sda.out‚ 779.44140625 disk.sda3.out‚ 779.44140625 network.sent‚ 1.7578669e+07 disk.drbd2165.in‚0 disk.rbd3.in‚0$ disk.nvme0c0n1.in‚34692.41015625 disk.sda.in‚ 75.29296875 disk.rbd0.out‚0 cpu‚25.574710342466965' disk.nvme0n1.out‚134071.34033203125 disk.sda1.out‚0$ memory_percent‚93.95406246185303( proc.memory.availableMB‚ 990.56640625Å 6“&‚(:ˆ «éÒ°©ø gpu.0.gpu‚62.0 gpu.0.memory‚20, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚335.505' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚67.101 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¿ýdk>‚&‚(:³ «éÒ°ýù$ disk./.usageGB‚561.2031860351562 disk.dm-1.out‚ 779.9453125 disk.sda3.out‚ 779.9453125* disk./.usagePercent‚37.662816695656275 disk.sda.out‚ 779.9453125 disk.dm-1.in‚ 75.29296875& disk.rbd0.in‚1.759218604441423e+13 disk.rbd0.out‚0 disk.rbd3.out‚0" disk.nvme0n1.in‚34692.61328125( proc.memory.availableMB‚ 990.57421875( disk.nvme0c0n1.out‚134086.4072265625 disk.rbd1.out‚ 39.646484375 cpu‚25.574720697357268 disk.sda3.in‚ 75.29296875 disk.sda2.out‚0 network.sent‚ 1.7583753e+07 disk.sda.in‚ 75.29296875 disk.dm-0.in‚34692.61328125 disk.rbd3.in‚0$ disk.nvme0c0n1.in‚34692.61328125 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd2165.in‚0& disk.nvme0n1.out‚134086.4072265625 proc.cpu.threads‚17 network.recv‚ 3.800808e+06 disk.drbd2165.out‚0 disk.sda2.in‚0" proc.memory.rssMB‚ 2804.8671875# disk.dm-0.out‚134086.4072265625 disk.rbd1.in‚ 0.50830078125# memory_percent‚93.9540147781372* proc.memory.percent‚17.119550704956055É€õ4ƒ&/ _wandbruntime‚15237  _runtime‚15237mt¬NÈ„& _runtime‚15237.373101242 _step‚754 & lr-AdamW/pg1‚0.0004011884312388235 & lr-AdamW/pg2‚0.0004011884312388235 trainer/global_step‚9199 % _timestamp‚1.7843050531527743e+09@¿Z¥Í…&Ç & lr-AdamW/pg1‚0.0004011884312388235 & lr-AdamW/pg2‚0.0004011884312388235 trainer/global_step‚9199 % _timestamp‚1.7843050531527743e+09 _runtime‚15237.373101242 _step‚754ò”±ÒÙ4†&/ _wandbruntime‚15237  _runtime‚15237ܱ"Ü(‡&€P WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚714125539536896 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚49667409 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚725289906433024 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚78403697 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚26071668 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11164769523712 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 158045988 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 _step‚755 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2650559748509184 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2641909810839552 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚725290309060608 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2640768869568512 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8635164483038 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8651309232640 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2641909784666112 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2641909784666112 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚49667668 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11164734987264 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚78403703 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚78403697 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2649404008623582 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚714125171445760 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚79632312 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8651309232640 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 158036009 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚26071414 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2640768844140544 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚79632312 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚23596000 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚254 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2650561120072192 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8649963843072 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8649963843072 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2649405378651872 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚714125539536896 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚49667409 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚725289906433024 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚78403697 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚26071668 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11164769523712 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 158045988 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 _step‚755 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2650559748509184 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2641909810839552 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚725290309060608 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2640768869568512 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8635164483038 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8651309232640 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2641909784666112 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2641909784666112 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚49667668 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11164734987264 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚78403703 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚78403697 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2649404008623582 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚714125171445760 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚79632312 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8651309232640 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 158036009 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚26071414 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2640768844140544 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚79632312 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚23596000 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚254 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2650561120072192 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8649963843072 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 ZHDI~HêeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8649963843072 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2649405378651872óÌpù4‰&/ _wandbruntime‚15238  _runtime‚15238+#±'Š&ŠN F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2642187811569664 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8652234275328 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚79640790 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11165956834304 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚78411903 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2642187785396224 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2641046725918720 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚78411909 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚78411903 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚23598543 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2650840045844992 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚78411909 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2649682813868004 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚49672691 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8652234275328 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚242 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚714203025944576 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 % _timestamp‚1.7843050545456533e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 _step‚756 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2649684183896406 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚79640790 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚79650777 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚725368580158464 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 158062686 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚49672938 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚247 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8650888878592 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚23598538 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8636087949284 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚725368982778880 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 trainer/global_step‚9199 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2642187811569664 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚79650777 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176#˜úñ'‹&N ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 _step‚756 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2649684183896406 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚79640790 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚79650777 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚725368580158464 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 158062686 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚49672938 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚247 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8650888878592 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚23598538 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8636087949284 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚725368982778880 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 trainer/global_step‚9199 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2642187811569664 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚79650777 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2642187811569664 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8652234275328 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚79640790 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11165956834304 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚78411903 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2642187785396224 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2641046725918720 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚78411909 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚78411903 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚23598543 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2650840045844992 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚78411909 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2649682813868004 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚49672691 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8652234275328 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚242 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚714203025944576 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 % _timestamp‚1.7843050545456533e+09ô‘A6­4Œ&/ _wandbruntime‚15238  _runtime‚15238~‡®&‰ 0loss/FRt2/assignment_loss‚0.5410564302483483 0loss/FBt1/detection_loss‚0.44397156133147125 %loss/total_loss‚6.829484294377586 0loss/FRt1/assignment_loss‚1.4190596217046003 2loss/FBt1/assignment_loss‚0.009704172339476423 3loss/FBt2/assignment_loss‚0.0030695730446061126 0loss/FRt1/detection_loss‚0.47802974912080254 /loss/FRt2/detection_loss‚1.4165005468605014 1loss/SRqqt2/detection_loss‚0.6276922904144465 trainer/global_step‚9199 % _timestamp‚1.7843050546314237e+09 2loss/SRqqt2/assignment_loss‚0.4159159310612116 0loss/FBt2/detection_loss‚0.33875574798418073 _step‚757 2loss/SRqqt1/assignment_loss‚0.4043936834614542 1loss/SRqqt1/detection_loss‚0.7313349868064859 epoch‚22 _runtime‚15238.848386892ŸõzŽ&Ž 2loss/FBt1/assignment_loss‚0.009704172339476423 3loss/FBt2/assignment_loss‚0.0030695730446061126 0loss/FRt1/detection_loss‚0.47802974912080254 /loss/FRt2/detection_loss‚1.4165005468605014 1loss/SRqqt2/detection_loss‚0.6276922904144465 trainer/global_step‚9199 % _timestamp‚1.7843050546314237e+09 2loss/SRqqt2/assignment_loss‚0.4159159310612116 0loss/FBt2/detection_loss‚0.33875574798418073 _step‚757 2loss/SRqqt1/assignment_loss‚0.4043936834614542 1loss/SRqqt1/detection_loss‚0.7313349868064859 epoch‚22 _runtime‚15238.848386892 0loss/FRt2/assignment_loss‚0.5410564302483483 0loss/FBt1/detection_loss‚0.44397156133147125 %loss/total_loss‚6.829484294377586 0loss/FRt1/assignment_loss‚1.4190596217046003õ‰©u”&‚(:‰ Ÿ«éÒ˜¬ßø gpu.0.gpu‚97.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚334.052' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚66.8104 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0wÀ…&D&‚(:¹ Ÿ«éÒ˜õÖù disk.sda2.out‚0$ memory_percent‚93.95420551300049 disk.dm-1.out‚ 780.23828125 disk.rbd3.out‚0" disk.nvme0n1.in‚34898.88671875 disk.rbd1.in‚ 0.50830078125 network.recv‚ 3.804315e+06 disk.rbd0.out‚0 disk.sda1.in‚0* proc.memory.percent‚17.119550704956055 cpu‚25.574645316913905$ disk./.usageGB‚561.2109146118164 disk.dm-0.in‚34898.88671875$ disk.nvme0c0n1.in‚34898.88671875) disk.nvme0c0n1.out‚134094.31884765625 disk.drbd2165.in‚0' disk.nvme0n1.out‚134094.31884765625" proc.memory.rssMB‚ 2804.8671875 network.sent‚ 1.7588883e+07 disk.sda3.in‚ 75.29296875 disk.rbd1.out‚ 39.708984375 disk.sda.in‚ 75.29296875 disk.sda.out‚ 780.23828125 disk.dm-1.in‚ 75.29296875& disk.rbd0.in‚1.759218604441423e+13 disk.sda3.out‚ 780.23828125 disk.drbd2165.out‚0 disk.rbd3.in‚0$ disk.dm-0.out‚134094.31884765625 disk.sda1.out‚0 disk.sda2.in‚0( proc.memory.availableMB‚ 990.54296875 proc.cpu.threads‚17) disk./.usagePercent‚37.66333536692065:ÞšŸ‘&‚(:” ®«éÒ³•¨÷ gpu.0.gpu‚58.0 gpu.0.memory‚17, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚37.0 gpu.0.powerWatts‚109.757' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.951400000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0k×¹=’&‚(:² ®«éÒ”ªåù disk.rbd3.out‚0$ disk.nvme0c0n1.in‚34899.30859375) proc.memory.percent‚17.43307113647461 disk.rbd0.out‚0 disk.sda2.out‚0 disk.rbd1.out‚ 39.708984375 disk.sda.in‚ 75.29296875 disk.sda.out‚ 780.5078125! proc.memory.rssMB‚ 2856.234375 proc.cpu.threads‚17* disk./.usagePercent‚37.663776211894614 disk.dm-1.in‚ 75.29296875 disk.sda3.out‚ 780.5078125 disk.drbd2165.out‚0 disk.sda1.in‚0$ memory_percent‚94.26910877227783 cpu‚25.576649827793347$ disk./.usageGB‚561.2174835205078# disk.dm-0.out‚134109.6103515625" disk.nvme0n1.in‚34899.30859375( proc.memory.availableMB‚ 938.94921875 network.recv‚ 3.809287e+06 disk.dm-1.out‚ 780.5078125& disk.nvme0n1.out‚134109.6103515625 disk.drbd2165.in‚0( disk.nvme0c0n1.out‚134109.6103515625 disk.sda2.in‚0& disk.rbd0.in‚1.759218604441423e+13 disk.dm-0.in‚34899.30859375 disk.sda3.in‚ 75.29296875 disk.rbd1.in‚ 0.50830078125 network.sent‚ 1.7652165e+07 disk.sda1.out‚0 disk.rbd3.in‚0¿1"F4“&/ _wandbruntime‚15266  _runtime‚15266UϽl”&ž{ ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9685273170471191 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 +particle/specificity‚0.9125515818595886 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 particle/accuracy_2_of_3‚1 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6041666269302368 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9814814329147339 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 *jet/accuracy_1_of_4‚0.9617059826850891 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8678609132766724 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8318405151367188 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 *jet/accuracy_1_of_2‚0.7476266026496887 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9583333134651184 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.015104766003787518 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9847887754440308 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.2604166865348816 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 *jet/accuracy_1_of_3‚0.9035845994949341 *jet/accuracy_3_of_3‚0.0704515129327774 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9285714030265808 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9909733533859253 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.13321802020072937 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.20452281832695007 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6548234224319458 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.19682693481445312 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3105551302433014 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9964339137077332 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.49679678678512573 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.04469915106892586 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6605452299118042 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5588337182998657 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6692768931388855 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6998847126960754 , Purity/0FRt2SRqqt*FBt/FBt_purity‚0.96875 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4670095443725586 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6802092790603638 6 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.650277316570282 'particle/f_score‚0.6367766261100769 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.07692307978868484 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.21375009417533875 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9972327947616577 8"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.674161434173584 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2457793802022934 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.20348528027534485 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9949505925178528 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6876727342605591 particle/accuracy_1_of_3‚1 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8414776921272278 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6717143058776855 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7420443296432495 particle/accuracy_1_of_4‚1 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8338624238967896 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7207198739051819 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9977760314941406 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7686885595321655 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 (particle/accuracy‚0.8489698767662048 particle/accuracy_5_of_5‚0 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6054564118385315 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9977760314941406 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 )jet/accuracy_1_of_1‚0.319002240896225 particle/accuracy_1_of_2‚1 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.04599812626838684 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6886081099510193 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8414776921272278 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 jet/accuracy_5_of_5‚0 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6214250326156616 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8019101023674011 )jet/accuracy_2_of_4‚0.769576907157898 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.8571428656578064 jet/accuracy_2_of_5‚1 _step‚758 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 jet/accuracy_3_of_5‚1 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6426964402198792 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9960559606552124 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.4547813832759857 6 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.462334543466568 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 9"Purity/2FRt0SRqqt0FBt/event_purity‚0.0289472546428442 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8019101023674011 )loss/val_total_loss‚8.148767511968183 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.36985883116722107 +jet/accuracy_2_of_2‚0.15327879786491394 particle/accuracy_1_of_1‚1 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.27276691794395447 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8728280663490295 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.11740951240062714 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 *jet/accuracy_2_of_3‚0.5469124913215637 epoch‚22 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9344097971916199 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9870357513427734 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 trainer/global_step‚9199 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7491930723190308 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6175073981285095 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.0833333358168602 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 8"Purity/0FRt2SRqqt0FBt/event_purity‚0.467989057302475 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7431625127792358 +jet/accuracy_4_"^jŸof_4‚0.01628045178949833 particle/accuracy_3_of_3‚1 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.07692307978868484 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9814814329147339 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.20837150514125824 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6321221590042114 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4652777910232544 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.20796555280685425 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 particle/accuracy_2_of_2‚1 jet/accuracy_1_of_5‚1 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.2770589292049408 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.13005496561527252 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.21375009417533875 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03368465229868889 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 /particle/accuracy_4_of_4‚0.9428318738937378 particle/accuracy_2_of_5‚1 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.22806796431541443 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 _runtime‚15266.853109847 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.18492960929870605 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.04109150916337967 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.2932699918746948 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8009259104728699 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6214250326156616 5 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.99383944272995 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.05012582987546921 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9930799603462219 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 $ _timestamp‚1.784305082635217e+09 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6434231996536255 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.046383947134017944 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6881461143493652 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.42824074625968933 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.04474375396966934 * Purity/2FRt*SRqqt2FBt/FRt_purity‚0.875 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7151128053665161 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6314900517463684 , Purity/0FRt2SRqqt1FBt/FBt_purity‚0.96875 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.22620442509651184 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6720708608627319 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6601932048797607 6 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.851549506187439 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 -"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4375 particle/accuracy_2_of_4‚1 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9615384340286255 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9942593574523926 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.0833333358168602 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9982911944389343 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9926468133926392 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.1736060082912445 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7526496052742004 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.11666667461395264 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.22578777372837067 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.22035972774028778 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.20578476786613464 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9938523173332214 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5324999690055847 8"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.673154354095459 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6397922039031982 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9004586338996887 6 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.184131920337677 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6438611745834351 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7424599528312683 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 particle/accuracy_3_of_5‚1 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9927904009819031 /particle/accuracy_3_of_4‚0.9965986609458923 particle/accuracy_4_of_5‚1 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3663497567176819 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 5 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.98426353931427 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6225270628929138 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.16239683330059052 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5680805444717407 6validation_average_jet_accuracy‚0.3592848777770996 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8009259104728699 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.05012582987546921 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.2148357331752777 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.45535513758659363 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7372016906738281 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6918917298316956 +"Purity/2FRt*SRqqt2FBt/event_purity‚0.75 jet/accuracy_4_of_5‚0 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.1980753391981125 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9120061993598938 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.041575051844120026 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.02319674752652645 +particle/sensitivity‚0.6164996027946472 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6450976133346558 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7273015975952148 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.19125418365001678 +jet/accuracy_3_of_4‚0.09484109282493591 particle/accuracy_1_of_5‚1 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.1461700201034546 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9829939603805542 8"Purity/0FRt1SRqqt1FBt/event_purity‚0.580028235912323 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2263561189174652 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969¤ÀLð©=•&£{ >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.9004586338996887 6 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.184131920337677 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6438611745834351 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7424599528312683 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 particle/accuracy_3_of_5‚1 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9927904009819031 /particle/accuracy_3_of_4‚0.9965986609458923 particle/accuracy_4_of_5‚1 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3663497567176819 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 5 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.98426353931427 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6225270628929138 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.16239683330059052 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5680805444717407 6validation_average_jet_accuracy‚0.3592848777770996 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8009259104728699 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.05012582987546921 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.2148357331752777 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.45535513758659363 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7372016906738281 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6918917298316956 +"Purity/2FRt*SRqqt2FBt/event_purity‚0.75 jet/accuracy_4_of_5‚0 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.1980753391981125 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9120061993598938 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.041575051844120026 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.02319674752652645 +particle/sensitivity‚0.6164996027946472 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6450976133346558 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7273015975952148 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.19125418365001678 +jet/accuracy_3_of_4‚0.09484109282493591 particle/accuracy_1_of_5‚1 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.1461700201034546 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9829939603805542 8"Purity/0FRt1SRqqt1FBt/event_purity‚0.580028235912323 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2263561189174652 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9685273170471191 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 +particle/specificity‚0.9125515818595886 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 particle/accuracy_2_of_3‚1 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6041666269302368 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9814814329147339 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 *jet/accuracy_1_of_4‚0.9617059826850891 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8678609132766724 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8318405151367188 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 *jet/accuracy_1_of_2‚0.7476266026496887 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9583333134651184 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.015104766003787518 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9847887754440308 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.2604166865348816 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 *jet/accuracy_1_of_3‚0.9035845994949341 *jet/accuracy_3_of_3‚0.0704515129327774 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9285714030265808 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9909733533859253 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.13321802020072937 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.20452281832695007 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6548234224319458 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.19682693481445312 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3105551302433014 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9964339137077332 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.49679678678512573 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.04469915106892586 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6605452299118042 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5588337182998657 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6692768931388855 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6998847126960754 , Purity/0FRt2SRqqt*FBt/FBt_purity‚0.96875 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4670095443725586 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6802092790603638 6 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.650277316570282 'particle/f_score‚0.6367766261100769 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.07692307978868484 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.21375009417533875 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9972327947616577 8"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.674161434173584 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2457793802022934 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.20348528027534485 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9949505925178528 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6876727342605591 particle/accuracy_1_of_3‚1 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8414776921272278 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6717143058776855 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7420443296432495 particle/accuracy_1_of_4‚1 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8338624238967896 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7207198739051819 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9977760314941406 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7686885595321655 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 (particle/accuracy‚0.8489698767662048 particle/accuracy_5_of_5‚0 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6054564118385315 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9977760314941406 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 )jet/accuracy_1_of_1‚0.319002240896225 particle/accuracy_1_of_2‚1 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.04599812626838684 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6886081099510193 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8414776921272278 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 jet/accuracy_5_of_5‚0 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6214250326156616 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8019101023674011 )jet/accuracy_2_of_4‚0.769576907157898 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.8571428656578064 jet/accuracy_2_of_5‚1 _step‚758 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 jet/accuracy_3_of_5‚1 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6426964402198792 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9960559606552124 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.4547813832759857 6 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.462334543466568 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 9"Purity/2FRt0SRqqt0FBt/event_purity‚0.0289472546428442 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8019101023674011 )loss/val_total_loss‚8.148767511968183 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.36985883116722107 +jet/accuracy_2_of_2‚0.15327879786491394 particle/accuracy_1_of_1‚1 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.27276691794395447 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8728280663490295 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.11740951240062714 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 *jet/accuracy_2_of_3‚0.5469124913215637 epoch‚22 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9344097971916199 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9870357513427734 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 trainer/global_step‚9199 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7491930723190308 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6175073981285095 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.0833333358168602 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 8"Purity/0FRt2SRqqt0FBt/event_purity‚0.467989057302475 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7431625127792358 +jet/accuracy_4_of_4‚0.01628045178949833 particle/accuracy_3_of_3‚1 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.07692307978868484 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9814814329147339 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.20837150514125824 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6321221590042114 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4652777910232544 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.20796555280685425 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 particle/accuracy_2_of_2‚1 jet/accuracy_1_of_5‚1 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.2770589292049408 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.13005496561527252 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.21375009417533875 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03368465229868889 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 /particle/accuracy_4_of_4‚0.9428318738937378 particle/accuracy_2_of_5‚1 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.22806796431541443 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 _runtime‚15266.853109847 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.18492960929870605 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.04109150916337967 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.2932699918746948 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8009259104728699 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6214250326156616 5 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.99383944272995 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.05012582987546921 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9930799603462219 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 $ _timestamp‚1.784305082635217e+09 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6434231996536255 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.046383947134017944 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6881461143493652 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.42824074625968933 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.04474375396966934 * Purity/2FRt*SRqqt2FBt/FRt_purity‚0.875 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7151128053665161 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6314900517463684 , Purity/0FRt2SRqqt1FBt/FBt_purity‚0.96875 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.22620442509651184 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6720708608627319 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6601932048797607 6 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.851549506187439 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 -"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4375 particle/accuracy_2_of_4‚1 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9615384340286255 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9942593574523926 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.0833333358168602 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9982911944389343 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9926468133926392 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.1736060082912445 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7526496052742004 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.11666667461395264 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.22578777372837067 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.22035972774028778 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.20578476786613464 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9938523173332214 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5324999690055847 8"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.673154354095459 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6397922039031982ö%åÌÞ–&‚(:“ ½«éÒ¡Øüö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚300.808' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚60.16160000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0u†øB—&‚(:· ½«éÒ±ôïù disk.sda3.out‚ 780.92578125 disk.sda2.in‚0 disk.sda.in‚ 75.29296875 disk.rbd3.in‚0' disk.nvme0n1.out‚134117.62646484375 disk.sda2.out‚0& disk.rbd0.in‚1.759218604441423e+13 cpu‚25.578192692975048 disk.drbd2165.out‚0 network.sent‚ 1.7657781e+07* disk./.usagePercent‚37.664063964363095$ disk./.usageGB‚561.2217712402344" disk.nvme0n1.in‚34899.51171875 disk.rbd1.out‚ 39.771484375 disk.dm-1.in‚ 75.29296875 disk.sda1.in‚0 disk.drbd2165.in‚0 disk.rbd3.out‚0$ disk.nvme0c0n1.in‚34899.51171875) disk.nvme0c0n1.out‚134117.62646484375 disk.sda.out‚ 780.92578125 disk.dm-0.in‚34899.51171875& proc.memory.availableMB‚ 989.578125 disk.sda1.out‚0$ disk.dm-0.out‚134117.62646484375# memory_percent‚93.9600944519043 proc.cpu.threads‚17 network.recv‚ 3.812807e+06 disk.rbd0.out‚0" proc.memory.rssMB‚ 2804.8671875 disk.dm-1.out‚ 780.92578125 disk.sda3.in‚ 75.29296875 disk.rbd1.in‚ 0.50830078125* proc.memory.percent‚17.119550704956055·2ç\œ˜&‚(:‘ Ì«éÒŽÅœ÷ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚40.0 gpu.0.powerWatts‚335.114' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚67.02279999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0•è@™&‚(:µ Ì«éÒñ†øù network.recv‚ 3.817383e+06* proc.memory.percent‚17.119550704956055 disk.sda.in‚ 75.29296875 network.sent‚ 1.7714885e+07& disk.rbd0.in‚1.759218604441423e+13 disk.sda1.in‚0 disk.sda1.out‚0( disk.nvme0c0n1.out‚134127.6455078125& disk.nvme0n1.out‚134127.6455078125 disk.sda2.in‚0 disk.dm-1.in‚ 75.29296875 disk.sda3.in‚ 75.29296875 disk.sda3.out‚ 781.08984375 disk.drbd2165.in‚0 disk.drbd2165.out‚0 cpu‚25.578197456620735$ disk./.usageGB‚561.2286376953125$ disk.nvme0c0n1.in‚34899.73046875 disk.sda2.out‚0 proc.cpu.threads‚17# disk.dm-0.out‚134127.6455078125 disk.rbd1.out‚ 39.771484375 disk.rbd0.out‚0 disk.dm-0.in‚34899.73046875 disk.rbd3.in‚0# memory_percent‚93.9629316329956 disk.rbd1.in‚ 0.50830078125 disk.sda.out‚ 781.08984375) disk./.usagePercent‚37.66452477792473 disk.rbd3.out‚0" disk.nvme0n1.in‚34899.73046875( proc.memory.availableMB‚ 989.11328125 disk.dm-1.out‚ 781.08984375" proc.memory.rssMB‚ 2804.8671875r SýŸš&‚(:” Û«éÒÆñ­÷ gpu.0.gpu‚90.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚302.407' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚60.481399999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0©G€45›&‚(:ª Û«éÒÁâú network.recv‚ 3.821084e+06$ disk./.usageGB‚561.2365493774414 disk.rbd3.in‚0$ disk.nvme0c0n1.in‚34960.47265625$ disk.nvme0n1.out‚134141.10546875) disk./.usagePercent‚37.66505573755074 disk.rbd1.in‚ 0.50830078125 disk.drbd2165.out‚0" disk.nvme0n1.in‚34960.47265625 disk.sda.in‚ 75.29296875 disk.dm-0.in‚34960.47265625 proc.cpu.threads‚17 disk.sda2.in‚0 disk.sda.out‚ 781.3359375 disk.dm-1.out‚ 781.3359375 disk.sda3.in‚ 75.29296875 disk.sda3.out‚ 781.3359375 disk.rbd1.out‚ 39.833984375 disk.sda1.in‚0% proc.memory.availableMB‚ 989.21875 network.sent‚ 1.7719972e+07 disk.sda2.out‚0* proc.memory.percent‚17.119550704956055 cpu‚25.578134346504207 disk.drbd2165.in‚0 disk.rbd3.out‚0& disk.nvme0c0n1.out‚134141.10546875 disk.dm-1.in‚ 75.29296875& disk.rbd0.in‚1.759218604441423e+13 disk.rbd0.out‚0! disk.dm-0.out‚134141.10546875 disk.sda1.out‚0$ memory_percent‚93.96228790283203" proc.memory.rssMB‚ 2804.8671875ÁAí’œ&‚(:‡ ê«éÒÝ€ýö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚41.0 gpu.0.powerWatts‚271.19' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.238 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0­~lß4&‚(:© ê«éÒ›´ú network.sent‚ 1.7725055e+07 disk.drbd2165.out‚0 disk.rbd1.in‚ 0.50830078125 disk.sda2.out‚0# disk.nvme0n1.out‚134153.0546875& disk.rbd0.in‚1.759218604441423e+13 disk.sda1.out‚0 disk.sda3.in‚ 75.29296875 disk.rbd3.in‚0 disk.rbd3.out‚0$ disk.nvme0c0n1.in‚35051.46484375% disk.nvme0c0n1.out‚134153.0546875 network.recv‚ 3.82466e+06 disk.dm-0.in‚35051.46484375 disk.sda1.in‚0 disk.sda.out‚ 781.6796875 disk.dm-1.in‚ 75.29296875$ memory_percent‚93.96440982818604 disk.sda3.out‚ 781.6796875 disk.rbd0.out‚0 cpu‚25.578093165512026 proc.cpu.threads‚17 disk.drbd2165.in‚0 disk.dm-1.out‚ 781.6796875( proc.memory.availableMB‚ 988.87109375 disk.rbd1.out‚ 39.833984375) disk./.usagePercent‚37.66461898869733" disk.nvme0n1.in‚35051.46484375" proc.memory.rssMB‚ 2804.8671875* proc.memory.percent‚17.119550704956055$ disk./.usageGB‚561.2300415039062 disk.dm-0.out‚134153.0546875 disk.sda2.in‚0 disk.sda.in‚ 75.29296875²ùò‘ž&‚(:† ù«éÒâˆÙö gpu.0.gpu‚99.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚338.45' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚67.69 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0éR`î<Ÿ&‚(:± ù«éÒè™øø' disk.nvme0n1.out‚134162.87255859375$ memory_percent‚93.96424293518066 disk.rbd1.in‚ 0.50830078125 network.recv‚ 3.827528e+06 disk.rbd0.out‚0 disk.dm-0.in‚35051.88671875 disk.sda3.in‚ 75.29296875 disk.drbd2165.in‚0) disk.nvme0c0n1.out‚134162.87255859375 cpu‚25.57806945627983 network.sent‚ 1.7729997e+07) disk./.usagePercent‚37.66514329212745 disk.sda.out‚ 781.921875 disk.dm-1.out‚ 781.921875& disk.rbd0.in‚1.759218604441423e+13 disk.rbd3.out‚0" proc.memory.rssMB‚ 2804.8671875$ disk.nvme0c0n1.in‚35051.88671875 disk.sda2.out‚0' proc.memory.availableMB‚ 988.8984375 disk.sda1.out‚0 disk.drbd2165.out‚0 disk.rbd1.out‚ 39.896484375$ disk./.usageGB‚561.2378540039062$ disk.dm-0.out‚134162.87255859375 disk.sda2.in‚0 disk.sda.in‚ 75.29296875 proc.cpu.threads‚17 disk.sda1.in‚0 disk.rbd3.in‚0* proc.memory.percent‚17.119550704956055 disk.dm-1.in‚ 75.29296875 disk.sda3.out‚ 781.921875" disk.nvme0n1.in‚35051.88671875dY§už &‚(:“ ˆ¬éÒâ° ø gpu.0.gpu‚97.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚272.86' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚54.571999999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¹™Ì•¡&‚(:º ˆ¬éÒ¸Ôþø) disk./.usagePercent‚37.66543949284457) disk.nvme0c0n1.out‚134183.46533203125 disk.dm-1.in‚ 75.29296875 network.sent‚ 1.7734941e+07$ disk./.usageGB‚561.2422676086426$ disk.nvme0c0n1.in‚35052.10546875" disk.nvme0n1.in‚35052.10546875' disk.nvme0n1.out‚134183.46533203125 disk.dm-1.out‚ 782.26171875 network.recv‚ 3.830738e+06 disk.sda1.in‚0 proc.cpu.threads‚17 disk.rbd3.in‚0 disk.sda2.out‚0 disk.dm-0.in‚35052.10546875 disk.sda3.in‚ 75.29296875 disk.rbd3.out‚0 disk.rbd1.in‚ 0.50830078125$ memory_percent‚92.10140705108643) proc.memory.availableMB‚ 1294.10546875$ disk.dm-0.out‚134183.46533203125 disk.drbd2165.in‚0 disk.rbd1.out‚ 39.974609375 disk.sda.out‚ 782.26171875 disk.rbd0.out‚0* proc.memory.percent‚17.119550704956055 disk.sda1.out‚0 disk.sda3.out‚ 782.26171875±óÝá° disk.sda.in‚ 75.29296875& disk.rbd0.in‚1.759218604441423e+13" proc.memory.rssMB‚ 2804.8671875 cpu‚25.578090709567746 disk.drbd2165.out‚0 disk.sda2.in‚0™8ÀÖ4¢&/ _wandbruntime‚15345  _runtime‚15345BÚöôÉ£&à _runtime‚15345.648032782 _step‚759 ' lr-AdamW/pg1‚0.00039844996394743527 ' lr-AdamW/pg2‚0.00039844996394743527 trainer/global_step‚9249 $ _timestamp‚1.784305161430853e+09/|ödΤ&È $ _timestamp‚1.784305161430853e+09 _runtime‚15345.648032782 _step‚759 ' lr-AdamW/pg1‚0.00039844996394743527 ' lr-AdamW/pg2‚0.00039844996394743527 trainer/global_step‚9249÷£èëÍ4¥&/ _wandbruntime‚15345  _runtime‚15345$²(¦&€P S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2668339924992000 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚23732063 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚80117046 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8703467774976 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2658488841981952 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8688575327905 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 159063049 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚26244631 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚49976694 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚26244884 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 159073028 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2658488867409920 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚80127019 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2659637828780032 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚718047604076544 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8687230727583 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚78946009 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚49976952 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚258 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚80127019 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2659637802606592 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8703467774976 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚78946003 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11232065257984 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 % _timestamp‚1.7843051614318717e+09 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚78946009 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚78946003 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2659637828780032 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚729280071962112 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2667177442737825 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 _runtime‚15345.649328412 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2659637802606592 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚80117046 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚253 _step‚760 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2668341296555008 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2667176072709535 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚729279669334528 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2668341296555008 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8702122385408 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 trainer/global_step‚9249 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2668339924992000 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚23732063 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚80117046 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8703467774976 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2658488841981952 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8688575327905 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 159063049 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚26244631 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚49976694 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚26244884 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 159073028 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2658488867409920 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚80127019 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2659637828780032 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚718047604076544 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8687230727583 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚78946009 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚49976952 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚258 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚80127019 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2659637802606592 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8703467774976 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚78946003 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11232065257984 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 % _timestamp‚1.7843051614318717e+09 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚78946009 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚78946003 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2659637828780032 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚729280071962112 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2667177442737825 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 _runtime‚15345.649328412 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2659637802606592 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚80117046 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚253 _step‚760 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2668341296555008 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2667176072709535 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚729279669334528 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2668341296555008 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8702122385408 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 trainer/global_step‚9249ør:Dt4¨&/ _wandbruntime‚15347  _runtime‚15347Zü J'©&ŠN =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 _runtime‚15347.035342029 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 159079733 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚242 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚729358762726912 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚718125090484224 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2658766723760128 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚49981990 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚78954215 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2659915829510144 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7843051628179045e+09 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26247384 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8689498794263 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 159079733 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 159089726 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚80135511 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚49982237 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2659915803336704 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2667454877953957 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚23734611 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚23734606 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2668620222327808 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚718125458575360 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648˜ €˜¶ª&N TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚80135511 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚49982237 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2659915803336704 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2667454877953957 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚23734611 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚23734606 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2668620222327808 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 cODeviceStatsMonitor.on_ ;÷_!train_batch_end/inactive_split_bytes.large_pool.allocated‚718125458575360 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 _runtime‚15347.035342029 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 159079733 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚242 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚729358762726912 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚718125090484224 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2658766723760128 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚49981990 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚78954215 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2659915829510144 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7843051628179045e+09 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26247384 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8689498794263 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 159079733 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 159089726ù†(vñ4«&/ _wandbruntime‚15347  _runtime‚15347`CÌ ¬&… 3loss/SRqqt1/assignment_loss‚0.34094966338275073 2loss/FBt2/assignment_loss‚0.006321030258802882 1loss/SRqqt1/detection_loss‚0.8039371119653982 1loss/SRqqt2/detection_loss‚0.6278795687136586 %loss/total_loss‚6.717066394295534 trainer/global_step‚9249 3loss/SRqqt2/assignment_loss‚0.38536091117843924 .loss/FRt2/detection_loss‚1.456885855948218 -loss/FBt1/detection_loss‚0.42661395527622 % _timestamp‚1.7843051629038637e+09 2loss/FBt1/assignment_loss‚0.003649112029106608 /loss/FRt1/detection_loss‚0.4867058826361217 0loss/FBt2/detection_loss‚0.40291505501577135 /loss/FRt1/assignment_loss‚1.286657296519243 1loss/FRt2/assignment_loss‚0.48919095137180424 epoch‚23 _runtime‚15347.120776715 _step‚762i‰C­&Š trainer/global_step‚9249 3loss/SRqqt2/assignment_loss‚0.38536091117843924 .loss/FRt2/detection_loss‚1.456885855948218 -loss/FBt1/detection_loss‚0.42661395527622 % _timestamp‚1.7843051629038637e+09 2loss/FBt1/assignment_loss‚0.003649112029106608 /loss/FRt1/detection_loss‚0.4867058826361217 0loss/FBt2/detection_loss‚0.40291505501577135 /loss/FRt1/assignment_loss‚1.286657296519243 1loss/FRt2/assignment_loss‚0.48919095137180424 epoch‚23 _runtime‚15347.120776715 _step‚762 3loss/SRqqt1/assignment_loss‚0.34094966338275073 2loss/FBt2/assignment_loss‚0.006321030258802882 1loss/SRqqt1/detection_loss‚0.8039371119653982 1loss/SRqqt2/detection_loss‚0.6278795687136586 %loss/total_loss‚6.717066394295534úf÷1W?®&‚(:´ —¬éÒ¥À…ù disk.dm-1.out‚ 782.50390625& disk.rbd0.in‚1.759218604441423e+13 disk.sda1.in‚0" proc.memory.rssMB‚ 2804.8671875 cpu‚25.578046848867174 network.recv‚ 3.83433e+06) disk./.usagePercent‚37.66677764422604 disk.sda.in‚ 75.30859375# disk.nvme0c0n1.in‚ 35052.3515625 disk.rbd1.out‚ 40.037109375 network.sent‚ 1.7739577e+07 disk.rbd1.in‚ 0.50830078125 disk.sda.out‚ 782.50390625 disk.dm-0.in‚ 35052.3515625) disk.nvme0c0n1.out‚134192.25732421875) proc.memory.availableMB‚ 1374.70703125$ disk.dm-0.out‚134192.25732421875 disk.sda1.out‚0 disk.sda3.out‚ 782.50390625 disk.drbd2165.out‚0 disk.sda2.out‚0 disk.rbd3.out‚0' disk.nvme0n1.out‚134192.25732421875 disk.sda2.in‚0 disk.sda3.in‚ 75.30859375 disk.drbd2165.in‚0 disk.rbd3.in‚0* proc.memory.percent‚17.119550704956055 proc.cpu.threads‚17" disk./.usageGB‚561.26220703125 disk.dm-1.in‚ 75.30859375 disk.rbd0.out‚0! disk.nvme0n1.in‚ 35052.3515625$ memory_percent‚91.60945415496826su_Ÿ¯&‚(:” —¬éÒŽ§†Š gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚272.035' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚54.407000000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0AŠ7š”°&‚(:‰ ¦¬éÒý·íö gpu.0.gpu‚100.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚335.505' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚67.101 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0’mâ´E±&‚(:º ¦¬éÒž¯¯ù$ disk.nvme0c0n1.in‚35196.14453125 disk.sda2.in‚0 disk.sda.in‚ 75.30859375 disk.rbd0.out‚0$ disk.dm-0.out‚134202.52685546875 disk.sda3.in‚ 75.30859375 disk.sda2.out‚0 disk.dm-1.in‚ 75.30859375 disk.rbd3.out‚0' disk.nvme0n1.out‚134202.52685546875) disk./.usagePercent‚37.66723999383288 disk.rbd1.in‚ 0.50830078125 disk.sda.out‚ 782.88671875 disk.sda1.in‚0$ memory_percent‚91.60816669464111* proc.memory.percent‚17.119550704956055 disk.drbd2165.in‚0 disk.drbd2165.out‚0) disk.nvme0c0n1.out‚134202.52685546875 disk.dm-1.out‚ 782.88671875 cpu‚25.578051072629005 proc.cpu.threads‚17 disk.sda1.out‚0 network.sent‚ 1.7802709e+07 network.recv‚ 3.838844e+06 disk.rbd3.in‚0" disk.nvme0n1.in‚35196.14453125& disk.rbd0.in‚1.759218604441423e+13) proc.memory.availableMB‚ 1374.91796875" proc.memory.rssMB‚ 2804.8671875$ disk./.usageGB‚561.2690963745117 disk.sda3.out‚ 782.88671875 disk.rbd1.out‚ 40.037109375 disk.dm-0.in‚35196.14453125‚ÜŠó”²&‚(:‰ µ¬éÒÈôö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚263.646' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.7292 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0í`Ì›@³&‚(:µ µ¬éÒΉåù disk.sda.in‚ 75.30859375 disk.sda.out‚ 783.23046875 disk.sda1.in‚0 disk.sda2.out‚0$ memory_percent‚91.60828590393066 proc.cpu.threads‚17 network.sent‚ 1.7808364e+07) disk./.usagePercent‚37.66755232302466& disk.nvme0n1.out‚134213.8271484375 disk.rbd1.in‚ 0.50830078125 disk.dm-1.in‚ 75.30859375 disk.drbd2165.in‚0" proc.memory.rssMB‚ 2804.8671875 disk.rbd3.out‚0& disk.rbd0.in‚1.759218604441423e+13( proc.memory.availableMB‚ 1374.8984375* proc.memory.percent‚17.119550704956055 disk.rbd0.out‚0 disk.drbd2165.out‚0# disk.dm-0.out‚134213.8271484375 disk.sda1.out‚0 disk.sda3.in‚ 75.30859375 disk.sda3.out‚ 783.23046875 disk.sda2.in‚0 disk.rbd3.in‚0 disk.rbd1.out‚ 40.099609375 disk.dm-1.out‚ 783.23046875 cpu‚25.57808697756409 network.recv‚ 3.842523e+06$ disk./.usageGB‚561.2737503051758 disk.dm-0.in‚35196.36328125$ disk.nvme0c0n1.in‚35196.36328125" disk.nvme0n1.in‚35196.36328125( disk.nvme0c0n1.out‚134213.8271484375ìÉ?÷”´&‚(:‰ ĬéÒŽÍ÷ gpu.0.gpu‚81.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚325.989' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚65.1978 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¨å…,;µ&‚(:° ĬéÒ¬¦æù proc.cpu.threads‚17) disk./.usagePercent‚37.66803131312122$ disk./.usageGB‚561.2808876037598 disk.sda1.out‚0% disk.nvme0n1.out‚134227.916015625 disk.rbd1.in‚ 0.50830078125 disk.sda1.in‚0 network.recv‚ 3.846028e+06 disk.sda.out‚ 783.4140625 disk.dm-1.in‚ 75.30859375 disk.drbd2165.in‚0 disk.sda2.in‚0) proc.memory.availableMB‚ 1667.78515625 disk.dm-0.in‚35196.56640625 disk.sda3.in‚ 75.30859375" proc.memory.rssMB‚ 2804.8671875 cpu‚25.57802727853487 disk.drbd2165.out‚0$ disk.nvme0c0n1.in‚35196.56640625 disk.rbd1.out‚ 40.099609375" disk.dm-0.out‚134227.916015625 disk.rbd3.in‚0 disk.sda2.out‚0* proc.memory.percent‚17.119550704956055 disk.sda.in‚ 75.30859375 disk.dm-1.out‚ 783.4140625 disk.rbd3.out‚0 disk.sda3.out‚ 783.4140625' disk.nvme0c0n1.out‚134227.916015625$ memory_percent‚89.82064723968506 network.sent‚ 1.7812912e+07& disk.rbd0.in‚1.759218604441423e+13 disk.rbd0.out‚0" disk.nvme0n1.in‚35196.56640625“—±b”¶&‚(:‰ Ó¬éÒ®ˆ†÷ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚256.398' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚51.2796 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0xJ‚C·&‚(:¸ Ó¬éÒÌ£¡ù disk.sda.out‚ 783.56640625 disk.dm-1.in‚ 75.30859375& disk.rbd0.in‚1.759218604441423e+13$ disk.dm-0.out‚134241.74365234375 disk.sda3.out‚ 783.56640625$ memory_percent‚88.42144012451172& proc.memory.availableMB‚ 1897.03125 network.sent‚ 1.7817983e+07 disk.rbd1.in‚ 0.50830078125 disk.rbd1.out‚ 40.240234375 disk.sda1.in‚0 disk.rbd3.out‚0 cpu‚25.577953343292624 disk.sda3.in‚ 75.30859375 disk.sda2.out‚0$ disk.nvme0c0n1.in‚35196.98828125" proc.memory.rssMB‚ 2804.8671875* disk./.usagePercent‚37.668503647021886" disk.nvme0n1.in‚35196.98828125' disk.nvme0n1.out‚134241.74365234375 disk.drbd2165.out‚0 proc.cpu.threads‚17 disk.sda.in‚ 75.30859375 disk.dm-1.out‚ 783.56640625 disk.drbd2165.in‚0) disk.nvme0c0n1.out‚134241.74365234375 network.recv‚ 3.849565e+06$ disk./.usageGB‚561.2879257202148 disk.sda2.in‚0 disk.rbd3.in‚0 disk.rbd0.out‚0 disk.dm-0.in‚35196.98828125 disk.sda1.out‚0* proc.memory.percent‚17.119550704956055õ('<4¸&/ _wandbruntime‚15425  _runtime‚15425"Äzʹ&Ä % _timestamp‚1.7843052413657794e+09 _runtime‚15425.582986365 _step‚763 ' lr-AdamW/pg1‚0.00039570812302604724 ' lr-AdamW/pg2‚0.00039570812302604724 trainer/global_step‚9299ù£„»Ïº&É ' lr-AdamW/pg1‚0.00039570812302604724 ' lr-AdamW/pg2‚0.00039570812302604724 trainer/global_step‚9299 % _timestamp‚1.7843052413657794e+09 _runtime‚15425.582986365 _step‚763û:¹4»&/ _wandbruntime‚15425  _runtime‚15425§h}w(¼&€P TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚733213982823424 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2681117768759733 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚26382152 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2673537839112192 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚79356303 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚721922292551680 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚79356309 % _timestamp‚1.7843052413665547e+09 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2682287648991232 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2672382956318720 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8733467840691 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2673537865285632 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8748438316032 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚79356303 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚80542798 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2682286277428224 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚80552771 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚50241365 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2672382930890752 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2673537865285632 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚26382426 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2681116398731443 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚279 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8748438316032 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚50241644 trainer/global_step‚9299 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2673537839112192 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚80552771 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚23859213 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 159909080 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 _step‚764 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 159899101 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚23859218 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11291690271744 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36633600 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚80542798 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚733213578098688 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚721921924460544 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2682286277428224 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚274 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11291653638144 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2682287648991232 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404724736 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648þ:󗎽&…P JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚23859218 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11291690271744 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36633600 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚80542798 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚733213578098688 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚721921924460544 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2682286277428224 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚274 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11291653638144 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2682287648991232 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404724736 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚733213982823424 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2681117768759733 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚26382152 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2673537839112192 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚79356303 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚721922292551680 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚79356309 % _timestamp‚1.7843052413665547e+09 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2682287648991232 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocationÆÚ}s.freed‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2672382956318720 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8733467840691 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2673537865285632 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8748438316032 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚79356303 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚80542798 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2682286277428224 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚80552771 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚50241365 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2672382930890752 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2673537865285632 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚26382426 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2681116398731443 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚279 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8748438316032 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚50241644 trainer/global_step‚9299 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2673537839112192 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚80552771 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚23859213 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 159909080 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 _step‚764 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 159899101 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚79364515 QDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚79364509 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2673815839842304 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 trainer/global_step‚9299 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2672660812668928 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 159915785 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2672660838096896 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2673815866015744 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚79364515 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚79364509 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚266 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 159925778 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚80561263 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2682565203193856 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚23861756 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚733292651944960 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8749363351552 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚50246658 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚721999410868224 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2682566574764032 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2673815839842304 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚271 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36626432 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8750708748288 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2682566574764032 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 159925778 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26384902 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 _step‚765 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚80551276 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 $ _timestamp‚1.784305242753547e+09 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚50246929 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8749363351552 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚721999778959360 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11292836359168 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2673815866015744 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404717568 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2681396574004267 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2681395203975865vh‡'À&ŽN [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8735735907371 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 _runtime‚15426.971011732 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11292872985600 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8750708748288 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2682566574764032 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 159925778 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26384902 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 _step‚765 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚80551276 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 $ _timestamp‚1.784305242753547e+09 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚50246929 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8749363351552 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚721999778959360 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11292836359168 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2673815866015744 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404717568 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2681396574004267 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2681395203975865 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚733292247227392 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8734391306937 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚80561263 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚26385168 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚79364515 QDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚79364509 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2673815839842304 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 trainer/global_step‚9299 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2672660812668928 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 159915785 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2672660838096896 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2673815866015744 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚79364515 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚79364509 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚266 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 159925778 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚80561263 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2682565203193856 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚23861756 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚733292651944960 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8749363351552 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚50246658 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚721999410868224 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2682566574764032 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2673815839842304 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚271 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36626432 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 EsÛbd2165.in‚0" disk.nvme0c0n1.in‚ 35363.859375 disk.sda2.out‚0 disk.rbd1.in‚ 0.51220703125 disk.sda3.in‚ 75.31640625 disk.nvme0n1.in‚ 35363.859375' disk.nvme0n1.out‚134289.66943359375$ disk.dm-0.out‚134289.66943359375 disk.rbd3.in‚0 disk.rbd3.out‚0( disk./.usagePercent‚37.6701172625102 disk.dm-1.out‚ 784.6171875) disk.nvme0c0n1.out‚134289.66943359375$ memory_percent‚87.58656978607178 proc.cpu.threads‚17 network.sent‚ 1.7896503e+07& disk.rbd0.in‚1.759218604441423e+13" proc.memory.rssMB‚ 2804.8671875$ disk./.usageGB‚561.3119697570801 disk.rbd0.out‚0 disk.dm-0.in‚ 35363.859375 disk.sda1.out‚0) proc.memory.availableMB‚ 2033.81640625* proc.memory.percent‚17.119550704956055 cpu‚25.577880391439514 disk.sda.in‚ 75.31640625 disk.sda1.in‚0 disk.sda3.out‚ 784.6171875 disk.drbd2165.out‚0 disk.sda2.in‚0 disk.dm-1.in‚ 75.31640625 disk.rbd1.out‚ 40.6201171875Ìä9LŸÌ&‚(:” ž­éÒÑÁ„÷ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚284.911' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚56.982200000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0%yV“;Í&‚(:° ž­éÒ‹â×ú) disk./.usagePercent‚37.67093904669512& disk.rbd0.in‚1.759218604441423e+13 disk.sda3.out‚ 786.85546875 disk.drbd2165.out‚0 disk.nvme0n1.in‚ 35498.78125 disk.rbd1.in‚ 0.51220703125$ disk./.usageGB‚561.3242149353027 disk.sda1.out‚0 disk.sda3.in‚ 75.31640625! disk.nvme0c0n1.in‚ 35498.78125' disk.nvme0n1.out‚134316.57958984375 disk.rbd0.out‚0$ disk.dm-0.out‚134316.57958984375 disk.sda2.out‚0 disk.dm-1.out‚ 786.85546875 network.sent‚ 1.7901618e+07 disk.sda2.in‚0$ memory_percent‚87.15963363647461* proc.memory.percent‚17.119550704956055 disk.dm-0.in‚ 35498.78125' proc.memory.availableMB‚ 2103.765625" proc.memory.rssMB‚ 2804.8671875 disk.sda1.in‚0) disk.nvme0c0n1.out‚134316.57958984375 disk.sda.out‚ 786.85546875 cpu‚25.577797243315437 proc.cpu.threads‚17 disk.drbd2165.in‚0 disk.rbd1.out‚ 40.6201171875 disk.sda.in‚ 75.31640625 network.recv‚ 3.867802e+06 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.dm-1.in‚ 75.31640625\%)4Î&/ _wandbruntime‚15505  _runtime‚15505³Î¸ÈÏ& _runtime‚15505.182035059 _step‚767 & lr-AdamW/pg1‚0.0003929630845779784 & lr-AdamW/pg2‚0.0003929630845779784 trainer/global_step‚9349 % _timestamp‚1.7843053209606874e+09Žª¸ÍÐ&Ç trainer/global_step‚9349 % _timestamp‚1.7843053209606874e+09 _runtime‚15505.182035059 _step‚767 & lr-AdamW/pg1‚0.0003929630845779784 & lr-AdamW/pg2‚0.0003929630845779784ÿ@w´ë4Ñ&/ _wandbruntime‚15505  _runtime‚15505Js(Ò&€P VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚23986363 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2686277045227520 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚737147043948544 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 _step‚768 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 160735153 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2687437901791232 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚79766603 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚725796244844544 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚26520077 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2687437875617792 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 % _timestamp‚1.7843053209678648e+09 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2696232629864448 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2695056724753351 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚50506440 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404724736 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2687437875617792 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8794754246656 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚80968550 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2686277019799552 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚273 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚278 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚26520350 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2696234001427456 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚80968550 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚737147448673280 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8779704953799 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 trainer/global_step‚9349 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚79766603 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2695058094781641 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2696234001427456 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8794754246656 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚23986368 _runtime‚15505.185204601Nróe (Ó&…P =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2696232629864448 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚80978523 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚79766603 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚725796244844544 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚26520077 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2687437875617792 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 % _timestamp‚1.7843053209678648e+09 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2696232629864448 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2695056724753351 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚50506440 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404724736 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2687437875617792 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8794754246656 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚80968550 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2686277019799552 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚273 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚278 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚26520350 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2696234001427456 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚80968550 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚737147448673280 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8779704953799 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 trainer/global_step‚9349 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚79766603 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2695058094781641 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2696234001427456 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8794754246656 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚23986368 _runtime‚15505.185204601 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚23986363 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2686277045227520 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚737147043948544 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 _step‚768 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 160735153 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2687437901791232 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8795679282176 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 160761830 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚80987015 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 160751837 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2695336900026175 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8797024678912 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚23988906 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8795679282176 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36626432 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2696511555630080 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 _step‚769 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2696512927200256 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8780628420045 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2687715902521344 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚50511617 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2686554927005696 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 160751837 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8797024678912 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚80987015 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2687715876347904 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8781973020479 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2687715902521344 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚80977028 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 137³ŽÔXS1570176 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2686554901577728 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 160761830 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚50511886 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2695335529997773 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995Q<î&'Ö&ŽN F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚725874099343360 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8780628420045 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2687715902521344 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚50511617 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2686554927005696 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 160751837 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8797024678912 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚80987015 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2687715876347904 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8781973020479 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2687715902521344 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚80977028 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2686554901577728 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 160761830 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚50511886 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2695335529997773 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8795679282176 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 160761830 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚80987015 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 160751837 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2695336900026175 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8797024678912 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚23988906 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8795679282176 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36626432 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2696511555630080 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 _step‚769 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2696512927200256 EDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2710180353863680 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8841070177280 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚81394302 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚729670565228544 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚81394302 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2700171108708352 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚80176909 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚50771609 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36633600 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚269 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2701337912123392 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚50771340 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚26658091 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚264 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 161581184 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚26657827 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2708998420803549 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2710178982300672 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2710178982300672 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚81404275 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚80176909 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2710180353863680 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8841070177280 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11410120261632 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 _step‚772 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚24113513 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2701337938296832 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚80176903 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404724736 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚81404275 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192Ac¦ë&…P YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 WJDeviceStatsM5—ó/e&onitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 _step‚772 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚24113513 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2701337938296832 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚80176903 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404724736 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚81404275 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8827286667229 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2701337938296832 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8842415566848 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚729670933319680 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2708997050775259 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚80176903 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚741081053581312 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2700171134136320 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _runtime‚15585.413410835 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8825942066907 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚24113518 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8842415566848 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2710180353863680 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8841070177280 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚81394302 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚729670565228544 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚81394302 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2700171108708352 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚80176909 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚50771609 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36633600 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚269 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2701337912123392 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚50771340 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚26658091 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚264 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 161581184 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚26657827 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2708998420803549 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2710178982300672 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2710178982300672 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚81404275 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚80176909 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2710180353863680 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8841070177280 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11410120261632 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175„œ¸@õ4ì&/ _wandbruntime‚15586  _runtime‚15586·Sv'í&ŠN bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11411309258752 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2701615912853504 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 161587889 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚80185109 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚741159324268544 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 % _timestamp‚1.7843054025836456e+09 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26660569 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11411272632320 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚81412767 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 161597882 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚267 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2701615912853504 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚50776625 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8826865533153 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8841995212800 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚50776892 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚741159728986112 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36626432 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚81402780 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚26660831 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚729748419727360 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2709275856019681 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚773 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚80185115 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚80185109 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚81402780 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚729748051636224 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2700448990486528 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚24116056 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚262 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚80185115 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2710459279636480 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2710459279636480 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404717568 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192’O¯'î&N L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 161597882 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚267 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2701615912853504 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚50776625 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8826865533153 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8841995212800 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚50776892 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚741159728986112 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36626432 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚81402780 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚26660831 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚729748419727360 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2709275856019681 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚773 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚80185115 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚80185109 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚81402780 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚729748051636224 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2700448990486528 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚24116056 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚262 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚80185115 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2710459279636480 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2710459279636480 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404717568 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11411309258752 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2701615912853504 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 161587889 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚80185109 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚741159324268544 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 % _timestamp‚1.7843054025836456e+09 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26660569 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11411272632320 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚81412767 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640…hªrp4ï&/ _wandbruntime‚15586  _runtime‚15586]`ð&‰ 0loss/FRt2/assignment_loss‚0.5061316796454134 3loss/SRqqt1/assignment_loss‚0.39636567528228034 0loss/FBt2/detection_loss‚0.37414717182140356 epoch‚23 _step‚774 0loss/FRt1/assignment_loss‚1.3461939398586587 4loss/FBt1/assignment_loss‚4.8484610797734885e-05 /loss/FRt1/detection_loss‚0.4537095669014271 /loss/FRt2/detection_loss‚1.3877442892234988 $ _timestamp‚1.784305402669194e+09 2loss/SRqqt2/assignment_loss‚0.4347582991020395 1loss/SRqqt2/detection_loss‚0.6027032388604414 /loss/FBt1/detection_loss‚0.7631253320826491 4loss/FBt2/assignment_loss‚0.00021015790272613222 0loss/SRqqt1/detection_loss‚0.814794691330677 %loss/total_loss‚7.079932526622012 trainer/global_step‚9399 _runtime‚15586.886163545% áñ&Ž 2loss/SRqqt2/assignment_loss‚0.4347582991020395 1loss/SRqqt2/detection_loss‚0.6027032388604414 /loss/FBt1/detection_loss‚0.7631253320826491 4loss/FBt2/assignment_loss‚0.00021015790272613222 0loss/SRqqt1/detection_loss‚0.814794691330677 %loss/total_loss‚7.079932526622012 trainer/global_step‚9399 _runtime‚15586.886163545 0loss/FRt2/assignment_loss‚0.5061316796454134 3loss/SRqqt1/assignment_loss‚0.39636567528228034 0loss/FBt2/detection_loss‚0.37414717182140356 epoch‚23 _step‚774 0loss/FRt1/assignment_loss‚1.3461939398586587 4loss/FBt1/assignment_loss‚4.8484610797734885e-05 /loss/FRt1/detection_loss‚0.4537095669014271 /loss/FRt2/detection_loss‚1.3877442892234988 $ _timestamp‚1.784305402669194e+09†Ý·° Ÿò&‚(:” ‡®éÒÊ•†÷ gpu.0.gpu‚68.0 gpu.0.memory‚21, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚37.0 gpu.0.powerWatts‚109.997' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.999399999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚06\ ó&‚(:´ ‡®éÒòªéù network.recv‚ 3.893518e+06 disk.drbd2165.out‚0) disk.nvme0c0n1.out‚134393.47216796875" proc.memory.rssMB‚ 2804.8671875* proc.memory.percent‚17.119550704956055 network.sent‚ 1.8044442e+07 disk.sda.in‚ 75.31640625& disk.rbd0.in‚1.759218604441423e+13$ disk.dm-0.out‚134393.47216796875 disk.rbd1.out‚ 41.08984375$ disk./.usageGB‚561.3620491027832 disk.sda3.out‚ 790.8671875 disk.drbd2165.in‚0$ disk.nvme0c0n1.in‚35665.91015625$ memory_percent‚87.16003894805908 disk.sda1.out‚0 disk.sda.out‚ 790.8671875 disk.dm-1.out‚ 790.8671875" disk.nvme0n1.in‚35665.91015625 disk.sda2.in‚0 cpu‚25.5775013106427) disk./.usagePercent‚37.67347812941971 disk.dm-0.in‚35665.910156Jȱ8225 disk.rbd3.out‚0' disk.nvme0n1.out‚134393.47216796875) proc.memory.availableMB‚ 2103.69921875 disk.sda1.in‚0 disk.sda3.in‚ 75.31640625 disk.rbd3.in‚0 disk.sda2.out‚0 proc.cpu.threads‚17 disk.dm-1.in‚ 75.31640625 disk.rbd0.out‚0 disk.rbd1.in‚ 0.51220703125@š•ô&‚(:Š –®éÒ¯±Š÷ gpu.0.gpu‚100.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚244.029' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚48.8058 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0êÀ ˆ:õ&‚(:¯ –®éÒ¾ÿ°ù disk.sda1.out‚0$ disk./.usageGB‚561.3689765930176 disk.sda.out‚ 792.4140625 disk.dm-1.out‚ 792.4140625 disk.sda3.out‚ 792.4140625 disk.drbd2165.in‚0 disk.rbd3.in‚0" disk.nvme0c0n1.in‚ 35850.828125 disk.rbd0.out‚0 disk.sda1.in‚0) disk.nvme0c0n1.out‚134408.10595703125 disk.sda.in‚ 75.31640625 disk.dm-1.in‚ 75.31640625$ disk.dm-0.out‚134408.10595703125 disk.sda3.in‚ 75.31640625 disk.drbd2165.out‚0$ memory_percent‚87.16003894805908" proc.memory.rssMB‚ 2804.8671875 network.recv‚ 3.897465e+06 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd1.out‚ 41.08984375 network.sent‚ 1.8094257e+07' disk.nvme0n1.out‚134408.10595703125& disk.rbd0.in‚1.759218604441423e+13 disk.dm-0.in‚ 35850.828125 disk.nvme0n1.in‚ 35850.828125* proc.memory.percent‚17.119550704956055 proc.cpu.threads‚17) disk./.usagePercent‚37.67394303910189 disk.rbd1.in‚ 0.51220703125 disk.rbd3.out‚0) proc.memory.availableMB‚ 2103.69921875 cpu‚25.57746045077716'ÎDÝžö&‚(:“ ¥®éÒ”Ý÷ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚272.83' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚54.565999999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0yÐ;(0÷&‚(:¥ ¥®éÒ½º€ù% disk.nvme0n1.out‚134421.728515625 disk.dm-0.in‚ 35851.046875 disk.sda3.out‚ 792.890625" disk.nvme0c0n1.in‚ 35851.046875' disk.nvme0c0n1.out‚134421.728515625 disk.sda1.out‚0 disk.sda3.in‚ 75.31640625 disk.rbd1.in‚ 0.51220703125* proc.memory.percent‚17.119550704956055 disk.sda.out‚ 792.890625 disk.dm-1.in‚ 75.31640625 disk.drbd2165.out‚0 disk.sda2.out‚0( proc.memory.availableMB‚ 2103.9453125 network.sent‚ 1.809938e+07* disk./.usagePercent‚37.674237447766274# disk./.usageGB‚561.373363494873 disk.sda2.in‚0 cpu‚25.577403589043776 network.recv‚ 3.901292e+06" disk.dm-0.out‚134421.728515625 disk.sda1.in‚0 disk.drbd2165.in‚0 disk.rbd3.out‚0" proc.memory.rssMB‚ 2804.8671875& disk.rbd0.in‚1.759218604441423e+13 disk.rbd0.out‚0 disk.rbd3.in‚0 disk.nvme0n1.in‚ 35851.046875 proc.cpu.threads‚17 disk.sda.in‚ 75.31640625 disk.dm-1.out‚ 792.890625 disk.rbd1.out‚ 41.21484375$ memory_percent‚87.15853691101074Jn4‡”ø&‚(:‰ ´®éÒܺ˜÷ gpu.0.gpu‚92.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚46.0 gpu.0.powerWatts‚333.913' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚66.7826 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¸'ùÛ&ù&‚(:› ´®éÒŒÁýù& disk.rbd0.in‚1.759218604441423e+13 disk.sda3.out‚ 793.2109375( disk./.usagePercent‚37.6738129872745 disk.sda3.in‚ 75.31640625 cpu‚25.577326063496162" disk.dm-0.out‚134431.572265625' disk.nvme0c0n1.out‚134431.572265625 disk.sda.in‚ 75.31640625 disk.sda.out‚ 793.2109375 disk.dm-0.in‚35851.25 proc.cpu.threads‚17 disk.sda2.out‚0 disk.nvme0n1.in‚35851.25% disk.nvme0n1.out‚134431.572265625' proc.memory.availableMB‚ 2103.390625 network.recv‚ 3.904689e+06 disk.drbd2165.in‚0 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.sda2.in‚0 disk.sda1.out‚0" proc.memory.rssMB‚ 2804.8671875 disk.sda1.in‚0 disk.drbd2165.out‚0 disk.nvme0c0n1.in‚35851.25 disk.rbd1.in‚ 0.51220703125 disk.rbd0.out‚0* proc.memory.percent‚17.119550704956055 network.sent‚ 1.8104551e+07$ disk./.usageGB‚561.3670387268066 disk.dm-1.in‚ 75.31640625$ memory_percent‚87.16192245483398 disk.rbd1.out‚ 41.21484375 disk.dm-1.out‚ 793.2109375®‹ìŸŸú&‚(:” îéÒÃøôö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚283.559' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚56.711800000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ŽF%Ý:û&‚(:¯ îéÒ™ÿ‰ù) disk./.usagePercent‚37.67521206844912 disk.sda.in‚ 75.31640625 disk.dm-1.out‚ 793.52734375 disk.sda1.in‚0 disk.drbd2165.out‚0 disk.nvme0n1.in‚ 35851.671875$ disk./.usageGB‚561.3878860473633 disk.rbd0.out‚0 disk.sda3.out‚ 793.52734375 disk.drbd2165.in‚0" proc.memory.rssMB‚ 2804.8671875# disk.dm-0.out‚134439.7568359375( disk.nvme0c0n1.out‚134439.7568359375 disk.rbd1.in‚ 0.51220703125 disk.rbd1.out‚ 41.27734375 proc.cpu.threads‚17 disk.sda.out‚ 793.52734375 disk.dm-0.in‚ 35851.671875 disk.sda1.out‚0 disk.rbd3.in‚0 disk.rbd3.out‚0& disk.nvme0n1.out‚134439.7568359375 disk.sda2.in‚0 disk.sda2.out‚0 network.sent‚ 1.8110192e+07 disk.sda3.in‚ 75.31640625* proc.memory.percent‚17.119550704956055 network.recv‚ 3.90825e+06$ memory_percent‚87.16189861297607 cpu‚25.577287070406275 disk.dm-1.in‚ 75.31640625& disk.rbd0.in‚1.759218604441423e+13" disk.nvme0c0n1.in‚ 35851.671875) proc.memory.availableMB‚ 2103.39453125T¼1#4ü&/ _wandbruntime‚15665  _runtime‚15665¡*¢ËÈý& & lr-AdamW/pg1‚0.0003874641205306052 & lr-AdamW/pg2‚0.0003874641205306052 trainer/global_step‚9449 % _timestamp‚1.7843054812037926e+09 _runtime‚15665.420984326 _step‚775_ýЊÍþ&Ç % _timestamp‚1.7843054812037926e+09 _runtime‚15665.420984326 _step‚775 & lr-AdamW/pg1‚0.0003874641205306052 & lr-AdamW/pg2‚0.0003874641205306052 trainer/global_step‚9449‡ ®¦4ÿ&/ _wandbruntime‚15665  _runtime‚15665Ü@—s(€'ÿO PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚80587209 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚286 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2715237974802432 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚291 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚733544885612544 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚51036149 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2722938746825457 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8887386107904 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚24240663 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚733545253703680 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11468999977984 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚81820054 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2714065197617152 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚26795195 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11468961247232 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 406821888 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2714065223045120 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2724126706299904 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2724125334736896 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚38730752 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 162407257 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2715237974802432 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2715237948628992 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8888731497472 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚80587203 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8872179180015 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2722937376797167 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚51035858 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8887386107904 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚24240668 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚80587203 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 trainer/global_step‚9449 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2724126706299904 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚745013846859776 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8888731497472 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979:d3 ('„P VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚81820054 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2714065197617152 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚26795195 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11468961247232 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 406821888 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2714065223045120 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2724126706299904 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2724125334736896 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚38730752 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 162407257 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2715237974802432 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2715237948628992 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8888731497472 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚80587203 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8872179180015 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2722937376797167 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚51035858 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8887386107904 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚24240668 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚80587203 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 trainer/global_step‚9449 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2724126706299904 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚745013846859776 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8888731497472 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚80587209 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚286 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2715237974802432 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚291 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚733544885612544 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚51036149 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2722938746825457 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8887386107904 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚24240663 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚733545253703680 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11468999977984 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192ˆiÓŒ4‚'/ _wandbruntime‚15666  _runtime‚15666íyÁUçƒ'ŠN \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8889656540160 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚733622740111360 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2714343079395328 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚81828532 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2723217552069991 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 162433934 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚80595415 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2724405632072704 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚51041414 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚80595409 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2715515949359104 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 162433934 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26797928 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XFDeviceStatsMonitor.onb¶À)_train_batch_end/requested_bytes.small_pool.freed‚ 8873102646261 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚26798203 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚745092507860992 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚38723584 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 trainer/global_step‚9449 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2724405632072704 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 162423941 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚81828532 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8874447246695 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚51041134 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚275 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8888311143424 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 _step‚777 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2715515975532544 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8889656540160 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0]>/Ð'„'N H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 162433934 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚80595415 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2724405632072704 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚51041414 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚80595409 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2715515949359104 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 162433934 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26797928 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8873102646261 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚26798203 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚745092507860992 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚38723584 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 trainer/global_step‚9449 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2724405632072704 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 162423941 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚81828532 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8874447246695 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚51041134 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚275 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8888311143424 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 _step‚777 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2715515975532544 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8889656540160 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8889656540160 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚733622740111360 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2714343079395328 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚81828532 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2723217552069991 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995‰S§4…'/ _wandbruntime‚15666  _runtime‚15666na‡­†'Š 1loss/SRqqt2/detection_loss‚0.6036517987071462 _step‚778 2loss/SRqqt2/assignment_loss‚0.3601616911385251 3loss/FBt1/assignment_loss‚4.561230137542943e-06 4loss/FBt2/assignment_loss‚2.9685407543833012e-05 0loss/FBt2/detection_loss‚0.42083673775206953 %loss/total_loss‚6.781114401965937 % _timestamp‚1.7843054826764178e+09 0loss/FRt1/assignment_loss‚1.3263343808163925 2loss/SRqqt1/assignment_loss‚0.4369910157175311 0loss/FRt1/detection_loss‚0.49598284869749515 1loss/SRqqt1/detection_loss‚0.8132308185279294 /loss/FBt1/detection_loss‚0.3045796629556112 epoch‚23 trainer/global_step‚9449 _runtime‚15666.893322724 0loss/FRt2/assignment_loss‚0.5564010635867638 /loss/FRt2/detection_loss‚1.4629101374287912åñŽ‡' 2loss/SRqqt2/assignment_loss‚0.3601616911385251 3loss/FBt1/assignment_loss‚4.561230137542943e-06 4loss/FBt2/assignment_loss‚2.9685407543833012e-05 0loss/FBt2/detection_loss‚0.42083673775206953 %loss/total_loss‚6.781114401965937 % _timestamp‚1.7843054826764178e+09 0loss/FRt1/assignment_loss‚1.3263343808163925 2loss/SRqqt1/assignment_loss‚0.4369910157175311 0loss/FRt1/detection_loss‚0.49598284869749515 1loss/SRqqt1/detection_loss‚0.8132308185279294 /loss/FBt1/detection_loss‚0.3045796629556112 epoch‚23 trainer/global_step‚9449 _runtime‚15666.893322724 0loss/FRt2/assignment_loss‚0.5564010635867638 /loss/FRt2/detection_loss‚1.4629101374287912 1loss/SRqqt2/detection_loss‚0.6036517987071462 _step‚778Š~€ïÓŸˆ'‚(:” Ò®éÒÊÊ÷ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚166.668' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚33.333600000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ý[Ó“:‰'‚(:¯ Ò®éÒê¿ñù" disk.nvme0n1.in‚35900.78515625& disk.nvme0n1.out‚134463.2802734375 disk.rbd0.out‚0 disk.sda.in‚ 75.31640625& disk.rbd0.in‚1.759218604441423e+13 disk.rbd3.in‚0( disk.nvme0c0n1.out‚134463.2802734375 disk.rbd1.in‚ 0.51220703125 disk.dm-1.in‚ 75.31640625 disk.sda3.in‚ 75.31640625 disk.sda3.out‚ 796.0234375 disk.rbd3.out‚0$ disk.nvme0c0n1.in‚35900.78515625 disk.sda2.out‚0 disk.sda.out‚ 796.0234375 disk.dm-0.in‚35900.78515625# disk.dm-0.out‚134463.2802734375 disk.sda1.in‚0 network.sent‚ 1.8114753e+07$ disk./.usageGB‚561.3986930847168 disk.drbd2165.in‚0 disk.sda1.out‚0# memory_percent‚87.1617317199707 cpu‚25.57733785351476 proc.cpu.threads‚17 network.recv‚ 3.911758e+06) disk./.usagePercent‚37.67593733779363 disk.sda2.in‚0 disk.rbd1.out‚ 41.27734375" proc.memory.rssMB‚ 2804.8671875 disk.drbd2165.out‚0* proc.memory.percent‚17.119550704956055 disk.dm-1.out‚ 796.0234375' proc.memory.availableMB‚ 2103.421875üJ€#“Š'‚(:ˆ á®éÒÍï‰÷ gpu.0.gpu‚100.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚46.0 gpu.0.powerWatts‚287.91' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚57.582 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ÿ`ß?‹'‚(:´ á®éÒËÛ€ú disk.sda.in‚ 75.31640625$ disk.dm-0.out‚134475.78466796875 disk.sda2.in‚0 disk.sda1.in‚0 network.sent‚ 1.8177711e+07 disk.dm-0.in‚35984.44140625 disk.drbd2165.in‚0" proc.memory.rssMB‚ 2804.8671875 disk.sda3.out‚ 796.6640625$ disk.nvme0c0n1.in‚35984.44140625' disk.nvme0n1.out‚134475.78466796875 disk.sda2.out‚0 disk.rbd1.out‚ 41.33984375* proc.memory.percent‚17.119550704956055 proc.cpu.threads‚17) disk./.usagePercent‚37.67631648495184$ disk./.usageGB‚561.4043426513672 disk.rbd0.out‚0( proc.memory.availableMB‚ 2103.4609375 disk.dm-1.out‚ 796.6640625 disk.sda3.in‚ 75.31640625 disk.drbd2165.out‚0# memory_percent‚87.1614933013916 disk.dm-1.in‚ 75.31640625& disk.rbd0.in‚1.759218604441423e+13 disk.rbd3.in‚0 disk.rbd3.out‚0) disk.nvme0c0n1.out‚134475.78466796875" disk.nvme0n1.in‚35984.44140625 disk.sda1.out‚0 disk.rbd1.in‚ 0.51220703125 disk.sda.out‚ 796.6640625 cpu‚25.577327308114103 network.recv‚ 3.915791e+06ÒÚábžŒ'‚(:“ ð®éÒÝøœ÷ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚290.855' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚58.17100000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‘»Ø@'‚(:µ ð®éÒéëƒú disk.drbd2165.in‚0 disk.rbd1.in‚ 0.51220703125 disk.sda.out‚ 797.35546875) proc.memory.availableMB‚ 2103.57421875* proc.memory.percent‚17.119550704956055 disk.rbd0.out‚0$ disk.dm-0.out‚134490.77294921875 disk.sda1.out‚0 disk.drbd2165.out‚0) disk.nvme0c0n1.out‚134490.77294921875' disk.nvme0n1.out‚134490.77294921875" proc.memory.rssMB‚ 2804.8671875 network.sent‚ 1.8185057e+07 disk.sda3.in‚ 75.31640625 disk.rbd3.in‚0 cpu‚25.57730111666787 proc.cpu.threads‚17 network.recv‚ 3.919959e+06& disk.rbd0.in‚1.759218604441423e+13 disk.dm-0.in‚ 35984.8515625 disk.dm-1.in‚ 75.31640625 disk.sda1.in‚0 disk.rbd3.out‚0# disk.nvme0c0n1.in‚ 35984.8515625 disk.sda2.in‚0 disk.rbd1.out‚ 41.33984375 disk.dm-1.out‚ 797.35546875# memory_percent‚87.1608018875122* disk./.usagePercent‚37.676728145066896$ disk./.usageGB‚561.4104766845703 disk.sda3.out‚ 797.35546875! disk.nvme0n1.in‚ 35984.8515625 disk.sda2.out‚0 disk.sda.in‚ 75.31640625¯€€YŽ'‚(:’ ÿ®éҭɤ÷ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚283.03' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚56.60599999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚03i¸¬7'‚(:¬ ÿ®éÒë‚âø( proc.memory.availableMB‚ 2103.8203125 proc.cpu.threads‚17 disk.sda.out‚ 797.6484375 disk.rbd3.in‚0 disk.sda2.out‚0 disk.sda3.in‚ 75.31640625 network.sent‚ 1.8188789e+07" disk.dm-0.out‚134499.322265625 disk.drbd2165.out‚0" proc.memory.rssMB‚ 2804.8671875 cpu‚25.577231806851263 disk.sda1.out‚0 disk.sda3.out‚ 797.6484375' disk.nvme0c0n1.out‚134499.322265625% disk.nvme0n1.out‚134499.322265625 network.recv‚ 3.923611e+06 disk.rbd0.out‚0# disk.nvme0c0n1.in‚ 35985.0703125* proc.memory.percent‚17.119550704956055& disk.rbd0.in‚1.759218604441423e+13 disk.sda1.in‚0 disk.drbd2165.in‚0 disk.rbd3.out‚0! disk.nvme0n1.in‚ 35985.0703125 disk.rbd1.in‚ 0.51220703125$ memory_percent‚87.15929985046387) disk./.usagePercent‚37.67708604359976$ disk./.usageGB‚561.4158096313477 disk.sda.in‚ 75.31640625 disk.rbd1.out‚ 41.40234375 disk.dm-1.in‚ 75.31640625 disk.dm-1.out‚ 797.6484375 disk.dm-0.in‚ 35985.0703125 disk.sda2.in‚0Ži¥Î“'‚(:ˆ Ž¯éÒÚÀùö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚290.665' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚58.133 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ÖymC‘'‚(:¸ Ž¯éÒÊéÈù network.sent‚ 1.8193916e+07 disk.drbd2165.in‚0) disk.nvme0c0n1.out‚134511.90869140625 network.recv‚ 3.926821e+06$ memory_percent‚87.15927600860596 proc.cpu.threads‚17 disk.sda1.in‚0( disk./.usagePercent‚37.6775483932066 disk.sda.in‚ 75.31640625 disk.dm-0.in‚35985.28515625 disk.rbd3.in‚0" disk.nvme0n1.in‚35985.28515625' disk.nvme0n1.out‚134511.90869140625" proc.memory.rssMB‚ 2804.8671875& disk.rbd0.in‚1.759218604441423e+13 disk.sda1.out‚0 disk.sda3.in‚ 75.31640625 disk.sda3.out‚ 798.06640625 disk.drbd2165.out‚0* proc.memory.percent‚17.119550704956055 cpu‚25.577187249457367 disk.rbd1.in‚ 0.51220703125$ disk./.usageGB‚561.4226989746094 disk.dm-1.in‚ 75.31640625 disk.dm-1.out‚ 798.06640625$ disk.nvme0c0n1.in‚35985.28515625 disk.sda2.out‚0 disk.sda.out‚ 798.06640625 disk.rbd3.out‚0 disk.sda2.in‚0 disk.rbd1.out‚ 41.40234375$ disk.dm-0.out‚134511.90869140625 disk.rbd0.out‚0) proc.memory.availableMB‚ 2103.82421875sù¯‰4’'/ _wandbruntime‚15746  _runtime‚15746Ux ZÊ“'Ä _runtime‚15746.225039305 _step‚779 ' lr-AdamW/pg1‚0.00038471054811948596 ' lr-AdamW/pg2‚0.00038471054811948596 trainer/global_step‚9499 % _timestamp‚1.7843055620078201e+091JøÏ”'É % _timestamp‚1.7843055620078201e+09 _runtime‚15746.225039305 _step‚779 ' lr-AdamW/pg1‚0.00038471054811948596 ' lr-AdamW/pg2‚0.00038471054811948596 trainer/global_step‚9499‹I)„’4•'/ _wandbruntime‚15746  _runtime‚157461& © –'ÿO \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚38730752 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 trainer/global_step‚9499 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚80997509 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2738071687173120 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2738071687173120 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8919760893445 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2738073058736128 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2729137985134592 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 163243309 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚51300638 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚24367818 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚280 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2736879072847365 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8935047428096 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚748947434446848 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚748947027624960 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2729138011308032 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚15746.226336304 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0v ~N (—'„P YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 trainer/global_step‚9499 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚80997509 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2738071687173120 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2738071687173120 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8919760893445 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2738073058736128 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2729137985134592 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 163243309 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚51300638 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚24367818 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚280 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2736879072847365 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8935047428096 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚748947434446848 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚748947027624960 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2729138011308032 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚15746.226336304 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚38730752 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0ŒÁ(<É4˜'/ _wandbruntime‚15747  _runtime‚15747:mÁ'™'ŠN E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚82264271 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚737496692404224 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 _step‚781 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2738351984508928 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 163269986 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚38723584 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2737156508063497 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚749026099629056 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _runtime‚15747.613011588 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26935277 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚24370361 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚82254284 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚81005709 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 406814720 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2728237193732096 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚51305633 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚26935550 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚737497060495360 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚749025692814336 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚278 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11529000410112 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2729416012038144 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2738351984508928 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2729415985864704 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚81005715 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚82254284 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 trainer/global_step‚9499 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8935972470784 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 163269986 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2729415985864704 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2729416012038144 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚24370356 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 163259993 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚81005715 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11529039133696 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚273 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8935972470784 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8919339759369 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2738350612938752 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚82264271 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 % _timestamp‚1.7843055633954854e+09çz'š'N REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚24370356 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 163259993 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚81005715 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11529039133696 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚273 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8935972470784 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8919339759369 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2738350612938752 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚82264271 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 % _timestamp‚1.7843055633954854e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚82264271 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚737496692404224 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 _step‚781 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2738351984508928 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 163269986 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚38723584 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2737156508063497 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚749026099629056 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _runtime‚15747.613011588 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26935277 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚24370361 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚82254284 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚81005709 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 406814720 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2728237193732096 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚51305633 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚26935550 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚737497060495360 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚749025692814336 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚278 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11529000410112 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2729416012038144 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2738351984508928 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2729415985864704 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚81005715 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚82254284 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 trainer/global_step‚9499 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8935972470784 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 163269986 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2729415985864704 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2729416012038144 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚05:L4›'/ _wandbruntime‚15747  _runtime‚15747•Oì½Üœ'„ 2loss/SRqqt1/assignment_loss‚0.4568326086471374 2loss/SRqqt2/assignment_loss‚0.3703425527590644 3loss/FBt2/assignment_loss‚0.0028890212649115305 /loss/FRt1/detection_loss‚0.5125979105551033 .loss/FRt2/detection_loss‚1.366816719215581 epoch‚23 % _timestamp‚1.7843055634813378e+09 0loss/SRqqt2/detection_loss‚0.610046070754281 trainer/global_step‚9499 _runtime‚15747.698294539 _step‚782 1loss/SRqqt1/detection_loss‚0.840,b.1310009733489 /loss/FBt1/detection_loss‚0.4307010391522653 .loss/FBt2/detection_loss‚0.471938253047743 0loss/FRt1/assignment_loss‚1.3790796182320895 0loss/FRt2/assignment_loss‚0.5753943395150549 3loss/FBt1/assignment_loss‚0.0014787642899181759 %loss/total_loss‚7.018247898406498ã¸Ú'‰ _step‚782 1loss/SRqqt1/detection_loss‚0.8401310009733489 /loss/FBt1/detection_loss‚0.4307010391522653 .loss/FBt2/detection_loss‚0.471938253047743 0loss/FRt1/assignment_loss‚1.3790796182320895 0loss/FRt2/assignment_loss‚0.5753943395150549 3loss/FBt1/assignment_loss‚0.0014787642899181759 %loss/total_loss‚7.018247898406498 2loss/SRqqt1/assignment_loss‚0.4568326086471374 2loss/SRqqt2/assignment_loss‚0.3703425527590644 3loss/FBt2/assignment_loss‚0.0028890212649115305 /loss/FRt1/detection_loss‚0.5125979105551033 .loss/FRt2/detection_loss‚1.366816719215581 epoch‚23 % _timestamp‚1.7843055634813378e+09 0loss/SRqqt2/detection_loss‚0.610046070754281 trainer/global_step‚9499 _runtime‚15747.698294539Ž,5.Sžž'‚(:“ ¯éÒ¸«­÷ gpu.0.gpu‚86.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚46.0 gpu.0.powerWatts‚333.764' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚66.75280000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0­QVL;Ÿ'‚(:° ¯éÒ²‘©ù disk.sda1.out‚0 disk.sda3.out‚ 798.28515625 network.sent‚ 1.8199125e+07 disk.rbd1.out‚ 41.46484375 disk.sda.out‚ 798.28515625 disk.dm-1.out‚ 798.28515625 disk.sda1.in‚0 disk.rbd3.out‚0) disk.nvme0c0n1.out‚134525.51806640625 disk.nvme0n1.in‚ 36079.46875) proc.memory.availableMB‚ 2103.33203125* disk./.usagePercent‚37.677994614338786$ disk./.usageGB‚561.4293479919434 disk.dm-1.in‚ 75.31640625 disk.rbd3.in‚0" proc.memory.rssMB‚ 2804.8671875 proc.cpu.threads‚17& disk.rbd0.in‚1.759218604441423e+13 disk.rbd0.out‚0 disk.drbd2165.in‚0 disk.sda.in‚ 75.31640625 cpu‚25.57725735403589 disk.dm-0.in‚ 36079.46875 disk.sda3.in‚ 75.31640625' disk.nvme0n1.out‚134525.51806640625* proc.memory.percent‚17.119550704956055 disk.rbd1.in‚ 0.51220703125 network.recv‚ 3.930592e+06$ disk.dm-0.out‚134525.51806640625 disk.drbd2165.out‚0! disk.nvme0c0n1.in‚ 36079.46875$ memory_percent‚87.16228008270264 disk.sda2.in‚0 disk.sda2.out‚0U †9ž '‚(:“ ¬¯éҴŇ÷ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚300.169' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚60.03379999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0%ËÖ–3¡'‚(:¨ ¬¯éÒ¥Êúù disk.sda2.in‚0 disk.rbd1.in‚ 0.51220703125 disk.rbd0.out‚0 disk.rbd3.in‚0) proc.memory.availableMB‚ 2103.41796875" proc.memory.rssMB‚ 2804.8671875% disk.nvme0n1.out‚134537.197265625 disk.dm-1.in‚ 75.31640625 disk.dm-0.in‚ 36079.6875 disk.sda1.out‚0' disk.nvme0c0n1.out‚134537.197265625 disk.nvme0n1.in‚ 36079.6875" disk.dm-0.out‚134537.197265625 proc.cpu.threads‚17 disk.dm-1.out‚ 798.66796875 disk.sda3.in‚ 75.31640625 disk.rbd3.out‚0$ disk./.usageGB‚561.4348754882812 disk.sda2.out‚0 disk.drbd2165.out‚0$ memory_percent‚87.16175556182861 network.sent‚ 1.8262251e+07 disk.rbd1.out‚ 41.46484375 cpu‚25.577387642240517 disk.sda.out‚ 798.66796875& disk.rbd0.in‚1.759218604441423e+13 disk.sda1.in‚0 disk.sda3.out‚ 798.66796875 disk.nvme0c0n1.in‚ 36079.6875 disk.sda.in‚ 75.31640625 disk.drbd2165.in‚0* proc.memory.percent‚17.119550704956055 network.recv‚ 3.935561e+06* disk./.usagePercent‚37.678365569255895ÃkC$Ÿ¢'‚(:” »¯éÒΡŠ÷ gpu.0.gpu‚99.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚131.312' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚26.262400000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¶ËHA£'‚(:¶ »¯éÒøÓîù" disk.nvme0n1.in‚36083.76171875 network.sent‚ 1.8267332e+07$ disk.nvme0c0n1.in‚36083.76171875 disk.dm-1.out‚ 799.13671875 disk.dm-0.in‚36083.76171875# disk.dm-0.out‚134548.8466796875* disk./.usagePercent‚37.678845071367526 disk.sda3.out‚ 799.13671875 disk.sda2.in‚0 disk.sda1.in‚0 disk.drbd2165.in‚0 cpu‚25.577377769958535 proc.cpu.threads‚17 disk.rbd0.out‚0 disk.sda1.out‚0 disk.sda.in‚ 75.31640625 disk.sda.out‚ 799.13671875 disk.dm-1.in‚ 75.31640625& disk.rbd0.in‚1.759218604441423e+13 disk.rbd3.out‚0( disk.nvme0c0n1.out‚134548.8466796875& disk.nvme0n1.out‚134548.8466796875 disk.rbd1.in‚ 0.51220703125$ memory_percent‚86.63287162780762( proc.memory.availableMB‚ 2190.0703125" proc.memory.rssMB‚ 2804.8671875* proc.memory.percent‚17.119550704956055 network.recv‚ 3.939205e+06$ disk./.usageGB‚561.4420204162598 disk.rbd3.in‚0 disk.sda2.out‚0 disk.rbd1.out‚ 41.52734375 disk.sda3.in‚ 75.31640625 disk.drbd2165.out‚0t¡”¤'‚(:‰ ʯéÒƒüø gpu.0.gpu‚90.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚259.652' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚51.9304 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0w®1¾B¥'‚(:· ʯéÒÃå¡ú* disk./.usagePercent‚37.679268763836696 disk.sda.in‚ 75.31640625$ disk.dm-0.out‚134567.14208984375 disk.sda1.in‚0 disk.rbd3.in‚0 proc.cpu.threads‚17 disk.rbd1.in‚ 0.51220703125 network.recv‚ 3.941863e+06 disk.rbd0.out‚0 disk.sda1.out‚0 disk.rbd3.out‚0# disk.nvme0c0n1.in‚ 36084.2109375! disk.nvme0n1.in‚ 36084.2109375' disk.nvme0n1.out‚134567.14208984375" proc.memory.rssMB‚ 2804.8671875 cpu‚25.577350464478332) disk.nvme0c0n1.out‚134567.14208984375 network.sent‚ 1.8272274e+07$ disk./.usageGB‚561.4483337402344) proc.memory.availableMB‚ 2752.56640625* proc.memory.percent‚17.119550704956055 disk.sda.out‚ 799.29296875& disk.rbd0.in‚1.759218604441423e+13 disk.dm-0.in‚ 36084.2109375 disk.sda2.in‚0 disk.dm-1.out‚ 799.29296875 disk.sda3.in‚ 75.31640625 disk.sda3.out‚ 799.29296875 disk.sda2.out‚0 disk.rbd1.out‚ 41.52734375 disk.dm-1.in‚ 75.31640625 disk.drbd2165.in‚0 disk.drbd2165.out‚0$ memory_percent‚83.19966793060303#ÏÍÄž¦'‚(:“ Ù¯éÒ ¿€÷ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚297.575' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚59.51499999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0J Æ;§'‚(:° Ù¯éÒŸÇöù disk.dm-1.out‚ 799.7421875" disk.dm-0.out‚134576.501953125 disk.sda2.out‚0) proc.memory.availableMB‚ 2752.56640625 network.sent‚ 1.8279011e+07 disk.sda1.out‚0 disk.drbd2165.out‚0' disk.nvme0c0n1.out‚134576.501953125% disk.nvme0n1.out‚134576.501953125 disk.sda.out‚ 799.7421875 disk.sda1.in‚0 disk.drbd2165.in‚0 disk.rbd3.in‚0 disk.rbd3.out‚0 disk.sda2.in‚0 proc.cpu.threads‚17* disk./.usagePercent‚37.679715496983945* proc.memory.percent‚17.119550704956055 network.recv‚ 3.946311e+06& disk.rbd0.in‚1.759218604441423e+13 disk.dm-0.in‚36277.91015625" proc.memory.rssMB‚ 2804.8671875 disk.dm-1.in‚ 75.31640625 disk.rbd0.out‚0 disk.sda3.out‚ 799.7421875$ disk.nvme0c0n1.in‚36277.91015625$ disk./.usageGB‚561.4549903869629 disk.sda.in‚ 75.31640625 disk.sda3.in‚ 75.31640625" disk.nvme0n1.in‚36277.91015625 disk.rbd1.in‚ 0.51220703125 disk.rbd1.out‚ 41.58984375$ memory_percent‚83.19966793060303 cpu‚25.57729410108294j~”¨'‚(:‰ è¯éÒ¸¼çö gpu.0.gpu‚88.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚46.0 gpu.0.powerWatts‚334.874' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚66.9748 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚04,V7©'‚(:¬ è¯éÒ†Ðù# disk./.usageGB‚561.462085723877 disk.sda2.out‚0 disk.rbd1.out‚ 41.58984375 disk.sda.in‚ 75.31640625 disk.sda3.in‚ 75.31640625 disk.rbd3.in‚0$ memory_percent‚83.19971561431885# disk.nvme0n1.out‚134586.7890625& disk.rbd0.in‚1.759218604441423e+13 disk.drbd2165.in‚0$ disk.nvme0c0n1.in‚36281.06640625 disk.sda.out‚ 800.03515625 network.sent‚ 1.8282743e+07) disk./.usagePercent‚37.68019167099763 disk.dm-0.in‚36281.06640625% disk.nvme0c0n1.out‚134586.7890625) proc.memory.availableMB‚ 2752.55859375 network.recv‚ 3.949482e+06 disk.sda2.in‚0 disk.rbd0.out‚0 disk.drbd2165.out‚0 cpu‚25.577287135221514 disk.dm-1.in‚ 75.31640625 disk.rbd1.in‚ 0.51220703125 disk.sda1.out‚0* proc.memory.percent‚17.119550704956055 disk.dm-1.out‚ 800.03515625 disk.sda3.out‚ 800.03515625 disk.rbd3.out‚0" proc.memory.rssMB‚ 2804.8671875" disk.nvme0n1.in‚36281.06640625 disk.dm-0.out‚134586.7890625 disk.sda1.in‚0 proc.cpu.threads‚17 4ª'/ _wandbruntime‚15826  _runtime‚15826Æ,N±Æ«'À & lr-AdamW/pg1‚0.0003819544845353715 & lr-AdamW/pg2‚0.0003819544845353715 trainer/global_step‚9549 $ _timestamp‚1.784305641999719e+09 _runtime‚15826.21694176 _step‚783YK`ˬ'Å $ _timestamp‚1.784305641999719e+09 _runtime‚15826.21694176 _step‚783 & lr-AdamW/pg1‚0.0003819544845353715 & lr-AdamW/pg2‚0.0003819544845353715 trainer/global_step‚9549*É-4­'/ _wandbruntime‚15826  _runtime‚15826E‰§ú(®'€P MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8980017969152 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚81407803 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚27069374 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚741293526380544 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2752018039609344 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚82671558 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2752018039609344 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚24494968 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 406821888 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 % _timestamp‚1.7843056420007958e+09 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2750819398869273 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8964653406231 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚81407803 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2743038021640192 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚752880644935680 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2750818028840983 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2743038047813632 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚24494963 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2752019411172352 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚282 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8981363358720 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚51564624 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 trainer/global_step‚9549 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2743038047813632 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8981363358720 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚81407809 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11586750464000 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 164089340 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚51564337 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚38730752 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _runtime‚15826.218145032 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚82681531 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8965998006553 _step‚784 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097Ƹvü (¯'…P KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚27069374 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚741293526380544 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2752018039609344 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚82671558 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2752018039609344 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚24494968 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 406821888 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 % _timestamp‚1.7843056420007958e+09 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2750819398869273 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8964653406231 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚81407803 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2743038021640192 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚752880644935680 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2750818028840983 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2743038047813632 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚24494963 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2752019411172352 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚282 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8981363358720 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚51564624 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 trainer/global_step‚9549 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2743038047813632 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8981363358720 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚81407809 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11586750464000 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 164089340 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚51564337 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚38730752 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _runtime‚15826.218145032 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚82681531 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8965998006553 _step‚784 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8980017969152 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚81407803 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 JDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.cu`ë%³(!rrent‚820 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2743316022370304 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2751096834085405 _step‚785 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚81416009 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚82680036 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 trainer/global_step‚9549 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 164096045 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2742131257212928 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚15827.605163355 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2751098204113807 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚82690023 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11587885983744 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 164096045 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚81416009 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2743316048543744 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2743316048543744 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8980943004672 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2752296965374976 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚82690023 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚82680036 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚51569629 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11587924707328 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 164106038 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚273 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2742131282640896 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640šˆž¯'²'ŽN U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2752298336945152 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8966921472911 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8965576872477 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚741371380879360 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚38723584 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2752298336945152 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8980943004672 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2743316022370304 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2751096834085405 _step‚785 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚81416009 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚82680036 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 trainer/global_step‚9549 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 164096045 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2742131257212928 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚15827.605163355 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2751098204113807 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚82690023 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11587885983744 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 164096045 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚81416009 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2743316048543744 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2743316048543744 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8980943004672 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2752296965374976 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚82690023 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚82680036 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚51569629 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11587924707328 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 164106038 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚273 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2742131282640896 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640‘­ÛSÙ4³'/ _wandbruntime‚15827  _runtime‚15827nÇ$] ´'ƒ 0loss/FRt2/assignment_loss‚0.5514212386677689 /loss/FBt1/detection_loss‚0.4854949382895773 _step‚786 2loss/FBt1/assignment_loss‚0.007697962932362595 1loss/SRqqt2/detection_loss‚0.6158547727051135 % _timestamp‚1.7843056434730096e+09 _runtime‚15827.689957536 2loss/SRqqt2/assignment_loss‚0.3717708465759244 /loss/FRt2/detection_loss‚1.4648075236476321 1loss/SRqqt1/detection_loss‚0.7885747338478459 /loss/FBt2/detection_loss‚0.5332765611627356 epoch‚23 trainer/global_step‚9549 0loss/FRt1/assignment_loss‚1.3460265819246238 1loss/SRqqt1/assignment_loss‚0.373667525410135 3loss/FBt2/assignment_loss‚0.0018286283337187285 .loss/FRt1/detection_loss‚0.481977327423522 $loss/total_loss‚7.02239864092096pÖ: µ'ˆ /loss/FBt1/detection_loss‚0.4854949382895773 _step‚786 2loss/FBt1/assignment_loss‚0.007697962932362595 1loss/SRqqt2/detection_loss‚0.6158547727051135 % _timestamp‚1.7843056434730096e+09 _runtime‚15827.689957536 2loss/SRqqt2/assignment_loss‚0.3717708465759244 /loss/FRt2/detection_loss‚1.4648075236476321 1loss/SRqqt1/detection_loss‚0.7885747338478459 /loss/FBt2/detection_loss‚0.5332765611627356 epoch‚23 trainer/global_step‚9549 0loss/FRt1/assignment_loss‚1.3460265819246238 1loss/SRqqt1/assignment_loss‚0.373667525410135 3loss/FBt2/assignment_loss‚0.0018286283337187285 .loss/FRt1/detection_loss‚0.481977327423522 $loss/total_loss‚7.02239864092096 0loss/FRt2/assignment_loss‚0.5514212386677689’!,㔶'‚(:‰ ÷¯éÒîÉðö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚293.271' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚58.6542 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÊÁÒÇ?·'‚(:´ ÷¯éÒõ¶Íù disk.sda1.out‚0 disk.sda2.out‚0 disk.dm-1.out‚ 801.20703125$ disk.dm-0.out‚134603.41943359375) disk.nvme0c0n1.out‚134603.41943359375 disk.sda.in‚ 75.31640625 disk.dm-0.in‚36281.26953125* proc.memory.percent‚17.119550704956055 cpu‚25.57725936066564' disk.nvme0n1.out‚134603.41943359375 disk.sda2.in‚0 disk.rbd1.in‚ 0.51220703125" proc.memory.rssMB‚ 2804.8671875 proc.cpu.threads‚17 network.sent‚ 1.828796e+07$ disk./.usageGB‚561.4631233215332 disk.dm-1.in‚ 75.31640625 disk.sda3.out‚ 801.20703125 disk.drbd2165.out‚0 disk.rbd3.out‚0$ disk.nvme0c0n1.in‚36281.26953125$ memory_percent‚83.20140838623047 network.recv‚ 3.953095e+06) disk./.usagePercent‚37.68026130504695 disk.sda.out‚ 801.20703125& disk.rbd0.in‚1.759218604441423e+13 disk.sda1.in‚0 disk.drbd2165.in‚0 disk.sda3.in‚ 75.31640625" disk.nvme0n1.in‚36281.26953125 disk.rbd0.out‚0 disk.rbd3.in‚0 disk.rbd1.out‚ 41.65234375& proc.memory.availableMB‚ 2752.28125¢ð@’¸'‚(:‡ †°éÒÇÁ›÷ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚344.24' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚68.848 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0;XÅ>¹'‚(:³ †°éÒƒéÚù disk.rbd3.out‚0) disk.nvme0c0n1.out‚134616.72607421875 disk.sda2.out‚0 disk.drbd2165.out‚0' disk.nvme0n1.out‚134616.72607421875 disk.sda3.out‚ 803.0546875 network.recv‚ 3.956698e+06 disk.sda.in‚ 75.31640625 disk.drbd2165.in‚0$ memory_percent‚83.20016860961914 proc.cpu.threads‚17$ disk./.usageGB‚561.4701347351074 disk.sda2.in‚0& disk.rbd0.in‚1.759218604441423e+13 disk.rbd0.out‚0 disk.sda3.in‚ 75.31640625' proc.memory.availableMB‚ 2752.484375* proc.memory.percent‚17.119550704956055 disk.sda.out‚ 803.0546875 disk.dm-1.in‚ 75.31640625 disk.sda1.out‚0" proc.memory.rssMB‚ 2804.8671875) disk./.usagePercent‚37.68073184689488$ disk.nvme0c0n1.in‚36281.48828125" disk.nvme0n1.in‚36281.48828125 disk.rbd1.in‚ 0.51220703125 disk.dm-0.in‚36281.48828125 disk.sda1.in‚0 network.sent‚ 1.8350891e+07 disk.rbd3.in‚0 disk.rbd1.out‚ 41.65234375 disk.dm-1.out‚ 803.0546875$ disk.dm-0.out‚134616.72607421875 cpu‚25.57723326009124¸ð— Ÿº'‚(:” •°éÒݬ£÷ gpu.0.gpu‚99.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚284.382' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚56.876400000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0û<û8/»'‚(:¤ •°éÒ—½ú network.sent‚ 1.8357658e+07 disk.drbd2165.out‚0$ memory_percent‚83.20324420928955) proc.memory.availableMB‚ 2751.98046875" proc.memory.rssMB‚ 2804.8671875 disk.sda.out‚ 803.36328125 disk.drbd2165.in‚0& disk.nvme0n1.out‚134629.1142578125 disk.sda2.out‚0 proc.cpu.threads‚17 disk.dm-1.out‚ 803.36328125 disk.sda3.in‚ 75.31640625 disk.rbd0.out‚0 disk.dm-0.in‚36411.75 disk.nvme0n1.in‚36411.75# disk.dm-0.out‚134629.1142578125* proc.memory.percent‚17.119550704956055 cpu‚25.577189275976586 network.recv‚ 3.961454e+06$ disk./.usageGB‚561.4763298034668 disk.sda3.out‚ 803.36328125 disk.rbd3.in‚0 disk.rbd3.out‚0( disk.nvme0c0n1.out‚134629.1142578125 disk.sda2.in‚0 disk.dm-1.in‚ 75.31640625 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd1.in‚ 0.51220703125 disk.rbd1.out‚ 41.71484375) disk./.usagePercent‚37.68114760313048 disk.sda.in‚ 75.31640625& disk.rbd0.in‚1.759218604441423e+13 disk.nvme0c0n1.in‚36411.75s¶,KŸ¼'‚(:” ¤°éÒãÀóö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚291.919' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚58.383799999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Øà= 6½'‚(:« ¤°éÒó‰Ñù disk.sda3.in‚ 75.31640625" disk.nvme0c0n1.in‚ 36411.953125 proc.cpu.threads‚17 disk.sda2.out‚0 disk.rbd1.in‚ 0.51220703125$ disk.dm-0.out‚134639.30712890625$ memory_percent‚83.19506645202637( proc.memory.availableMB‚ 2753.3203125 network.recv‚ 3.96401e+06 disk.dm-1.in‚ 75.31640625 disk.dm-1.out‚ 803.703125 disk.drbd2165.in‚0 disk.rbd3.out‚0 cpu‚25.577103574975837& disk.rbd0.in‚1.759218604441423e+13 disk.rbd0.out‚0 disk.sda1.out‚0 disk.sda3.out‚ 803.703125 disk.drbd2165.out‚0) disk.nvme0c0n1.out‚134639.30712890625* proc.memory.percent‚17.119550704956055) disk./.usagePercent‚37.68144892399828' disk.nvme0n1.out‚134639.30712890625$ disk./.usageGB‚561.4808197021484 disk.nvme0n1.in‚ 36411.953125 disk.sda2.in‚0 disk.sda1.in‚0 disk.rbd3.in‚0 disk.rbd1.out‚ 41.71484375 disk.sda.in‚ 75.31640625 disk.sda.out‚ 803.703125 network.sent‚ 1.8360476e+07 disk.dm-0.in‚ 36411.953125" proc.memory.rssMB‚ 2804.8671875\±¹Ÿ”¾'‚(:‰ ³°éÒá©öö gpu.0.gpu‚46.0 gpu.0.memory‚15, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚337.393' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚67.4786 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¨dãå0¿'‚(:¥ ³°éÒÍΪù disk.rbd3.out‚0" disk.nvme0n1.in‚36412.38671875 cpu‚25.577063332279383 disk.dm-0.out‚ 134648.203125 disk.drbd2165.out‚0 disk.sda.out‚ 804.04296875% proc.memory.availableMB‚ 2753.3125* proc.memory.percent‚17.119550704956055) disk./.usagePercent‚37.68197373944348 disk.rbd0.out‚0 proc.cpu.threads‚17" disk.nvme0n1.out‚ 134648.203125 disk.dm-1.in‚ 75.31640625 disk.dm-1.out‚ 804.04296875& disk.rbd0.in‚1.759218604441423e+13 network.recv‚ 3.967575e+06 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.sda2.out‚0 disk.rbd1.in‚ 0.51220703125 disk.rbd1.out‚ 41.77734375$ memory_percent‚83.19511413574219 disk.dm-0.in‚36412.38671875 disk.sda3.in‚ 75.31640625 disk.sda3.out‚ 804.04296875 disk.rbd3.in‚0$ disk.nvme0c0n1.in‚36412.38671875$ disk.nvme0c0n1.out‚ 134648.203125 disk.sda2.in‚0" proc.memory.rssMB‚ 2804.8671875 disk.sda1.in‚0 disk.sda.in‚ 75.31640625 network.sent‚ 1.8365588e+07# disk./.usageGB‚561.488639831543‡°¹4À'/ _wandbruntime‚15905  _runtime‚15905o¿{ÿÈÁ' & lr-AdamW/pg2‚0.0003791961067950757 trainer/global_step‚9599 % _timestamp‚1.7843057216578283e+09 _runtime‚15905.874927935 _step‚787 & lr-AdamW/pg1‚0.0003791961067950757QÀg6ÍÂ'Ç & lr-AdamW/pg1‚0.0003791961067950757 & lr-AdamW/pg2‚0.0003791961067950757 trainer/global_step‚9599 % _timestamp‚1.7843057216578283e+09 _runtime‚15905.874927935 _step‚787“ü•‚<4Ã'/ _wandbruntime‚15905  _runtime‚15905 ìUæÄ'€P VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2765964392045568 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2765964392045568 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚81818103 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚27206903 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11645992626176 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11645955992576 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2765965763608576 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚81818109 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚83097310 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 164925392 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚756814207481856 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚83097310 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚788 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2764758354862891 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚24622118 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404724736 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚745168214855680 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2755747464343552 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚51828747 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚745167846764544 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9010890519339 trainer/global_step‚9599 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 164915413 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚51829021 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚269 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2764759724891181 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 % _timestamp‚1.7843057216615326e+09 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9027679289344 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚83107283 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚83107283 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2756938058145792 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2755747489771520 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2755747464343552 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚51828747 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚745167846764544 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9010890519339 trainer/global_step‚9599 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 164915413 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚51829021 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚269 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2764759724891181 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 % _timestamp‚1.7843057216615326e+09 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9027679289344 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚83107283 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚83107283 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2756938058145792 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2755747489771520 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2765964392045568 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚81818103 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚27206903 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11645992626176 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11645955992576 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2765965763608576 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚81818109 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚83097310 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 164925392 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚756814207481856 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚83097310 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚788 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2764758354862891 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚24622118 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404724736 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚745168214855680 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 164942090 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚745245333172224 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚83105788 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 164932097 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9013158586019 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚51834053 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9027258935296 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2765037160107313 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9027258935296 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚265 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚745245701263360 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚83115775 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404717568 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11647144177664 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9028604332032 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2766243317811200 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚81826309 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚260 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9011813985585 _step‚789 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 164932097 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 trainer/global_step‚9599 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚81826309 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚51834318 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 _runtime‚15907.264337806 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2757216058875904 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚27209397 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 164942090 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0ý’=nÈ'N D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 164932097 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 trainer/global_step‚9599 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚81826309 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚51834318 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 _runtime‚15907.264337806 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2757216058875904 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚27209397 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 164942090 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2766244689381376 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2756025346121728 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚81826315 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚24624656 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 164942090 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚745245333172224 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚83105788 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 164932097 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9013158586019 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K>DeviceStatŽQ§ sMonitor.on_train_batch_end/inactive_split.all.freed‚51834053 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9027258935296 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2765037160107313 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9027258935296 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚265 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚745245701263360 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚83115775 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404717568 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11647144177664 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9028604332032 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2766243317811200 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚81826309 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚260 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9011813985585 _step‚789•—ž4É'/ _wandbruntime‚15907  _runtime‚159070¦¸ìÊ'ˆ 3loss/SRqqt2/assignment_loss‚0.34708071109881344 /loss/FRt2/detection_loss‚1.5121751932281222 trainer/global_step‚9599 _step‚790 0loss/FRt1/assignment_loss‚1.4038505420680307 %loss/total_loss‚7.006896067904055 0loss/FRt2/assignment_loss‚0.5626864867939066 3loss/SRqqt1/assignment_loss‚0.47402091261877366 4loss/FBt1/assignment_loss‚0.00034130711844412566 1loss/SRqqt2/detection_loss‚0.5329673270810464 /loss/FBt2/detection_loss‚0.4113850702521439 epoch‚23 2loss/FBt2/assignment_loss‚0.010443189051521373 0loss/FRt1/detection_loss‚0.49614298817450514 1loss/SRqqt1/detection_loss‚0.6606929435798105 /loss/FBt1/detection_loss‚0.5951093968389363 # _timestamp‚1.78430572313254e+09 _runtime‚15907.349479169zëàªË' 2loss/FBt2/assignment_loss‚0.010443189051521373 0loss/FRt1/detection_loss‚0.49614298817450514 1loss/SRqqt1/detection_loss‚0.6606929435798105 /loss/FBt1/detection_loss‚0.5951093968389363 # _timestamp‚1.78430572313254e+09 _runtime‚15907.349479169 3loss/SRqqt2/assignment_loss‚0.34708071109881344 /loss/FRt2/detection_loss‚1.5121751932281222 trainer/global_step‚9599 _step‚790 0loss/FRt1/assignment_loss‚1.4038505420680307 %loss/total_loss‚7.006896067904055 0loss/FRt2/assignment_loss‚0.5626864867939066 3loss/SRqqt1/assignment_loss‚0.47402091261877366 4loss/FBt1/assignment_loss‚0.00034130711844412566 1loss/SRqqt2/detection_loss‚0.5329673270810464 /loss/FBt2/detection_loss‚0.4113850702521439 epoch‚23–†7”Ì'‚(:‰ °éÒäÌŽ÷ gpu.0.gpu‚97.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚341.052' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚68.2104 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¤J‡AÍ'‚(:¶ °éÒÊÇú$ disk.dm-0.out‚134665.43017578125# memory_percent‚83.1946849822998 proc.cpu.threads‚17) disk./.usagePercent‚37.68230910931333) disk.nvme0c0n1.out‚134665.43017578125" disk.nvme0n1.in‚36412.60546875 disk.rbd1.in‚ 0.51220703125 disk.sda.in‚ 75.31640625 disk.dm-1.in‚ 75.31640625 disk.sda3.out‚ 804.64453125* proc.memory.percent‚17.119550704956055 network.recv‚ 3.970771e+06$ disk.nvme0c0n1.in‚36412.60546875 disk.sda2.out‚0 disk.sda1.out‚0 disk.sda3.in‚ 75.31640625 network.sent‚ 1.8370655e+07 disk.sda2.in‚0 disk.sda.out‚ 804.64453125( proc.memory.availableMB‚ 2753.3828125 disk.rbd3.out‚0 disk.rbd1.out‚ 41.77734375 disk.drbd2165.in‚0 disk.drbd2165.out‚0" proc.memory.rssMB‚ 2804.8671875$ disk./.usageGB‚561.4936370849609 disk.rbd3.in‚0 cpu‚25.57785110432645 disk.dm-0.in‚36412.60546875 disk.sda1.in‚0' disk.nvme0n1.out‚134665.43017578125 disk.dm-1.out‚ 804.64453125& disk.rbd0.in‚1.759218604441423e+13 disk.rbd0.out‚0X‡‡žÎ'‚(:“ Ñ°éÒ¶¤÷ gpu.0.gpu‚100.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚38.0 gpu.0.powerWatts‚137.8' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚27.560000000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÝU.0Ï'‚(:¥ Ñ°éÒìÿÛú disk.sda2.out‚0" disk.dm-0.out‚134674.751953125% disk.nvme0n1.out‚134674.751953125) proc.memory.availableMB‚ 2702.06640625 disk.rbd1.in‚ 0.51220703125 disk.dm-1.in‚ 75.31640625 disk.sda1.in‚0) proc.memory.percent‚17.43307113647461 cpu‚25.579797237885646 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.sda.in‚ 75.31640625 disk.dm-1.out‚ 804.86328125 network.sent‚ 1.8433866e+07 network.recv‚ 3.976209e+06& disk.rbd0.in‚1.759218604441423e+13 disk.dm-0.in‚ 36534.0625 disk.sda3.in‚ 75.31640625$ memory_percent‚83.50789546966553 disk.rbd3.in‚0' disk.nvme0c0n1.out‚134674.751953125 disk.sda.out‚ 804.86328125 disk.sda3.out‚ 804.86328125 disk.rbd3.out‚0 disk.nvme0n1.in‚ 36534.0625 proc.cpu.threads‚17 disk.sda2.in‚0 disk.rbd1.out‚ 41.77734375 disk.rbd0.out‚0 disk.drbd2165.out‚0 disk.nvme0c0n1.in‚ 36534.0625! proc.memory.rssMB‚ 2856.234375) disk./.usagePercent‚37.68275584246058$ disk./.usageGB‚561.5002937316895px¾4Ð'/ _wandbruntime‚15935  _runtime‚15935Vw$‚ž=Ñ'˜{ 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5204131007194519 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.04503845050930977 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.019714532420039177 particle/accuracy_4_of_5‚1 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.5275102853775024 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.18627379834651947 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.03774959221482277 (particle/accuracy‚0.8460130095481873 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9977869391441345 . Purity/2FRt0SRqqt1FBt/FRt_purity‚ 0.6328125 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9951693415641785 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.17183277010917664 7"Purity/*FRt1SRqqt*FBt/event_purity‚0.31193807721138 _runtime‚15935.451085108 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 *jet/accuracy_1_of_4‚0.9598044157028198 jet/accuracy_4_of_5‚0 particle/accuracy_3_of_5‚1 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.02765931747853756 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8348073363304138 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9398148059844971 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.4955384433269501 8"Purity/0FRt1SRqqt*FBt/event_purity‚0.515718936920166 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5940409302711487 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 6 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.998410165309906 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.6554129123687744 +particle/sensitivity‚0.6210513710975647 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 epoch‚23 8"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.551692545413971 9"Purity/2FRt*SRqqt1FBt/event_purity‚0.1316666603088379 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.2864583134651184 particle/accuracy_1_of_5‚1 8"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.514376699924469 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6581560969352722 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9922723174095154 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 /particle/accuracy_3_of_4‚0.9974937438964844 6validation_average_jet_accuracy‚0.3323717713356018 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.48804032802581787 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.14930394291877747 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 +jet/accuracy_3_of_3‚0.07413727045059204 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 _step‚791 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.12037037312984467 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8285526037216187 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9976841807365417 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9989176988601685 *jet/accuracy_1_of_1‚0.2906622290611267 6 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.942307710647583 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 particle/accuracy_2_of_5‚1 particle/accuracy_5_of_5‚0 , Purity/0FRt2SRqqt*FBt/FBt_purity‚0.96875 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7197318077087402 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.22626395523548126 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9911719560623169 -"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4375 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.6507231593132019 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7142857313156128 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.8571428656578064 jet/accuracy_1_of_5‚1 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.23869527876377106 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4745370149612427 particle/accuracy_1_of_4‚1 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.2142527848482132 *jet/accuracy_1_of_3‚0.8867884874343872 jet/accuracy_3_of_5‚1 )loss/val_total_loss‚8.080052069387632 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5751488208770752 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9943593144416809 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9959182739257812 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6176688075065613 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8654665946960449 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.23059317469596863 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 trainer/global_step‚9599 % _timestamp‚1.7843057512330546e+09 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 6 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.988095223903656 *jet/accuracy_2_of_3‚0.4919077157974243 6 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.986643373966217 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9900732636451721 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9961844086647034 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 7 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.1034887433052063 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.23879927396774292 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7475505471229553 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 jet/accuracy_2_of_5‚1 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.6330689191818237 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.42824074625968933 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9968253374099731 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.10209041833877563 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6503148078918457 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.22568239271640778 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.45709383487701416 , Purity/0FRt2SRqqt1FBt/FBt_purity‚0.96875 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.5649652481079102 8"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.685211181640625 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.14644667506217957 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9814814329147339 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9222812652587891 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 8"Purity/0FRt1SRqqt1FBt/event_purity‚0.566839337348938 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6034979820251465 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.12037037312984467 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.26244083046913147 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.7382786273956299 particle/accuracy_1_of_3‚1 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9338803291320801 8"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.545585036277771 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.3146618902683258 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.5864487290382385 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6580306887626648 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2760090231895447 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.19950254261493683 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6245524287223816 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.1849830448627472 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8287226557731628 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 particle/accuracy_2_of_4‚1 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6505759358406067 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.6290258765220642 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.11538461595773697 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.2878784239292145 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9814814329147339 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.4919798672199249 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8888888955116272 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9989176988601685 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 jet/accuracy_5_of_5‚0 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3052431643009186 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.2222222238779068 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.10376834869384766 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.22196760773658752 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 8"Purity/0FRt0SRqqt*FBt/event_purity‚0.995028555393219 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.33587807416915894 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.25010746717453003 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.1712568998336792 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5525000095367432 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.10376834869384766 6 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.209212526679039 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9801465272903442 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9934239983558655 particle/accuracy_3_of_3‚1 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7475956082344055 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8366591930389404 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03281472250819206 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4848506450653076 *jet/accuracy_2_of_4‚0.7247831225395203 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5313621163368225 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8009259104728699 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2384493201971054 ,jet/accuracy_4_of_4‚0.013899499550461769 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.053128816187381744 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.5808358788490295 particle/accuracy_2_of_2‚1 'particle/f_score‚0.6340034604072571 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9909418821334839 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.5819425582885742 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.2176433503627777 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6070650219917297 +jet/accuracy_2_of_2‚0.13821475207805634 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.28947919607162476 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.22089283168315887 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7550832033157349 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.625 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9965855479240417 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.20858117938041687 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8724216222763062 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.5647763013839722 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.7382786273956299 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8976695537567139 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.11538461595773697 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8461451530456543 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.3259134292602539 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 *jet/accuracy_1_of_2‚0.7146170139312744 particle/accuracy_2_of_3‚1 /particle/accuracy_4_of_4‚0.9409509301185608 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.22626395523548126 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.2222222238779068 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.5683695673942566 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9993098974227905 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.6545366644859314 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.1820434331893921 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 particle/accuracy_1_of_1‚1 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.8125 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.5879195332527161 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 particle/accuracy_1_of_2‚1 +particle/specificity‚0.9075427055358887 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 8"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.658951997756958 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.10382819175720215 8"Purity/0FRt1SRqqt0FBt/event_purity‚0.514376699924469 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5826796293258667 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.5922213196754456 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8009259104728699 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.21816599369049072 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.23782873153686523 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 +jet/accuracy_3_of_4‚0.09714484214782715 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9855948686599731 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.10468605905771255Aª(úB#Ò'{ =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.02765931747853756 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8348073363304138 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9398148059844971 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.4955384433269501 8"Purity/0FRt1SRqqt*FBt/event_purity‚0.515718936920166 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5940409302711487 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 6 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.998410165309906 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.6554129123687744 +particle/sensitivity‚0.6210513710975647 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 epoch‚23 8"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.551692545413971 9"Purity/2FRt*SRqqt1FBt/event_purity‚0.1316666603088379 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.2864583134651184 particle/accuracy_1_of_5‚1 8"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.514376699924469 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6581560969352722 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9922723174095154 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 /particle/accuracy_3_of_4‚0.9974937438964844 6validation_average_jet_accuracy‚0.3323717713356018 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.48804032802581787 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.14930394291877747 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 +jet/accuracy_3_of_3‚0.07413727045059204 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 _step‚791 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.12037037312984467 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8285526037216187 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9976841807365417 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9989176988601685 *jet/accuracy_1_of_1‚0.2906622290611267 6 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.942307710647583 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 particle/accuracy_2_of_5‚1 particle/accuracy_5_of_5‚0 , Purity/0FRt2SRqqt*FBt/FBt_purity‚0.96875 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7197318077087402 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.22626395523548126 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9911719560623169 -"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4375 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.6507231593132019 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7142857313156128 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.8571428656578064 jet/accuracy_1_of_5‚1 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.23869527876377106 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4745370149612427 particle/accuracy_1_of_4‚1 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.2142527848482132 *jet/accuracy_1_of_3‚0.8867884874343872 jet/accuracy_3_of_5‚1 )loss/val_total_loss‚8.080052069387632 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5751488208770752 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9943593144416809 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9959182739257812 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6176688075065613 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8654665946960449 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.23059317469596863 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 trainer/global_step‚9599 % _timestamp‚1.7843057512330546e+09 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 6 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.988095223903656 *jet/accuracy_2_of_3‚0.4919077157974243 6 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.986643373966217 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9900732636451721 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9961844086647034 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 7 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.1034887433052063 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.23879927396774292 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7475505471229553 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 jet/accuracy_2_of_5‚1 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.6330689191818237 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.42824074625968933 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9968253374099731 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.10209041833877563 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6503148078918457 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.22568239271640778 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.45709383487701416 , Purity/0FRt2SRqqt1FBt/FBt_purity‚0.96875 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.5649652481079102 8"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.685211181640625 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.14644667506217957 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9814814329147339 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9222812652587891 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 8"Purity/0FRt1SRqqt1FBt/event_purity‚0.566839337348938 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6034979820251465 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.12037037312984467 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.26244083046913147 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.7382786273956299 particle/accuracy_1_of_3‚1 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9338803291320801 8"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.545585036277771 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.3146618902683258 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.5864487290382385 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6580306887626648 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2760090231895447 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.19950254261493683 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6245524287223816 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.1849830448627472 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8287226557731628 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 particle/accuracy_2_of_4‚1 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6505759358406067 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.6290258765220642 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.11538461595773697 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.2878784239292145 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9814814329147339 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.4919798672199249 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8888888955116272 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9989176988601685 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 jet/accuracy_5_of_5‚0 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3052431643009186 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.2222222238779068 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.10376834869384766 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.22196760773658752 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 8"Purity/0FRt0SRqqt*FBt/event_purity‚0.995028555393219 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 :"Pu2k-Warity/0FRt2SRqqt0FBt/event_purity‚0.33587807416915894 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.25010746717453003 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.1712568998336792 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5525000095367432 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.10376834869384766 6 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.209212526679039 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9801465272903442 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9934239983558655 particle/accuracy_3_of_3‚1 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7475956082344055 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8366591930389404 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03281472250819206 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4848506450653076 *jet/accuracy_2_of_4‚0.7247831225395203 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5313621163368225 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8009259104728699 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.2384493201971054 ,jet/accuracy_4_of_4‚0.013899499550461769 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.053128816187381744 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.5808358788490295 particle/accuracy_2_of_2‚1 'particle/f_score‚0.6340034604072571 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9909418821334839 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.5819425582885742 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.2176433503627777 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6070650219917297 +jet/accuracy_2_of_2‚0.13821475207805634 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.28947919607162476 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.22089283168315887 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7550832033157349 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.625 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9965855479240417 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.20858117938041687 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8724216222763062 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.5647763013839722 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.7382786273956299 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8976695537567139 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.11538461595773697 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8461451530456543 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.3259134292602539 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 *jet/accuracy_1_of_2‚0.7146170139312744 particle/accuracy_2_of_3‚1 /particle/accuracy_4_of_4‚0.9409509301185608 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.22626395523548126 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.2222222238779068 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.5683695673942566 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9993098974227905 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.6545366644859314 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.1820434331893921 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 particle/accuracy_1_of_1‚1 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.8125 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.5879195332527161 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 particle/accuracy_1_of_2‚1 +particle/specificity‚0.9075427055358887 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 8"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.658951997756958 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.10382819175720215 8"Purity/0FRt1SRqqt0FBt/event_purity‚0.514376699924469 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5826796293258667 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.5922213196754456 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8009259104728699 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.21816599369049072 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.23782873153686523 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 +jet/accuracy_3_of_4‚0.09714484214782715 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9855948686599731 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.10468605905771255 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5204131007194519 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.04503845050930977 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.019714532420039177 particle/accuracy_4_of_5‚1 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.5275102853775024 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.18627379834651947 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.03774959221482277 (particle/accuracy‚0.8460130095481873 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9977869391441345 . Purity/2FRt0SRqqt1FBt/FRt_purity‚ 0.6328125 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9951693415641785 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.17183277010917664 7"Purity/*FRt1SRqqt*FBt/event_purity‚0.31193807721138 _runtime‚15935.451085108 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 *jet/accuracy_1_of_4‚0.9598044157028198 jet/accuracy_4_of_5‚0 particle/accuracy_3_of_5‚1—趞Ó'‚(:“ à°éÒ“»§÷ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚297.28' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚59.455999999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0$ÜÊÖ4Ô'‚(:© à°éÒ¨›Èú' disk.nvme0n1.out‚134687.88134765625 disk.sda.in‚ 75.31640625 disk.dm-1.out‚ 805.1328125 disk.rbd0.out‚0 disk.rbd3.out‚0" disk.nvme0c0n1.in‚ 36534.484375' proc.memory.availableMB‚ 2753.578125 disk.nvme0n1.in‚ 36534.484375 disk.rbd1.in‚ 0.51220703125 disk.rbd3.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda1.in‚0# memory_percent‚83.1934928894043 network.recv‚ 3.979727e+06 disk.dm-1.in‚ 75.31640625 disk.dm-0.in‚ 36534.484375 disk.drbd2165.in‚0$ disk./.usageGB‚561.5076293945312$ disk.dm-0.out‚134687.88134765625 disk.drbd2165.out‚0) disk.nvme0c0n1.out‚134687.88134765625& disk.rbd0.in‚1.759218604441423e+13 disk.sda3.out‚ 805.1328125" proc.memory.rssMB‚ 2804.8671875 disk.sda1.out‚0 cpu‚25.58044297671368 network.sent‚ 1.8439e+07) disk./.usagePercent‚37.68324814494893 disk.rbd1.out‚ 41.83984375 disk.sda.out‚ 805.1328125 disk.sda3.in‚ 75.31640625* proc.memory.percent‚17.119550704956055 proc.cpu.threads‚17ÿ íÆ”Õ'‚(:‰ ï°éÒœ‘çö gpu.0.gpu‚84.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚335.937' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚67.1874 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0=‡>è4Ö'‚(:© ï°éҬͭù disk.dm-0.in‚ 36534.703125 disk.drbd2165.in‚0 disk.nvme0n1.in‚ 36534.703125 disk.sda.in‚ 75.31640625* proc.memory.percent‚17.119550704956055# disk./.usageGB‚561.512134552002 disk.drbd2165.out‚0( disk.nvme0c0n1.out‚134699.7939453125 disk.rbd1.in‚ 0.51220703125 disk.sda.out‚ 805.421875 network.sent‚ 1.8496242e+07* disk./.usagePercent‚37.683550489846866# memory_percent‚83.1920862197876 cpu‚25.580449711522064# disk.dm-0.out‚134699.7939453125 disk.dm-1.in‚ 75.31640625 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda3.in‚ 75.31640625 disk.rbd3.in‚0 proc.cpu.threads‚17& disk.nvme0n1.out‚134699.7939453125 disk.sda2.in‚0) proc.memory.availableMB‚ 2753.80859375 disk.rbd0.out‚0 disk.sda3.out‚ 805.421875 disk.rbd1.out‚ 41.83984375& disk.rbd0.in‚1.759218604441423e+13 disk.rbd3.out‚0" disk.nvme0c0n1.in‚ 36534.703125 disk.sda2.out‚0" proc.memory.rssMB‚ 2804.8671875 network.recv‚ 3.984214e+06 disk.dm-1.out‚ 805.421875Øgd×”×'‚(:‰ þ°éÒ€ãŒø gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚260.702' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.1404 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0p—ND;Ø'‚(:° þ°éÒ̸÷ø disk.sda.in‚ 75.31640625 disk.dm-1.in‚ 75.31640625 disk.drbd2165.in‚0 disk.rbd3.out‚0( proc.memory.availableMB‚ 2753.3203125* proc.memory.percent‚17.119550704956055 disk.sda3.out‚ 805.78515625' disk.nvme0n1.out‚134709.89208984375 network.sent‚ 1.8501851e+07) disk.nvme0c0n1.out‚134709.89208984375 disk.nvme0n1.in‚ 36534.90625* disk./.usagePercent‚37.683121933234546$ disk./.usageGB‚561.5057487487793 disk.dm-0.in‚ 36534.90625 disk.sda3.in‚ 75.31640625 cpu‚25.580375672806476 disk.rbd0.out‚0 disk.sda1.out‚0" proc.memory.rssMB‚ 2804.8671875 proc.cpu.threads‚17 disk.sda2.in‚0 disk.sda.out‚ 805.78515625 disk.dm-1.out‚ 805.78515625 disk.rbd1.in‚ 0.51220703125& disk.rbd0.in‚1.759218604441423e+13 disk.sda1.in‚0! disk.nvme0c0n1.in‚ 36534.90625$ memory_percent‚83.19506645202637 disk.drbd2165.out‚0 disk.rbd3.in‚0 network.recv‚ 3.987464e+06 disk.sda2.out‚0 disk.rbd1.out‚ 41.90234375$ disk.dm-0.out‚134709.89208984375]2)Ù'‚(:’ ±éÒ‹¾Ï÷ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚329.85' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚65.97000000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ô2áE=Ú'‚(:² ±éÒ´Åú cpu‚25.580343271427072 disk.drbd2165.out‚0 disk.nvme0n1.in‚ 36610.234375 disk.sda.out‚ 806.33203125 disk.dm-0.in‚ 36610.234375 disk.sda3.out‚ 806.33203125$ disk./.usageGB‚561.5126113891602 disk.sda1.out‚0 disk.rbd3.out‚0" disk.nvme0c0n1.in‚ 36610.234375 disk.sda3.in‚ 75.31640625 disk.sda.in‚ 75.31640625 disk.dm-1.out‚ 806.33203125 network.recv‚ 3.99134e+06 disk.rbd0.out‚0) disk.nvme0c0n1.out‚134722.26416015625 disk.sda2.in‚0 disk.rbd1.in‚ 0.51220703125 disk.rbd1.out‚ 41.90234375( proc.memory.availableMB‚ 2753.5703125 proc.cpu.threads‚17$ disk.dm-0.out‚134722.26416015625" proc.memory.rssMB‚ 2804.8671875' disk.nvme0n1.out‚134722.26416015625 disk.sda2.out‚0& disk.rbd0.in‚1.759218604441423e+13 disk.sda1.in‚0 disk.drbd2165.in‚0 disk.rbd3.in‚0$ memory_percent‚83.19354057312012* proc.memory.percent‚17.119550704956055 network.sent‚ 1.8507127e+07* disk./.usagePercent‚37.683582490788645 disk.dm-1.in‚ 75.31640625 OwžŸÛ'‚(:” œ±éÒû¼ ø gpu.0.gpu‚90.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚273.863' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚54.772600000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0EæýQ;Ü'‚(:° œ±éÒÓ¡Çû disk.sda1.out‚0 disk.rbd3.in‚0 disk.sda2.out‚0$ memory_percent‚83.19449424743652* proc.memory.percent‚17.119550704956055 network.recv‚ 3.994428e+06 disk.sda.out‚ 806.74609375 disk.sda3.out‚ 806.74609375 disk.drbd2165.out‚0) disk.nvme0c0n1.out‚134733.82666015625 disk.rbd1.in‚ 0.51220703125 cpu‚25.580294490730616 network.sent‚ 1.851168e+07 disk.rbd0.out‚0 disk.dm-0.in‚ 36610.453125 disk.sda1.in‚0" disk.nvme0c0n1.in‚ 36610.453125 proc.cpu.threads‚17& disk.rbd0.in‚1.759218604441423e+13 disk.sda3.in‚ 75.31640625 disk.rbd1.out‚ 41.96484375( disk./.usagePercent‚37.6838909798674 disk.sda.in‚ 75.31640625 disk.drbd2165.in‚0" proc.memory.rssMB‚ 2804.8671875$ disk./.usageGB‚561.5172080993652 disk.dm-1.in‚ 75.31640625 disk.rbd3.out‚0 disk.nvme0n1.in‚ 36610.453125' disk.nvme0n1.out‚134733.82666015625 disk.dm-1.out‚ 806.74609375$ disk.dm-0.out‚134733.82666015625 disk.sda2.in‚0( proc.memory.availableMB‚ 2753.4140625ºUüÐ4Ý'/ _wandbruntime‚16013  _runtime‚16013ÏkoÉÞ'à ' lr-AdamW/pg1‚0.00037643559206404617 ' lr-AdamW/pg2‚0.00037643559206404617 trainer/global_step‚9649 $ _timestamp‚1.784305829609795e+09 _runtime‚16013.830165439 _step‚792$åù"Îß'È _runtime‚16013.830165439 _step‚792 ' lr-AdamW/pg1‚0.00037643559206404617 ' lr-AdamW/pg2‚0.00037643559206404617 trainer/global_step‚9649 $ _timestamp‚1.784305829609795e+09˜7å…]4à'/ _wandbruntime‚16013  _runtime‚16013†¤¬(á'€P LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9078492442112 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚83592017 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚749090647486464 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 _runtime‚16013.831165314 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 165952432 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚83592017 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2782530418948844 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2783744568528384 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚52135732 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 trainer/global_step‚9649 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2773467462184960 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚82360415 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11713315956224 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2782531788977134 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2783744568528384 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9079837831680 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚237 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚83582044 _step‚793 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 165942453 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2774666102259712 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚27377551 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9062956763884 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚82360409 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2774666102259712 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚760803997979136 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚83592017 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2782530418948844 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2783744568528384 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚52135732 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 trainer/global_step‚9649 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2773467462184960 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚82360415 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11713315956224 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2782531788977134 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 17196646:.S¨ 40 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2783744568528384 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9079837831680 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚237 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚83582044 _step‚793 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 165942453 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2774666102259712 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚27377551 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9062956763884 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚82360409 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2774666102259712 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚760803997979136 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9079417477632 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 165959137 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚52141302 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚52141059 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚27380335 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 165969130 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2774944076816384 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚24760729 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 _runtime‚16015.217898561 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚82368615 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 165969130 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚749167765803008 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚82368621 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2773745369391104 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚83600509 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚760882691358208 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚82368615 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚794 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11714557464064 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚83590522 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚27380573 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008—ük'å'N EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 165969130 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚749167765803008 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚82368621 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2773745369391104 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚83600509 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚760882691358208 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚82368615 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚794 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11714557464064 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚83590522 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚27380573 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9080762874368 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2784024865864192 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚749168133894144 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9079417477632 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 165959137 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚52141302 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚52141059 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚27380335 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 165969130 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2774944076816384 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚24760729 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 _runtime‚16015.217898561 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚82368615 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175šmfΓ4æ'/ _wandbruntime‚16015  _runtime‚16015Rk¥ ç'‡ 1loss/FRt2/assignment_loss‚0.39875628042346883 3loss/SRqqt2/assignment_loss‚0.30234766866836743 1loss/FBt2/assignment_loss‚0.00659285117000479 %loss/total_loss‚6.168193246129482 _step‚795 0loss/FRt1/assignment_loss‚1.2136486073048727 1loss/SRqqt1/detection_loss‚0.6314373698878843 1loss/SRqqt2/detection_loss‚0.5337815570407285 0loss/FBt2/detection_loss‚0.36145771767253115 % _timestamp‚1.7843058310861363e+09 2loss/SRqqt1/assignment_loss‚0.3125430107190842 0loss/FRt1/detection_loss‚0.45755480060832027 epoch‚24 trainer/global_step‚9649 _runtime‚16015.303084187 1loss/FBt1/assignment_loss‚0.01348178545976724 /loss/FRt2/detection_loss‚1.4050618365239893 /loss/FBt1/detection_loss‚0.5315297606504625 ):Ûè'Œ epoch‚24 trainer/global_step‚9649 _runtime‚16015.303084187 1loss/FBt1/assignment_loss‚0.01348178545976724 /loss/FRt2/detection_loss‚1.4050618365239893 /loss/FBt1/detection_loss‚0.5315297606504625 1loss/FRt2/assignment_loss‚0.39875628042346883 3loss/SRqqt2/assignment_loss‚0.30234766866836743 1loss/FBt2/assignment_loss‚0.00659285117000479 %loss/total_loss‚6.168193246129482 _step‚795 0loss/FRt1/assignment_loss‚1.2136486073048727 1loss/SRqqt1/detection_loss‚0.6314373698878843 1loss/SRqqt2/detection_loss‚0.5337815570407285 0loss/FBt2/detection_loss‚0.36145771767253115 % _timestamp‚1.7843058310861363e+09 2loss/SRqqt1/assignment_loss‚0.3125430107190842 0loss/FRt1/detection_loss‚0.45755480060832027›Óî\«Ÿé'‚(:” «±éÒíþ÷ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚285.002' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚57.000400000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0RÕÿÒ4ê'‚(:© «±éÒ·âöù disk.dm-1.in‚ 75.31640625 network.recv‚ 3.99806e+06) disk./.usagePercent‚37.68549153897147 disk.sda.out‚ 807.08984375 disk.rbd0.out‚0 disk.sda3.in‚ 75.31640625$ disk./.usageGB‚561.5410575866699 disk.rbd1.out‚ 41.96484375 disk.sda1.in‚0 disk.drbd2165.in‚0 disk.sda.in‚ 75.31640625 disk.dm-1.out‚ 807.08984375 disk.sda3.out‚ 807.08984375 disk.nvme0c0n1.in‚ 36610.875) disk.nvme0c0n1.out‚134743.40673828125$ memory_percent‚83.19456577301025 disk.sda2.out‚0& disk.rbd0.in‚1.759218604441423e+13$ disk.dm-0.out‚134743.40673828125 disk.rbd3.in‚0) proc.memory.availableMB‚ 2753.40234375* proc.memory.percent‚17.119550704956055 network.sent‚ 1.8516757e+07 disk.sda2.in‚0 disk.rbd1.in‚ 0.51220703125 disk.rbd3.out‚0 disk.nvme0n1.in‚ 36610.875' disk.nvme0n1.out‚134743.40673828125 proc.cpu.threads‚17 disk.dm-0.in‚ 36610.875 disk.sda1.out‚0 disk.drbd2165.out‚0" proc.memory.rssMB‚ 2804.8671875 cpu‚25.580178064431028{Ò_ •ë'‚(:Š º±éÒýŠïö gpu.0.gpu‚100.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚335.402' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚67.0804 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0e¨á<ì'‚(:± º±éÒ¸’§ù disk.nvme0n1.in‚ 36611.078125 network.sent‚ 1.8580663e+07 disk.dm-1.out‚ 807.31640625$ disk.dm-0.out‚134759.79443359375 disk.sda1.in‚0 disk.rbd1.out‚ 42.02734375& disk.rbd0.in‚1.759218604441423e+13$ memory_percent‚83.19439888000488* proc.memory.percent‚17.119550704956055 proc.cpu.threads‚17 disk.rbd1.in‚ 0.51220703125 disk.rbd0.out‚0 network.recv‚ 4.002433e+06# disk./.usageGB‚561.545223236084 disk.dm-1.in‚ 75.31640625 disk.rbd3.in‚0 disk.rbd3.out‚0 cpu‚25.580198663727373 disk.sda2.out‚0 disk.sda1.out‚0 disk.sda3.in‚ 75.31640625 disk.drbd2165.in‚0 disk.drbd2165.out‚0' disk.nvme0n1.out‚134759.79443359375) disk./.usagePercent‚37.68577109919887 disk.sda.in‚ 75.31640625 disk.dm-0.in‚ 36611.078125" disk.nvme0c0n1.in‚ 36611.078125( proc.memory.availableMB‚ 2753.4296875" proc.memory.rssMB‚ 2804.8671875 disk.sda.out‚ 807.31640625 disk.sda2.in‚0 disk.sda3.out‚ 807.31640625) disk.nvme0c0n1.out‚134759.79443359375*ÝŸí'‚(:” ɱéÒ§ª…÷ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚283.559' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚56.711800000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¾  %î'‚(:š ɱéÒœÎù disk.drbd2165.out‚0 disk.rbd3.out‚0 disk.nvme0n1.in‚ 36751.15625 disk.rbd1.out‚ 42.02734375 disk.dm-1.in‚ 75.31640625 disk.sda1.out‚0$ memory_percent‚83.19592475891113" proc.memory.rssMB‚ 2804.8671875 proc.cpu.threads‚17 disk.rbd1.in‚ 0.51220703125! disk.nvme0c0n1.in‚ 36751.15625$ disk.dm-0.out‚134768.76025390625 disk.sda1.in‚0 disk.drbd2165.in‚0* proc.memory.percent‚17.119550704956055) disk.nvme0c0n1.out‚134768.76025390625 disk.sda2.in‚0( proc.memory.availableMB‚ 2753.1796875 cpu‚25.580138372550934 disk.sda2.out‚0 disk.sda.out‚807.5' disk.nvme0n1.out‚134768.76025390625 disk.sda.in‚ 75.31640625 disk.dm-1.out‚807.5& disk.rbd0.in‚1.759218604441423e+13 disk.rbd0.out‚0 disk.dm-0.in‚ 36751.15625 disk.sda3.out‚807.5 disk.rbd3.in‚0 network.sent‚ 1.8585722e+07) disk./.usagePercent‚37.68623421682831 disk.sda3.in‚ 75.31640625 network.recv‚ 4.005868e+06$ disk./.usageGB‚561.5521240234375’É]Êžï'‚(:“ رéÒÝßþö gpu.0.gpu‚96.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚332.989' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚66.59779999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Wé‹ø=ð'‚(:² رéÒÅÁ¼ù network.sent‚ 1.8590256e+07 disk.rbd0.out‚0 disk.rbd3.in‚0 disk.sda.in‚ 75.31640625( proc.memory.availableMB‚ 2753.6015625 disk.sda2.in‚0) disk./.usagePercent‚37.68663282055912 disk.sda1.in‚0) disk.nvme0c0n1.out‚134784.05712890625 disk.sda2.out‚0 disk.dm-1.out‚ 807.69140625& disk.rbd0.in‚1.759218604441423e+13$ disk.dm-0.out‚134784.05712890625 disk.drbd2165.out‚0 disk.rbd3.out‚0 network.recv‚ 4.008955e+06 disk.sda3.in‚ 75.31640625" disk.nvme0c0n1.in‚ 36751.578125 disk.nvme0n1.in‚ 36751.578125$ memory_percent‚83.19334983825684 disk.sda1.out‚0 disk.drbd2165.in‚0' disk.nvme0n1.out‚134784.05712890625 disk.dm-0.in‚ 36751.578125 disk.sda3.out‚ 807.69140625 disk.rbd1.in‚ 0.51220703125 disk.sda.out‚ 807.69140625" proc.memory.rssMB‚ 2804.8671875 proc.cpu.threads‚17$ disk./.usageGB‚561.5580635070801 disk.rbd1.out‚ 42.08984375 disk.dm-1.in‚ 75.31640625* proc.memory.percent‚17.119550704956055 cpu‚25.580095176569962“i:j”ñ'‚(:‰ ç±éҳʉ÷ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚292.107' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚58.4214 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0qÐI8ò'‚(:­ ç±éÒ¢óÒù disk.dm-0.in‚ 36751.78125* proc.memory.percent‚17.119550704956055 cpu‚25.580004491360537 disk.rbd3.in‚0# disk.dm-0.out‚134792.1396484375 disk.rbd1.in‚ 0.51220703125! disk.nvme0c0n1.in‚ 36751.78125 disk.drbd2165.in‚0 proc.cpu.threads‚17) disk./.usagePercent‚37.68691263679404& disk.nvme0n1.out‚134792.1396484375 disk.rbd1.out‚ 42.08984375 disk.sda.in‚ 75.31640625 disk.sda1.out‚0 disk.drbd2165.out‚0$ memory_percent‚83.19566249847412) proc.memory.availableMB‚ 2753.22265625 disk.sda3.in‚ 75.31640625 disk.sda3.out‚ 808.03515625 network.recv‚ 4.012798e+06$ disk./.usageGB‚561.5622329711914( disk.nvme0c0n1.out‚134792.1396484375 disk.sda2.in‚0 network.sent‚ 1.8595895e+07 disk.sda.out‚ 808.03515625 disk.rbd0.out‚0 disk.sda2.out‚0 disk.dm-1.in‚ 75.31640625& disk.rbd0.in‚1.759218604441423e+13" proc.memory.rssMB‚ 2804.8671875 disk.sda1.in‚0 disk.rbd3.out‚0 disk.nvme0n1.in‚ 36751.78125 disk.dm-1.out‚ 808.035156250ZÙ¼4ó'/ _wandbruntime‚16093  _runtime‚16093³°CÈô' & lr-AdamW/pg2‚0.0003736731176449859 trainer/global_step‚9699 % _timestamp‚1.7843059093608239e+09 _runtime‚16093.578102151 _step‚796 & lr-AdamW/pg1‚0.0003736731176449859|%*(Íõ'Ç trainer/global_step‚9699 % _timestamp‚1.7843059093608239e+09 _runtime‚16093.578102151 _step‚796 & lr-AdamW/pg1‚0.0003736731176449859 & lr-AdamW/pg2‚0.0003736731176449859œmkþè4ö'/ _wandbruntime‚16093  _runtime‚16093%P1ž©÷'€P [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9126153762304 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚752964967G=¸õ]&870464 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 166788484 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2788566138765312 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚27513864 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11773629617664 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9110538477314 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚82770715 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚84017769 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9126153762304 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚82770709 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2787361576521728 _step‚797 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚84007796 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2797692292527616 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2788566138765312 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9109193876992 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2797690920964608 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚764738597488128 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2787361551093760 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648ºû4 (ø'…P RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚752964967870464 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 166788484 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2788566138765312 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚27513864 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11773629617664 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9110538477314 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚82770715 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚84017769 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9126153762304 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚82770709 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2787361576521728 _step‚797 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚84007796 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2797692292527616 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2788566138765312 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9109193876992 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2797690920964608 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚764738597488128 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2787361551093760 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9126153762304 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175q+ã4ù'/ _wandbruntime‚16094  _runtime‚16094ï˜ 'ú'ŠN K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚52404492 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚764816885887488 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚24887874 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2788844113321984 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚82778921 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _runtime‚16094.965592424 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2788844139495424 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚753042454278144 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚82778921 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 % _timestamp‚1.7843059107481904e+09 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9127078804992 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9125733408256 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚24887879 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚27516863 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9110117343238 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 166805182 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2787639432871936 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2797971218300416 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚250 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚27516618 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚764817288507904 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11774799700480 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 166805182 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚84026261 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 trainer/global_step‚9699 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2797971218300416 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚84026261 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9125733408256 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚52404742 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚84016274 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚82778915 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9125733408256 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚52404742 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚84016274 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚82778915 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚52404492 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚764816885887488 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚24887874 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2788844113321984 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚82778921 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _runtime‚16094.965592424 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2788844139495424 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚753042454278144 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚82778921 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 % _timestamp‚1.7843059107481904e+09 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9127078804992 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9125733408256 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚24887879 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚27516863 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9110117343238 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 166805182 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2787639432871936 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2797971218300416 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚250 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚27516618 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚764817288507904 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11774799700480 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 166805182žõÐ+Ÿü'‚(:” ö±éÒëÆÿ÷ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚289.503' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚57.900600000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0m¡2÷:ý'‚(:¯ ö±éÒËðŽú disk.rbd3.out‚0 disk.nvme0n1.in‚ 36752.203125 disk.rbd0.out‚0( disk.nvme0c0n1.out‚134802.4404296875$ memory_percent‚83.19709300994873 cpu‚25.57999051570821 network.recv‚ 4.015895e+06& disk.nvme0n1.out‚134802.4404296875 disk.sda2.in‚0" proc.memory.rssMB‚ 2804.8671875 disk.sda3.out‚ 808.48828125 disk.drbd2165.out‚0 disk.sda2.out‚0* proc.memory.percent‚17.119550704956055 disk.rbd1.out‚ 42.15234375" disk.nvme0c0n1.in‚ 36752.203125 network.sent‚ 1.8600963e+07 disk.sda.out‚ 808.48828125 disk.dm-1.in‚ 75.31640625 disk.dm-0.in‚ 36752.203125 disk.sda3.in‚ 75.31640625 disk.sda.in‚ 75.31640625 disk.drbd2165.in‚0 disk.rbd1.in‚ 0.51220703125) proc.memory.availableMB‚ 2752.98828125 proc.cpu.threads‚17$ disk./.usageGB‚561.5719375610352& disk.rbd0.in‚1.759218604441423e+13) disk./.usagePercent‚37.68756391996115 disk.dm-1.out‚ 808.48828125# disk.dm-0.out‚134802.4404296875 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd3.in‚0CHâ4þ'/ _wandbruntime‚16095  _runtime‚16095œôEû ÿ'† trainer/global_step‚9699 _runtime‚16095.05121899 _step‚799 0loss/FRt2/assignment_loss‚0.4304149011920269 0loss/FRt1/detection_loss‚0.46284015583802474 1loss/SRqqt2/detection_loss‚0.5809634725067424 /loss/FBt2/detection_loss‚0.4431107949193581 % _timestamp‚1.7843059108343158e+09 3loss/SRqqt1/assignment_loss‚0.31280638274047046 2loss/SRqqt2/assignment_loss‚0.3267127105114809 1loss/SRqqt1/detection_loss‚0.7160063079059816 /loss/FBt1/detection_loss‚0.5469744630703112 %loss/total_loss‚6.567373889799492 epoch‚24 0loss/FRt1/assignment_loss‚1.2354459760342946 /loss/FRt2/detection_loss‚1.4677915730703865 1loss/FBt1/assignment_loss‚0.04380471301737807 3loss/FBt2/assignment_loss‚0.0005024389930360354fÉF€(‹ %loss/total_loss‚6.567373889799492 epoch‚24 0loss/FRt1/assignment_loss‚1.2354459760342946 /loss/FRt2/detection_loss‚1.4677915730703865 1loss/FBt1/assignment_loss‚0.04380471301737807 3loss/FBt2/assignment_loss‚0.0005024389930360354 trainer/global_step‚9699 _runtime‚16095.05121899 _step‚799 0loss/FRt2/assignment_loss‚0.4304149011920269 0loss/FRt1/detection_loss‚0.46284015583802474 1loss/SRqqt2/detection_loss‚0.5809634725067424 /loss/FBt2/detection_loss‚0.4431107949193581 % _timestamp‚1.7843059108343158e+09 3loss/SRqqt1/assignment_loss‚0.31280638274047046 2loss/SRqqt2/assignment_loss‚0.3267127105114809 1loss/SRqqt1/detection_loss‚0.7160063079059816 /loss/FBt1/detection_loss‚0.5469744630703112ŸaMt•(‚(:Š …²éÒ©ÆŠ÷ gpu.0.gpu‚100.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚330.816' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚66.1632 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0VùóË3‚(‚(:¨ …²éÒÏæ™ù disk.rbd1.out‚ 42.15234375 cpu‚25.579981146621066 network.recv‚ 4.020791e+06 disk.dm-0.in‚ 36853.203125 disk.sda1.out‚0 disk.sda3.in‚ 75.31640625 disk.drbd2165.in‚0" proc.memory.rssMB‚ 2804.8671875) disk./.usagePercent‚37.68786421679881 disk.dm-1.out‚ 809.08203125! disk.dm-0.out‚134814.75390625 disk.sda3.out‚ 809.08203125& disk.nvme0c0n1.out‚134814.75390625 disk.rbd1.in‚ 0.51220703125 network.sent‚ 1.8662773e+07 disk.sda.out‚ 809.08203125& disk.rbd0.in‚1.759218604441423e+13 disk.sda1.in‚0 disk.rbd3.out‚0$ disk.nvme0n1.out‚134814.75390625 disk.sda.in‚ 75.31640625 disk.nvme0n1.in‚ 36853.203125 disk.dm-1.in‚ 75.31640625 disk.rbd3.in‚0 disk.sda2.in‚0* proc.memory.percent‚17.119550704956055$ disk./.usageGB‚561.5764122009277 disk.drbd2165.out‚0" disk.nvme0c0n1.in‚ 36853.203125 disk.sda2.out‚0$ memory_percent‚83.19711685180664 proc.cpu.threads‚17' proc.memory.availableMB‚ 2752.984375 disk.rbd0.out‚0$ÆŸžƒ(‚(:“ ”²éÒóëù÷ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚290.855' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚58.17100000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0û¬áè5„(‚(:ª ”²éÒäÅÓù& disk.rbd0.in‚1.759218604441423e+13 disk.drbd2165.in‚0$ disk.nvme0c0n1.in‚36894.30859375* disk./.usagePercent‚37.688371367724145 disk.sda.out‚ 809.3203125 disk.dm-1.out‚ 809.3203125% proc.memory.availableMB‚ 2753.0625 cpu‚25.579967270386973$ disk./.usageGB‚561.5839691162109$ memory_percent‚83.19664001464844" proc.memory.rssMB‚ 2804.8671875 proc.cpu.threads‚17$ disk.nvme0n1.out‚134823.49609375 disk.sda2.out‚0 disk.sda1.in‚0 disk.sda3.in‚ 75.31640625 disk.sda1.out‚0" disk.nvme0n1.in‚36894.30859375 disk.rbd0.out‚0 network.sent‚ 1.8705766e+07 disk.rbd1.in‚ 0.51220703125 disk.sda.in‚ 75.31640625 disk.dm-1.in‚ 75.31640625 network.recv‚ 4.025001e+06 disk.rbd1.out‚ 42.27734375 disk.rbd3.out‚0& disk.nvme0c0n1.out‚134823.49609375 disk.dm-0.in‚36894.30859375! disk.dm-0.out‚134823.49609375 disk.sda3.out‚ 809.3203125 disk.drbd2165.out‚0* proc.memory.percent‚17.119550704956055 disk.rbd3.in‚0 disk.sda2.in‚07á EŸ…(‚(:” £²éÒ«ó…÷ gpu.0.gpu‚81.0 gpu.0.memory‚24, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚36.0 gpu.0.powerWatts‚109.757' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.951400000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚07ãbyC†(‚(:¸ £²éÒÕÔù$ disk./.usageGB‚561.5910034179688 disk.rbd3.in‚0 disk.sda.in‚ 75.31640625 disk.dm-1.in‚ 75.31640625 disk.rbd0.out‚0 disk.sda1.out‚0 cpu‚25.579970384490647) disk.nvme0c0n1.out‚134831.59130859375& disk.rbd0.in‚1.759218604441423e+13 disk.dm-0.in‚36894.73046875 network.recv‚ 4.028901e+06 disk.rbd1.out‚ 42.27734375 disk.sda1.in‚0( proc.memory.availableMB‚ 2752.8203125) disk./.usagePercent‚37.68884344561728 disk.drbd2165.in‚0" disk.nvme0n1.in‚36894.73046875 disk.rbd1.in‚ 0.51220703125 disk.sda.out‚ 809.63671875 disk.sda3.in‚ 75.31640625$ memory_percent‚83.19811820983887 disk.sda3.out‚ 809.63671875 network.sent‚ 1.8710935e+07 disk.rbd3.out‚0' disk.nvme0n1.out‚134831.59130859375 disk.sda2.out‚0" proc.memory.rssMB‚ 2804.8671875* proc.memory.percent‚17.119550704956055 proc.cpu.threads‚17 disk.drbd2165.out‚0$ disk.nvme0c0n1.in‚36894.73046875 disk.sda2.in‚0 disk.dm-1.out‚ 809.63671875$ disk.dm-0.out‚134831.59130859375ÞúÞ8Ÿ‡(‚(:” ²²éÒœ™ˆ÷ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚263.941' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚52.788199999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ꛆf>ˆ(‚(:³ ²²éÒñâ•ú* disk./.usagePercent‚37.689149118613166 disk.rbd3.out‚0 disk.dm-0.in‚36894.94921875 disk.rbd1.in‚ 0.51220703125 disk.sda.in‚ 75.31640625 disk.dm-1.out‚ 809.828125) proc.memory.availableMB‚ 2752.56640625* proc.memory.percent‚17.119550704956055 network.sent‚ 1.8716047e+07 disk.drbd2165.in‚0' disk.nvme0n1.out‚134845.47509765625 disk.rbd1.out‚ 42.33984375 proc.cpu.threads‚17 disk.sda2.in‚0 disk.sda.out‚ 809.828125 disk.rbd0.out‚0$ disk.dm-0.out‚134845.47509765625 network.recv‚ 4.03223e+06 disk.sda1.in‚0 disk.sda3.in‚ 75.31640625" disk.nvme0n1.in‚36894.94921875 cpu‚25.579879337076704$ disk.nvme0c0n1.in‚36894.94921875 disk.sda2.out‚0$ memory_percent‚83.19966793060303 disk.sda3.out‚ 809.828125 disk.drbd2165.out‚0 disk.rbd3.in‚0) disk.nvme0c0n1.out‚134845.47509765625 disk.dm-1.in‚ 75.31640625& disk.rbd0.in‚1.759218604441423e+13$ disk./.usageGB‚561.5955581665039 disk.sda1.out‚0" proc.memory.rssMB‚ 2804.8671875î>@‰(‚(:’ Á²éÒÃúáö gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚40.0 gpu.0.powerWatts‚302.16' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚60.43200000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ú´˜ECŠ(‚(:¸ Á²éÒÀßù disk.sda1.out‚0 network.sent‚ 1.872113e+07) disk./.usagePercent‚37.68958893955699 disk.sda.in‚ 75.31640625 disk.sda.out‚ 810.30859375 disk.drbd2165.in‚0" proc.memory.rssMB‚ 2804.8671875& disk.rbd0.in‚1.759218604441423e+13 disk.drbd2165.out‚0 disk.rbd3.out‚0) disk.nvme0c0n1.out‚134852.39111328125$ memory_percent‚83.19661617279053 cpu‚25.579916281544705 disk.sda3.in‚ 75.31640625$ disk.nvme0c0n1.in‚36895.15234375 disk.sda2.in‚0 disk.rbd1.out‚ 42.33984375* proc.memory.percent‚17.119550704956055 disk.rbd0.out‚0 disk.sda1.in‚0$ disk./.usageGB‚561.6021118164062 disk.dm-1.in‚ 75.31640625 disk.sda3.out‚ 810.30859375" disk.nvme0n1.in‚36895.15234375' disk.nvme0n1.out‚134852.39111328125 disk.sda2.out‚0 disk.rbd1.in‚ 0.51220703125) proc.memory.availableMB‚ 2753.06640625 disk.dm-1.out‚ 810.30859375 disk.dm-0.in‚36895.15234375$ disk.dm-0.out‚134852.39111328125 disk.rbd3.in‚0 network.recv‚ 4.035667e+06 proc.cpu.threads‚17º– -4‹(/ _wandbruntime‚16173  _runtime‚16173}ð£ÊŒ(Ä ' lr-AdamW/pg2‚0.00037090886096646477 trainer/global_step‚9749 % _timestamp‚1.7843059891757529e+09 _runtime‚16173.392944442 _step‚800 ' lr-AdamW/pg1‚0.00037090886096646477 }Ð?Ï(É ' lr-AdamW/pg1‚0.00037090886096646477 ' lr-AdamW/pg2‚0.00037090886096646477 trainer/global_step‚9749 % _timestamp‚1.7843059891757529e+09 _runtime‚16173.392944442 _step‚800 ç§+y4Ž(/ _wandbruntime‚16173  _runtime‚16173HÄ*×((€P H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚27650058 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚27649802 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9156775590422 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2811638644963840 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2811637273400832 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2811638644963840 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚768672432680448 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚756838920163328 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9155430990100 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9171124303360 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2802466149097472 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚83181015 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚768672835308032 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11833512517120 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11833547053568 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 _runtime‚16173.394151621 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2802466175270912 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2801255640002560 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2810412441020950 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2802466175270912 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚83181009 % _timestamp‚1.7843059891768057e+09 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9171124303360 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚25012481 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚83181009 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚52662283 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚83181015 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚84433548 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2811637273400832 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚84443521 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0æÄ\h(…P H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚27650058 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚27649802 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9156775590422 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2811638644963840 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2811637273400832 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2811638644963840 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚768672432680448 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚756838920163328 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9155430990100 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9171124303360 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2802466149097472 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚83181015 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚768672835308032 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚«‹B[ô 402627584 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11833512517120 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11833547053568 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 _runtime‚16173.394151621 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2802466175270912 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2801255640002560 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2810412441020950 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2802466175270912 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚83181009 % _timestamp‚1.7843059891768057e+09 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9171124303360 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚25012481 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚83181009 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚52662283 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚83181015 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚84433548 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2811637273400832 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚84443521 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0¡æ5g4‘(/ _wandbruntime‚16174  _runtime‚16174–þÞÆ'’(ŠN Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚27652516 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 _runtime‚16174.780183936 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2811916199166464 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚756916774662144 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2802744176001024 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2802744149827584 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚83189215 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2801533521780736 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2802744176001024 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚52667789 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 _step‚802 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚83189221 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚52667540 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9172049338880 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 167641234 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚249 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2810689876237082 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7843059905627792e+09 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9173394735616 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚84442026 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2802744149827584 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2811917570736640ª“'“(N K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 167631241 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2801533547208704 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9172049338880 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚244 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚768751515588096 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚756916406571008 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚84452013 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚84442026 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚768751112967680 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9156354456346 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 167641234 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚83189221 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11834740925952 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚249 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2810689876237082 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7843059905627792e+09 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9173394735616 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚84442026 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2802744149827584 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2811917570736640 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚27652516 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 _runtime‚16174.780183936 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2811916199166464 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚756916774662144 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2802744176001024 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2802744149827584 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚83189215 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2801533521780736 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2802744176001024 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚52667789 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 _step‚802 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚83189221 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚52667540 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9172049338880 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 167641234 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 F (‚(:³ Œ³éÒÆù®ù disk.dm-1.in‚ 75.31640625 disk.sda1.out‚0& disk.rbd0.in‚1.759218604441423e+13 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.sda2.in‚0$ disk./.usageGB‚561.6346969604492& disk.nvme0n1.out‚134910.4970703125 disk.sda2.out‚0$ memory_percent‚82.99717903137207* proc.memory.percent‚16.889333724975586 disk.sda.out‚ 812.01953125 disk.rbd0.out‚0 disk.rbd3.in‚0 disk.rbd1.out‚ 42.52734375# disk.dm-0.out‚134910.4970703125 disk.sda1.in‚0 disk.sda3.in‚ 75.31640625 disk.sda3.out‚ 812.01953125 disk.rbd3.out‚0$ disk.nvme0c0n1.in‚37426.44921875 cpu‚25.58059239759482 proc.cpu.threads‚17 disk.dm-1.out‚ 812.01953125" disk.nvme0n1.in‚37426.44921875( proc.memory.availableMB‚ 2785.7421875" proc.memory.rssMB‚ 2767.1484375 network.sent‚ 1.880382e+07( disk.nvme0c0n1.out‚134910.4970703125 disk.rbd1.in‚ 0.51220703125 network.recv‚ 4.052734e+06 disk.sda.in‚ 75.31640625 disk.dm-0.in‚37426.44921875) disk./.usagePercent‚37.69177575591447˜h².4¡(/ _wandbruntime‚16252  _runtime‚16252‹‘”³È¢( & lr-AdamW/pg1‚0.0003681429995715234 & lr-AdamW/pg2‚0.0003681429995715234 trainer/global_step‚9799 % _timestamp‚1.7843060684337833e+09 _runtime‚16252.650911282 _step‚804ÂA;PÍ£(Ç _step‚804 & lr-AdamW/pg1‚0.0003681429995715234 & lr-AdamW/pg2‚0.0003681429995715234 trainer/global_step‚9799 % _timestamp‚1.7843060684337833e+09 _runtime‚16252.650911282¤ÅY•z4¤(/ _wandbruntime‚16252  _runtime‚16252‚6ËÝ(¥(€P XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚25139636 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11893398629888 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9217440233984 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚270 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2825583625837056 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9218785623552 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36633600 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚27785687 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2815149754339328 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2825583625837056 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2816366185603072 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 168460588 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 _step‚805 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 168450609 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚84869273 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2816366211776512 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚25139631 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11893435263488 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚83591309 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚52925053 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2824352767042858 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2825584997400064 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚84859300 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚83591315#/ (¦(…P WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9217440233984 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚265 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9201668103208 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚83591315 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚772607043901952 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2824351397014568 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚83591309 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚27785687 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2815149754339328 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2825583625837056 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2816366185603072 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 168460588 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 _step‚805 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 168450609 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚84869273 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2816366211776512 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚25139631 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11893435263488 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚83591309 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚52925053 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2824352767042858 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2825584997400064 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚84859300 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚83591315 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚25139636 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11893398629888 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9217440233984 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚270 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2825583625837056 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9218785623552 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36633600 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚84867778 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚52930259 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 168477286 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚25142179 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2824631572287392 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚84877765 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚83599521 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2825863923172864 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9219710666240 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 % _timestamp‚1.7843060698213873e+09 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2816644186333184 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚760791095046144 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 trainer/global_step‚9799 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2816644186333184 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚806 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11894579770880 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 _runtime‚16254.038823391 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 168477286 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚27788085 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚265 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚772685711443456 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚52930524 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11894616397312 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2815427636117504 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚260 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2816644212506624 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 NADeviceSta§ªž×tsMonitor.on_train_batch_end/allocation.large_pool.freed‚83599515 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 QDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚265 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚772685711443456 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚52930524 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11894616397312 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2815427636117504 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚260 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2816644212506624 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚83599515 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚84867778 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚52930259 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 168477286 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚25142179 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2824631572287392 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚84877765 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚83599521 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2825863923172864 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9219710666240 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 % _timestamp‚1.7843060698213873e+09 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2816644186333184 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚760791095046144 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 trainer/global_step‚9799 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2816644186333184 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚806 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11894579770880 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 _runtime‚16254.038823391 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 168477286 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚27788085 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 FDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9249249816638 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9263756164608 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 169296640 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚25266781 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2830266248282112 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚84001609 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 trainer/global_step‚9849 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9265101554176 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚776540373770752 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9263756164608 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2838291723036476 % _timestamp‚1.7843061479426334e+09 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 169286661 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2839529978273280 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚53186866 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚263 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2829043843248128 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2838293093064766 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9265101554176 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚25266786 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2839531349836288 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2839529978273280 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2839529978273280 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚53186866 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚263 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2829043843248128 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2838293093064766 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 =7DeviceStatsMonitor.on_train_batch_start/num_device_+ˆÓí2"free‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9265101554176 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚25266786 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2839531349836288 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2839529978273280 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9249249816638 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9263756164608 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 169296640 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚25266781 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2830266248282112 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚84001609 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 trainer/global_step‚9849 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9265101554176 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚776540373770752 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9263756164608 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2838291723036476 % _timestamp‚1.7843061479426334e+09 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 169286661 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6©æÚ4½(/ _wandbruntime‚16333  _runtime‚16333óeP'¾(ŠN \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9266026596864 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚84009815 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2839810275609088 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9264681200128 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9248828682562 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚84009821 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 169313338 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 _runtime‚16333.546994733 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚764665415430144 _step‚810 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚53192034 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2839810275609088 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2830544222838784 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2838570528280898 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2829321699598336 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 169313338 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚85303517 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚85303517 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚776619460421120 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚53191777 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2839808904038912 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚84009821 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9264681200128 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚25269329 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11954010461696 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2829321725026304 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚85293530 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2838571898309300 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11954044990976 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 169313338 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 _runtime‚16333.546994733 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚764665415430144 _step‚810 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚53192034 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2839810275609088 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2830544222838784 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2838570528280898 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2829321699598336 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 169313338 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚85303517 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚85303517 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚776619460421120 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚53191777 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2839808904038912 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚84009821 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9264681200128 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚25269329 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11954010461696 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2829321725026304 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚85293530 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2838571898309300 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736ªüQsÎ4À(/ _wandbruntime‚16333  _runtime‚16333êΙe Á(„ 2loss/FBt1/assignment_loss‚0.000573178130965779 /loss/FRt2/detection_loss‚1.4268357799298919 % _timestamp‚1.7843061494155948e+09 0loss/FRt2/assignment_loss‚0.5511230711753552 2loss/SRqqt1/assignment_loss‚0.3119011336129131 1loss/FBt2/assignment_loss‚0.00243891995452964 1loss/SRqqt1/detection_loss‚0.6654479252256849 %loss/total_loss‚6.678073535664701 _runtime‚16333.632528532 0loss/FRt1/assignment_loss‚1.3078810622673909 /loss/FRt1/detection_loss‚0.4932666030942427 .loss/FBt1/detection_loss‚0.554718028428033 /loss/FBt2/detection_loss‚0.3376311426830048 epoch‚24 trainer/global_step‚9849 _step‚811 3loss/SRqqt2/assignment_loss‚0.42990799006438113 1loss/SRqqt2/detection_loss‚0.5963487010983093Ó*ñþÂ(‰ 2loss/FBt1/assignment_loss‚0.000573178130965779 /loss/FRt2/detection_loss‚1.4268357799298919 % _timestamp‚1.7843061494155948e+09 0loss/FRt2/assignment_loss‚0.5511230711753552 2loss/SRqqt1/assignment_loss‚0.3119011336129131 1loss/FBt2/assignment_loss‚0.00243891995452964 1loss/SRqqt1/detection_loss‚0.6654479252256849 %loss/total_loss‚6.678073535664701 _runtime‚16333.632528532 0loss/FRt1/assignment_loss‚1.3078810622673909 /loss/FRt1/detection_loss‚0.4932666030942427 .loss/FBt1/detection_loss‚0.554718028428033 /loss/FBt2/detection_loss‚0.3376311426830048 epoch‚24 trainer/global_step‚9849 _step‚811 3loss/SRqqt2/assignment_loss‚0.42990799006438113 1loss/SRqqt2/detection_loss‚0.5963487010983093«Xª¹6ŸÃ(‚(:” æ³éÒë¤òö gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚292.983' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚58.596599999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ê`-?Ä(‚(:´ æ³éÒã‰ù disk.rbd1.in‚ 0.51220703125" disk.dm-0.out‚134983.576171875* proc.memory.percent‚16.889333724975586$ disk./.usageGB‚561.6706581115723 disk.drbd2165.out‚0 disk.rbd3.out‚0 network.sent‚ 1.8891985e+07$ disk.nvme0c0n1.in‚37528.33984375% disk.nvme0n1.out‚134983.576171875$ memory_percent‚82.95199871063232) proc.memory.availableMB‚ 2793.14453125" proc.memory.rssMB‚ 2767.1484375 disk.rbd3.in‚0" disk.nvme0n1.in‚37528.33984375& disk.rbd0.in‚1.759218604441423e+13 disk.dm-0.in‚37528.33984375 disk.sda3.in‚ 75.31640625 disk.sda3.out‚ 814.13671875 proc.cpu.threads‚17 disk.drbd2165.in‚0 disk.dm-1.in‚ 75.31640625 disk.dm-1.out‚ 814.13671875 disk.sda1.out‚0 network.recv‚ 4.074235e+06* disk./.usagePercent‚37.694189138939755 disk.rbd0.out‚0 disk.sda1.in‚0' disk.nvme0c0n1.out‚134983.576171875 disk.sda2.in‚0 disk.rbd1.out‚ 42.77734375 disk.sda.in‚ 75.31640625 disk.sda.out‚ 814.13671875 disk.sda2.out‚0 cpu‚25.581265563363782H‰å"žÅ(‚(:“ õ³éÒîÂòö gpu.0.gpu‚59.0 gpu.0.memory‚19, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚330.288' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚66.05760000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Œ#[³_Æ(‚(:· õ³éÒΡªù* proc.memory.percent‚16.889333724975586 cpu‚25.58144985675133 network.sent‚ 1.8955839e+07) disk./.usagePercent‚37.69465123253906 disk.rbd3.out‚0 disk.rbd1.in‚ 0.51220703125" proc.memory.rssMB‚ 2767.1484375 network.recv‚ 4.079122e+06 disk.dm-1.in‚ 75.31640625 disk.rbd3.in‚0$ disk.n¯Œçãvme0c0n1.in‚37528.76171875 disk.sda2.out‚0 disk.rbd1.out‚ 42.77734375 disk.sda.in‚ 75.31640625 disk.dm-1.out‚ 814.33203125$ disk.dm-0.out‚134998.55517578125 disk.sda1.out‚0 disk.sda3.in‚ 75.31640625) disk.nvme0c0n1.out‚134998.55517578125( proc.memory.availableMB‚ 2793.1796875 disk.rbd0.out‚0 disk.sda1.in‚0 disk.drbd2165.out‚0' disk.nvme0n1.out‚134998.55517578125& disk.rbd0.in‚1.759218604441423e+13 disk.dm-0.in‚37528.76171875 disk.sda3.out‚ 814.33203125 disk.drbd2165.in‚0" disk.nvme0n1.in‚37528.76171875$ memory_percent‚82.95178413391113 proc.cpu.threads‚17 disk.sda2.in‚0$ disk./.usageGB‚561.6775436401367 disk.sda.out‚ 814.33203125M IXžÇ(‚(:“ „´éÒýýõö gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚288.679' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚57.73579999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ðf1;*È(‚(:Ÿ „´éÒ­ŒŸù network.sent‚ 1.8961062e+07$ disk./.usageGB‚561.6817817687988& disk.rbd0.in‚1.759218604441423e+13 disk.rbd0.out‚0 disk.nvme0n1.in‚37604.75 disk.sda.in‚ 75.31640625' proc.memory.availableMB‚ 2793.078125 disk.dm-1.out‚ 814.48828125 disk.sda1.in‚0 cpu‚25.58157391727888 disk.dm-0.in‚37604.75 disk.drbd2165.out‚0( disk.nvme0c0n1.out‚135006.6201171875 disk.rbd1.in‚ 0.51220703125 disk.sda.out‚ 814.48828125 proc.cpu.threads‚17 disk.sda3.out‚ 814.48828125 disk.dm-1.in‚ 75.31640625# disk.dm-0.out‚135006.6201171875# memory_percent‚82.9524040222168 network.recv‚ 4.082242e+06( disk./.usagePercent‚37.6949356569096 disk.drbd2165.in‚0 disk.rbd3.out‚0 disk.nvme0c0n1.in‚37604.75 disk.sda1.out‚0 disk.sda3.in‚ 75.31640625 disk.rbd3.in‚0 disk.sda2.in‚0 disk.sda2.out‚0" proc.memory.rssMB‚ 2767.1484375* proc.memory.percent‚16.889333724975586& disk.nvme0n1.out‚135006.6201171875 disk.rbd1.out‚ 42.83984375ø£ðÐ’É(‚(:‡ “´éÒ²ˆõö gpu.0.gpu‚3.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚325.461' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚65.0922 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Y蜘<Ê(‚(:± “´éÒËÈëù disk.rbd0.out‚0 disk.dm-0.in‚ 37605.171875 disk.sda3.out‚ 815.0703125' disk.nvme0n1.out‚135016.88916015625* proc.memory.percent‚16.889333724975586 network.sent‚ 1.8966245e+07 disk.drbd2165.out‚0 disk.rbd3.in‚0) disk.nvme0c0n1.out‚135016.88916015625$ memory_percent‚82.95519351959229 disk.sda3.in‚ 75.31640625 disk.sda2.out‚0 disk.sda1.out‚0 disk.sda.out‚ 815.0703125$ disk.dm-0.out‚135016.88916015625 disk.rbd3.out‚0" disk.nvme0c0n1.in‚ 37605.171875 disk.rbd1.in‚ 0.51220703125) proc.memory.availableMB‚ 2792.62109375* disk./.usagePercent‚37.695537530622595 disk.dm-1.out‚ 815.0703125 disk.drbd2165.in‚0 disk.nvme0n1.in‚ 37605.171875 disk.rbd1.out‚ 42.83984375 proc.cpu.threads‚17 network.recv‚ 4.085621e+06 disk.dm-1.in‚ 75.31640625" proc.memory.rssMB‚ 2767.1484375 cpu‚25.581725962711225 disk.sda2.in‚0 disk.sda1.in‚0$ disk./.usageGB‚561.6907501220703 disk.sda.in‚ 75.31640625& disk.rbd0.in‚1.759218604441423e+13¹…Ö”Ë(‚(:‰ ¢´éÒÇ­ù÷ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚296.933' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚59.3866 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0[8Ì(‚(:­ ¢´éÒžêÀù$ disk./.usageGB‚561.6952629089355 disk.sda3.in‚ 75.31640625 disk.sda2.in‚0$ memory_percent‚82.95531272888184 proc.cpu.threads‚17 disk.sda1.in‚0" disk.nvme0c0n1.in‚ 37605.390625" proc.memory.rssMB‚ 2767.1484375 network.sent‚ 1.8971295e+07 disk.dm-0.in‚ 37605.390625 disk.dm-1.out‚ 815.421875$ disk.dm-0.out‚135029.65673828125 disk.sda1.out‚0 disk.rbd1.out‚ 42.90234375* proc.memory.percent‚16.889333724975586* disk./.usagePercent‚37.695840387535604 disk.dm-1.in‚ 75.31640625& disk.rbd0.in‚1.759218604441423e+13 disk.sda3.out‚ 815.421875 disk.drbd2165.out‚0 disk.rbd3.in‚0) disk.nvme0c0n1.out‚135029.65673828125 disk.sda2.out‚0 network.recv‚ 4.088825e+06 disk.sda.in‚ 75.31640625 disk.rbd3.out‚0 disk.rbd1.in‚ 0.51220703125( proc.memory.availableMB‚ 2792.6015625 disk.sda.out‚ 815.421875 disk.rbd0.out‚0 disk.drbd2165.in‚0 disk.nvme0n1.in‚ 37605.390625' disk.nvme0n1.out‚135029.65673828125 cpu‚25.581804691212728<Æ<”Í(‚(:‰ ±´éÒˆóëö gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚193.131' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚38.6262 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0š•­6Î(‚(:« ±´éÒ„öØù disk.sda.out‚ 815.88671875& disk.rbd0.in‚1.759218604441423e+13 disk.rbd3.out‚0( disk.nvme0c0n1.out‚135039.7158203125& disk.nvme0n1.out‚135039.7158203125 disk.sda2.in‚0 disk.sda2.out‚0) disk./.usagePercent‚37.69628968075819 disk.rbd0.out‚0# disk.dm-0.out‚135039.7158203125 disk.sda1.in‚0 disk.rbd1.out‚ 42.90234375$ memory_percent‚82.95531272888184( proc.memory.availableMB‚ 2792.6015625 network.recv‚ 4.092306e+06 disk.drbd2165.out‚0 network.sent‚ 1.8975872e+07 disk.nvme0n1.in‚ 37605.59375" proc.memory.rssMB‚ 2767.1484375* proc.memory.percent‚16.889333724975586 disk.sda.in‚ 75.31640625 disk.sda3.in‚ 75.31640625 disk.rbd1.in‚ 0.51220703125 disk.dm-1.in‚ 75.31640625 disk.dm-1.out‚ 815.88671875 disk.dm-0.in‚ 37605.59375 disk.sda3.out‚ 815.88671875 disk.rbd3.in‚0 disk.drbd2165.in‚0 proc.cpu.threads‚17 disk.sda1.out‚0! disk.nvme0c0n1.in‚ 37605.59375 cpu‚25.58198800300945$ disk./.usageGB‚561.7019577026367cG@W4Ï(/ _wandbruntime‚16411  _runtime‚16411¬€DFÊÐ(Ä _runtime‚16411.434046744 _step‚812 ' lr-AdamW/pg1‚0.00036260717330847486 ' lr-AdamW/pg2‚0.00036260717330847486 trainer/global_step‚9899 % _timestamp‚1.7843062272167647e+09ö‘ÏÑ(É _runtime‚16411.434046744 _step‚812 ' lr-AdamW/pg1‚0.00036260717330847486 ' lr-AdamW/pg2‚0.00036260717330847486 trainer/global_step‚9899 % _timestamp‚1.7843062272167647e+09¬ôÑñ4Ò(/ _wandbruntime‚16411  _runtime‚16411¾Øék(Ó(ÿO ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9311417484800 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 trainer/global_step‚9899 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚85710804 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚780474995436032 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2852233419086674 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2842937906728960 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚25393936 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚85720777 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12012711493120 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚84411915 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2853477702272512 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚25393931 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 $ _timestamp‚1.784306227221545e+09 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚53449762 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 _runtime‚16411.438878255 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2842937932156928 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚28055826 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2853476330709504 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚84411909 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12012746029568 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 170122713 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚768461881315328 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2853477702272512 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9310072095232 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚85720777 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2852232049058384 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2853476330709504 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚768462249406464 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 170132692 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9294142329424 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚85710804 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2844166258614272 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0˜\Œï (Ô(„P dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12012746029568 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 170122713 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚768461881315328 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2853477702272512 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9310072095232 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚85720777 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2852232049058384 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2853476330709504 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚768462249406464 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 170132692 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9294142329424 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚85710804 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2844166258614272 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9311417484800 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 trainer/global_step‚9899 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚85710804 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚780474995436032 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2852233419086674 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2842937906728960 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚25393936 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚85720777 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12012711493120 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚84411915 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2853477702272512 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚25393931 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 $ _timestamp‚1.784306227221545e+09 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚53449762 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 _runtime‚16411.438878255 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2842937932156928 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚28055826 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2853476330709504 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚84411909 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584­{÷í4Õ(/ _wandbruntime‚16412  _runtime‚164120ñY6Ö(ŠN TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 170149390 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚236 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12013944317440 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2843215813935104 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚25396474 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9312342527488 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚53454999 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 170149390 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2844444285517824 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚84420115 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2844444259344384 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9310997130752 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚768539367723008 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚84420121 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2852512224331208 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9296410396104 TDDeviceStatsMonitor.on_train_batch_end/r•/§‡Úeserved_bytes.large_pool.peak‚ 47351595008 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚241 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚84420115 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 % _timestamp‚1.7843062286098542e+09 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚53454758 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚780553680131584 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12013909788160 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚25396479 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 170139397 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚780553277511168 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9310997130752 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2843215788507136 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2853755256475136 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 170139397 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚28058284 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚85729269 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2844444285517824 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 170149390 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚236 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12013944317440 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2843215813935104 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚25396474 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9312342527488 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚53454999 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 170149390 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2844444285517824 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚84420115 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2844444259344384 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9310997130752 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚768539367723008 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚84420121 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2852512224331208 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9296410396104 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚241 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚84420115 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 % _timestamp‚1.7843062286098542e+09 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚53454758 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚780553680131584 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12013909788160 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚25396479 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 170139397 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚780553277511168 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9310997130752 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2843215788507136 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2853755256475136 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 170139397 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚28058284 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚85729269 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2844444285517824 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 QDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2858066321293312 % _timestamp‚1.7843063069035451e+09 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9356388025856 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚53712627 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚25521081 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚254 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚772336569790464 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2867424054708736 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚784408900127232 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2858066295119872 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _step‚817 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚86146529 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2867422683145728 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 trainer/global_step‚9949 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 170958765 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚784409302754816 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9357733415424 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚86146529 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2866173745108582 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2856832021065728 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚25521086 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9357733415424 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚84822209 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12072732964352 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚84822215 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 170968744 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚28191292 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2867422683145728 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640ÆÐeV (ê(…P UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚86146529 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2867422683145728 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 trainer/global_step‚9949 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 170958765 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚784409302754816 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9357733415424 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚86146529 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2866173745108582 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2856832021065728 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚25521086 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9357733415424 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚84822209 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12072732964352 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚84822215 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 170968744 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚28191292 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2867422683145728 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚84822215 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2858066321293312 % _timestamp‚1.7843063069035451e+09 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9356388025856 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚53712627 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚25521081 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚254 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚772336569790464 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2867424054708736 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚784408900127232 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2858066295119872 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _step‚817± ùDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚53717445 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚28194062 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚86145034 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2866452550353116 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 170985442 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 170975449 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚772414056198144 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚246 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 170985442 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚25523629 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚9949 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9358658458112 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9341302908778 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2858344295849984 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9357313061376 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚784487989465600 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9358658458112 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚84830415 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚772413688107008 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9342647509212 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚84830421 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2867702980481536 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚241 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚53717691 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚784487586845184 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 % _timestamp‚1.7843063082936885e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚86155021 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2866451180324714 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2857109877415936 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _step‚818 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2858344295849984 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚86145034 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚25523624 _runtime‚16492.511081653øW^e'í(N J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚241 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚53717691 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚784487586845184 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 % _timestamp‚1.7843063082936885e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚86155021 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2866451180324714 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2857109877415936 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _step‚818 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2858344295849984 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚86145034 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚25523624 _runtime‚16492.511081653 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚53717445 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚28194062 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚86145034 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2866452550353116 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 170985442 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 170975449 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚772414056198144 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚246 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 170985442 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚25523629 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚9949 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9358658458112 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9341302908778 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2858344295849984 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9357313061376 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚784487989465600 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9358658458112 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚84830415 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚772413688107008 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9342647509212 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚84830421 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2867702980481536 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440²WÈ%4î(/ _wandbruntime‚16492  _runtime‚16492‹”7[@ï(„ _runtime‚16492.596204141 2loss/SRqqt2/assignment_l½²ü/Êoss‚0.3794071590205403 2loss/FBt2/assignment_loss‚0.004847997232130309 epoch‚24 trainer/global_step‚9949 % _timestamp‚1.7843063083792992e+09 _step‚819 0loss/FRt1/assignment_loss‚1.3242409941128321 1loss/FRt2/assignment_loss‚0.44919977537714517 /loss/FRt1/detection_loss‚0.4993792197255763 0loss/SRqqt2/detection_loss‚0.494753469620918 /loss/FBt1/detection_loss‚0.7573897563398067 2loss/SRqqt1/assignment_loss‚0.3445989683621967 2loss/FBt1/assignment_loss‚0.007176289654750971 /loss/FRt2/detection_loss‚1.3787967788946565 /loss/FBt2/detection_loss‚0.3615575807243141 0loss/SRqqt1/detection_loss‚0.631309537334083 %loss/total_loss‚6.632657526398949‹Ñfð(‰ 0loss/FRt1/assignment_loss‚1.3242409941128321 1loss/FRt2/assignment_loss‚0.44919977537714517 /loss/FRt1/detection_loss‚0.4993792197255763 0loss/SRqqt2/detection_loss‚0.494753469620918 /loss/FBt1/detection_loss‚0.7573897563398067 2loss/SRqqt1/assignment_loss‚0.3445989683621967 2loss/FBt1/assignment_loss‚0.007176289654750971 /loss/FRt2/detection_loss‚1.3787967788946565 /loss/FBt2/detection_loss‚0.3615575807243141 0loss/SRqqt1/detection_loss‚0.631309537334083 %loss/total_loss‚6.632657526398949 _runtime‚16492.596204141 2loss/SRqqt2/assignment_loss‚0.3794071590205403 2loss/FBt2/assignment_loss‚0.004847997232130309 epoch‚24 trainer/global_step‚9949 % _timestamp‚1.7843063083792992e+09 _step‚819³d„ižñ(‚(:“ ‹µéÒ”²ˆ÷ gpu.0.gpu‚90.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚275.487' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚55.09740000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0³<†9ò(‚(:® ‹µéÒ®ÊÇù proc.cpu.threads‚17& disk.rbd0.in‚1.759218604441423e+13 disk.sda3.in‚ 75.31640625) proc.memory.availableMB‚ 2792.38671875" proc.memory.rssMB‚ 2767.1484375 disk.rbd3.in‚0 disk.rbd3.out‚0! disk.nvme0c0n1.in‚ 38223.34375) disk.nvme0c0n1.out‚135104.13623046875 disk.sda2.in‚0 disk.rbd1.in‚ 0.51220703125$ disk.dm-0.out‚135104.13623046875) disk./.usagePercent‚37.69875682536567$ disk./.usageGB‚561.7387199401855 disk.sda.in‚ 75.31640625 disk.rbd0.out‚0 disk.dm-0.in‚ 38223.34375# memory_percent‚82.9566240310669 network.sent‚ 1.9063554e+07 network.recv‚ 4.113163e+06 disk.sda.out‚ 817.47265625 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.dm-1.out‚ 817.47265625 disk.sda1.in‚0 cpu‚25.58178397361507' disk.nvme0n1.out‚135104.13623046875 disk.dm-1.in‚ 75.31640625 disk.nvme0n1.in‚ 38223.34375 disk.sda2.out‚0 disk.rbd1.out‚ 43.16796875 disk.drbd2165.out‚0* proc.memory.percent‚16.889333724975586 disk.sda3.out‚ 817.47265625v-™Ážó(‚(:“ šµéÒ¡¨Ðû gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚283.847' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚56.76939999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Èë®5ô(‚(:ª šµéÒ¹£Çü disk.sda2.in‚0 network.sent‚ 1.9128817e+07 disk.sda.out‚ 817.66796875 disk.dm-0.in‚ 38223.765625 disk.drbd2165.out‚0 disk.sda.in‚ 75.31640625 disk.dm-1.in‚ 75.31640625 disk.rbd0.out‚0& disk.rbd0.in‚1.759218604441423e+13 disk.sda3.in‚ 75.31640625 disk.sda2.out‚0" proc.memory.rssMB‚ 2767.1484375 disk.dm-1.out‚ 817.66796875 disk.drbd2165.in‚0% disk.nvme0n1.out‚135118.169921875' proc.memory.availableMB‚ 2792.453125 disk.nvme0n1.in‚ 38223.765625 network.recv‚ 4.117791e+06$ disk./.usageGB‚561.7455940246582 disk.rbd1.in‚ 0.51220703125 disk.sda1.in‚0 disk.rbd3.in‚0" disk.nvme0c0n1.in‚ 38223.765625' disk.nvme0c0n1.out‚135118.169921875) disk./.usagePercent‚37.69921815094237 disk.sda1.out‚0 disk.rbd3.out‚0 cpu‚25.58171517291085 proc.cpu.threads‚17 disk.rbd1.out‚ 43.23046875 disk.sda3.out‚ 817.66796875$ memory_percent‚82.95621871948242" disk.dm-0.out‚135118.169921875* proc.memory.percent‚16.889333724975586¬²¢Ÿõ(‚(:” ©µéÒŸµÊö gpu.0.gpu‚99.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚273.894' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚54.778800000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0(Š$Ì?ö(‚(:´ ©µéÒüŸù& disk.rbd0.in‚1.759218604441423e+13 disk.sda1.out‚0 disk.drbd2165.out‚0 disk.rbd3.in‚0 disk.sda2.in‚0 disk.rbd1.in‚ 0.51220703125$ disk.dm-0.out‚135130.71240234375 disk.sda3.in‚ 75.31640625" disk.nvme0c0n1.in‚ 38223.984375 disk.dm-0.in‚ 38223.984375* proc.memory.percent‚16.889333724975586 network.sent‚ 1.9131781e+07 network.recv‚ 4.121578e+06$ disk./.usageGB‚561.7501945495605 disk.rbd0.out‚0 disk.rbd3.out‚0 disk.rbd1.out‚ 43.23046875$ memory_percent‚82.95328617095947* disk./.usagePercent‚37.699526896028665' disk.nvme0n1.out‚135130.71240234375 cpu‚25.581651867995774 disk.sda1.in‚0 disk.sda.in‚ 75.31640625 disk.dm-1.in‚ 75.31640625 disk.dm-1.out‚ 817.88671875 disk.nvme0n1.in‚ 38223.984375 disk.sda2.out‚0" proc.memory.rssMB‚ 2767.1484375 disk.sda3.out‚ 817.88671875) disk.nvme0c0n1.out‚135130.71240234375 disk.sda.out‚ 817.88671875 disk.drbd2165.in‚0) proc.memory.availableMB‚ 2792.93359375 proc.cpu.threads‚17;`ækž÷(‚(:“ ¸µéÒÓ©À÷ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚289.743' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚57.94859999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0EàØõ$ø(‚(:™ ¸µéÒÅ¥û disk.rbd1.out‚ 43.29296875& disk.nvme0n1.out‚135140.0361328125 network.sent‚ 1.9137398e+07$ disk./.usageGB‚561.7595252990723 disk.sda3.in‚ 75.31640625 disk.dm-1.out‚818.125$ memory_percent‚82.95555114746094" proc.memory.rssMB‚ 2767.1484375 cpu‚25.58158736515636& disk.rbd0.in‚1.759218604441423e+13 disk.dm-0.in‚ 38224.40625 disk.sda1.in‚0 disk.nvme0n1.in‚ 38224.40625 disk.sda.in‚ 75.31640625 network.recv‚ 4.124975e+06 disk.sda3.out‚818.125 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda.out‚818.125) disk./.usagePercent‚37.70015309045742 disk.rbd0.out‚0# disk.dm-0.out‚135140.0361328125 disk.drbd2165.in‚0 disk.rbd3.in‚0 disk.rbd3.out‚0( disk.nvme0c0n1.out‚135140.0361328125* proc.memory.percent‚16.889333724975586 proc.cpu.threads‚17 disk.drbd2165.out‚0 disk.rbd1.in‚ 0.51220703125 disk.dm-1.in‚ 75.31640625% proc.memory.availableMB‚ 2792.5625 disk.sda1.out‚0! disk.nvme0c0n1.in‚ 38224.40625æ†á’ù(‚(:‡ ǵéÒ‘Ö‹÷ gpu.0.gpu‚5.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚334.442' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚66.8884 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0£Þ?ú(‚(:´ ǵéÒ²ìÍù$ disk./.usageGB‚561.7624092102051 disk.rbd1.in‚ 0.51220703125 disk.sda3.out‚ 818.484375 disk.drbd2165.in‚0 disk.rbd3.out‚0 disk.sda1.in‚0 disk.sda3.in‚ 75.31640625) proc.memory.availableMB‚ 2792.55078125' disk.nvme0n1.out‚135155.57958984375 disk.dm-0.in‚38371.82421875 proc.cpu.threads‚17 disk.rbd3.in‚0$ disk.nvme0c0n1.in‚38371.82421875 disk.sda.out‚ 818.484375 disk.dm-1.in‚ 75.31640625 disk.sda1.out‚0* disk./.usagePercent‚37.700346632153305 disk.sda2.out‚0 disk.dm-1.out‚ 818.484375$ memory_percent‚82.95562267303467 network.recv‚ 4.128604e+06 disk.drbd2165.out‚0) disk.nvme0c0n1.out‚135155.57958984375 disk.sda2.in‚0" proc.memory.rssMB‚ 2767.1484375* proc.memory.percent‚16.889333724975586 network.sent‚ 1.9142039e+07" disk.nvme0n1.in‚38371.82421875 disk.rbd1.out‚ 43.29296875& disk.rbd0.in‚1.759218604441423e+13 disk.rbd0.out‚0 disk.sda.in‚ 75.31640625$ disk.dm-0.out‚135155.57958984375 cpu‚25.581533112912403MI‚¡4û(/ _wandbruntime‚16570  _runtime‚16570×”,Æü(À trainer/global_step‚9999 % _timestamp‚1.7843063867207613e+09 _runtime‚16570.941049976 _step‚820 % lr-AdamW/pg1‚0.000357067061056109 % lr-AdamW/pg2‚0.000357067061056109åÖÂËý(Å % _timestamp‚1.7843063867207613e+09 _runtime‚16570.941049976 _step‚820 % lr-AdamW/pg1‚0.000357067061056109 % lr-AdamW/pg2‚0.000357067061056109 trainer/global_step‚9999´x¥õ4þ(/ _wandbruntime‚16570  _runtime‚16570mÎ¥¢(ÿ(ÿO BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9404049346048 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2881369035581952 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2871966357798912 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2871966331625472 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9402703956480 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2880114071130490 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2870726084546560 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9386616555640 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2871966331625472 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚776210522083328 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚25648236 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 % _timestamp‚1.7843063867247174e+09 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 171804796 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚788343556059136 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _step‚821 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 _runtime‚16570.94202693 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚25648231 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚243 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 171794817 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚28326607 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚86572281 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9402703956480 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚86572281 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚28326850 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚86562308 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12132631348224 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2881369035581952 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚85232515 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚776210890174464 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚85232509 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9404049346048 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚248 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6Ë𺳠(€)„P M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚25648231 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚243 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 171794817 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚28326607 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚86572281 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9402703956480 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚86572281 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚28326850 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚86562308 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12132631348224 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2881369035581952 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚85232515 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚776210890174464 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚85232509 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9404049346048 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚248 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9404049346048 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2881369035581952 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2871966357798912 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2871966331625472 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9402703956480 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2880114071130490 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2870726084546560 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9386616555640 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2871966331625472 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚776210522083328 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚25648236 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 % _timestamp‚1.7843063867247174e+09 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 171804796 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚788343556059136 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _step‚821 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 _runtime‚16570.94202693µ¶Wûî4)/ _wandbruntime‚16572  _runtime‚16572Ž×´TD ‚)ŠN E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9403628992000 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9387540021886 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚86580773 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚85240715 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚788422237756416 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2871003966324736 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 _step‚822 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚245 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 % _timestamp‚1.7843063881128998e+09 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚25650774 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚86570786 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStat¾ÁgÌsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9388884622320 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2872244332355584 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2872244358529024 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚53980344 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚776288008491008 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 171821494 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 171821494 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚85240721 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12133826644992 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚53980099 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚788421835136000 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 171811501 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2880391506346622 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚86570786 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚85240715 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚240 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚28329325 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2881649332917760 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚25650779 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9404974388736 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2880392876375024 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚16572.330962297 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12133861174272 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2871003991752704ª•J0'ƒ)N ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚86570786 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚85240715 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚240 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚28329325 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2881649332917760 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚25650779 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9404974388736 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2880392876375024 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚16572.330962297 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12133861174272 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2871003991752704 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9403628992000 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9387540021886 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚86580773 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚85240715 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚788422237756416 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2871003966324736 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 _step‚822 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚245 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 % _timestamp‚1.7843063881128998e+09 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚25650774 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚86570786 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9388884622320 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2872244332355584 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2872244358529024 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚53980344 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚776288008491008 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 171821494 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 171821494 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚85240721 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12133826644992 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚53980099 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚788421835136000 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 171811501 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2880391506346622¶ëfܺ4„)/ _wandbruntime‚16572  _runtime‚16572ë ‚Ö …)… /loss/FRt2/detection_loss‚1.4449410580251107 %loss/total_loss‚6.994560831015159 # _timestamp‚1.78430638819624e+09 _step‚823 4loss/FBt1/assignment_loss‚0.00012689594692089404 4loss/FBt2/assignment_loss‚2.4439950219866676e-05 0loss/FBt2/detection_loss‚0.30897926235880757 epoch‚24 2loss/SRqqt2/assignment_loss‚0.3912346955892109 0loss/SRqqt1/detection_loss‚0.737541619705327 0loss/SRqqt2/detection_loss‚0.622548681025496 0loss/FRt2/assignment_loss‚0.5204034420135228 /loss/FRt1/detection_loss‚0.4869325077226007 /loss/FBt1/detection_loss‚0.7431816025157733 trainer/global_step‚9999 _runtime‚16572.413232377 /loss/FRt1/assignment_loss‚1.346546307533652 2loss/SRqqt1/assignment_loss‚0.3921003186285172§É܆)Š 2loss/SRqqt2/assignment_loss‚0.3912346955892109 0loss/SRqqt1/detection_loss‚0.737541619705327 0loss/SRqqt2/detection_loss‚0.622548681025496 0loss/FRt2/assignment_loss‚0.5204034420135228 /loss/FRt1/detection_loss‚0.4869325077226007 /loss/FBt1/detection_loss‚0.7431816025157733 trainer/global_step‚9999 _runtime‚16572.413232377 /loss/FRt1/assignment_loss‚1.346546307533652 2loss/SRqqt1/assignment_loss‚0.3921003186285172 /loss/FRt2/detection_loss‚1.4449410580251107 %loss/total_loss‚6.994560831015159 # _timestamp‚1.78430638819624e+09 _step‚823 4loss/FBt1/assignment_loss‚0.00012689594692089404 4loss/FBt2/assignment_loss‚2.4439950219866676e-05 0loss/FBt2/detection_loss‚0.30897926235880757 epoch‚24·"#”Gž‡)‚(:“ ÖµéÒÓ¨™÷ gpu.0.gpu‚19.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚37.0 gpu.0.powerWatts‚109.757' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.951400000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0HãÌ:>ˆ)‚(:³ ÖµéÒõ²ìù) disk.nvme0c0n1.out‚135163.93017578125' disk.nvme0n1.out‚135163.93017578125 disk.sda2.out‚0 disk.rbd1.out‚ 43.35546875$ disk.dm-0.out‚135163.93017578125 disk.sda1.out‚0 disk.drbd2165.in‚0 network.sent‚ 1.9147769e+07 network.recv‚ 4.131836e+06 disk.rbd1.in‚ 0.51220703125 disk.rbd0.out‚0# disk.nvme0c0n1.in‚ 38372.0703125* proc.memory.percent‚17.081403732299805! disk.nvme0n1.in‚ 38372.0703125 disk.dm-1.in‚ 75.31640625$ memory_percent‚83.14898014068604 proc.cpu.threads‚17 disk.rbd3.out‚0 disk.sda2.in‚0 disk.sda.in‚ 75.31640625 disk.dm-0.in‚ 38372.0703125 cpu‚25.581729774426076) disk./.usagePercent‚37.70089218420877$ disk./.usageGB‚561.7705383300781 disk.dm-1.out‚ 818.7734375 disk.sda3.out‚ 818.7734375" proc.memory.rssMB‚ 2798.6171875 disk.sda.out‚ 818.7734375& disk.rbd0.in‚1.759218604441423e+13 disk.sda1.in‚0 disk.sda3.in‚ 75.31640625 disk.drbd2165.out‚0 disk.rbd3.in‚0) proc.memory.availableMB‚ 2760.87109375ýZÍH’‰)‚(:‡ åµéÒå“Ç÷ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚41.0 gpu.0.powerWatts‚241.606' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚48.3212 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0€U §4Š)‚(:© åµéÒðØÆú disk.sda.out‚ 819.16796875) proc.memory.availableMB‚ 2760.57421875 proc.cpu.threads‚17 disk.sda3.out‚ 819.16796875 disk.drbd2165.in‚0 disk.drbd2165.out‚0# disk.nvme0c0n1.in‚ 38372.4921875 disk.dm-1.out‚ 819.16796875& disk.rbd0.in‚1.759218604441423e+13' disk.nvme0n1.out‚135179.26025390625 disk.rbd1.out‚ 43.35546875 network.recv‚ 4.136772e+06 disk.sda.in‚75.4375 disk.sda1.in‚0# memory_percent‚83.1507921218872$ disk./.usageGB‚561.7771873474121 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd0.out‚0$ disk.dm-0.out‚135179.26025390625* proc.memory.percent‚17.083430290222168 cpu‚25.583354162843108 disk.dm-0.in‚ 38372.4921875 network.sent‚ 1.921088e+07) disk./.usagePercent‚37.70133840534095 disk.sda3.in‚75.4375 disk.rbd3.out‚0! disk.nvme0n1.in‚ 38372.4921875 disk.rbd1.in‚ 0.51220703125 disk.rbd3.in‚0) disk.nvme0c0n1.out‚135179.26025390625# proc.memory.rssMB‚ 2798.94921875 disk.dm-1.in‚75.4375 disk.sda1.out‚0¾zD4‹)/ _wandbruntime‚16600  _runtime‚16600\c‚q­)Œ)°{ 6 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.984049379825592 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.2871857285499573 epoch‚24 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6230626106262207 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8000189661979675 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.07407407462596893 8"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.636248767375946 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 6 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.995473325252533 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9992673993110657 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7857142686843872 8"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.623096764087677 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.22093741595745087 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4244142770767212 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.5739468932151794 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9946363568305969 8"Purity/0FRt1SRqqt0FBt/event_purity‚0.577727198600769 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.2677956819534302 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.20317871868610382 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9841469526290894 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2407269924879074 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.1937226504087448 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9814814329147339 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.1695120632648468 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 *jet/accuracy_2_of_4‚0.7627373933792114 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6435250043869019 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9111376404762268 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.07407407462596893 )loss/val_total_loss‚8.200541960496997 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8119425773620605 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9934239983558655 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 ) Purity/2FRt*SRqqt2FBt/FRt_purity‚0.75 jet/accuracy_4_of_5‚0 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8898369073867798 8"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.628451943397522 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7091548442840576 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.2193652093410492 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9992673993110657 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.4608497619628906 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.06880778074264526 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.22956469655036926 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.22895470261573792 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6577109694480896 particle/accuracy_3_of_3‚1 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4453125 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 particle/accuracy_2_of_2‚1 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.22956469655036926 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 (particle/accuracy‚0.8431687951087952 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.22096343338489532 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6592785716056824 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7142857313156128 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9814814329147339 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.030561288818717003 /particle/accuracy_4_of_4‚0.9554693102836609 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.46836066246032715 8"Purity/0FRt0SRqqt2FBt/event_purity‚0.967779278755188 8"Purity/0FRt1SRqqt1FBt/event_purity‚0.530422031879425 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.07728736102581024 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_3_of_4‚1 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 particle/accuracy_5_of_5‚0 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.2222222238779068 'particle/f_score‚0.6259215474128723 % _timestamp‚1.7843064160407379e+09 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9994588494300842 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.1829332709312439 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9960039854049683 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 9"Purity/2FRt*SRqqt1FBt/event_purity‚0.1274999976158142 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.7947667241096497 particle/accuracy_1_of_2‚1 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9629628658294678 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.20952801406383514 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6291900873184204 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5583664178848267 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.23831206560134888 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6586085557937622 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9944872260093689 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3499999940395355 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.625 particle/accuracy_1_of_5‚1 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.42824074625968933 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8572884798049927 8"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.577727198600769 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.24137987196445465 *jet/accuracy_2_of_3‚0.5250979661941528 particle/accuracy_1_of_1‚1 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.5897037982940674 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.06880778074264526 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6070290803909302 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 6 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.997705340385437 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9934527277946472 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6222434043884277 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.24138611555099487 , Purity/0FRt2SRqqt*FBt/FBt_purity‚0.96875 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7555729746818542 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.20883361995220184 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6145286560058594 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.02327420935034752 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.02922837808728218 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 particle/accuracy_1_of_3‚1 particle/accuracy_2_of_4‚1 jet/accuracy_5_of_5‚0 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.46220311522483826 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.07830582559108734 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.44212961196899414 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6849466562271118 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 *jet/accuracy_1_of_1‚0.3114287853240967 *jet/accuracy_1_of_3‚0.9062774181365967 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9194869995117188 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.03846153989434242 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5879109501838684 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7101277709007263 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7731481194496155 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8595771789550781 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.2894499599933624 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.17446844279766083 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5354166626930237 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9984145164489746 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.044893279671669006 +jet/accuracy_2_of_2‚0.14055269956588745 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8888888955116272 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7111278176307678 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.34384772181510925 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3404027223587036 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2212146520614624 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 particle/accuracy_4_of_5‚1 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7255368232727051 +particle/sensitivity‚0.6109967231750488 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6395161747932434 *jet/accuracy_1_of_4‚0.9659546613693237 7validation_average_jet_accuracy‚0.35258054733276367 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6360713243484497 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9941149950027466 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚]í‹ 0.9970238208770752 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 , Purity/0FRt2SRqqt1FBt/FBt_purity‚0.96875 trainer/global_step‚9999 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 *jet/accuracy_1_of_2‚0.7510433197021484 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9807692170143127 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9863505363464355 6 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.644136905670166 8"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.587178647518158 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8000189661979675 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 jet/accuracy_3_of_5‚1 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9977208971977234 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.07475484162569046 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.6808780431747437 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.2864583134651184 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6223958730697632 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9949269890785217 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.3901388645172119 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5645931363105774 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.015115022659301758 _runtime‚16600.258593226 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.2222222238779068 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.21889731287956238 particle/accuracy_2_of_5‚1 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 6 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.983889639377594 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6499647498130798 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.17115171253681183 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.24021141231060028 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8111489415168762 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.17313158512115479 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7332388758659363 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.17982056736946106 6 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.988095223903656 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.18319837749004364 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7731481194496155 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.25531670451164246 particle/accuracy_1_of_4‚1 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5580130219459534 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6262549757957458 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5800457000732422 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.3784973919391632 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9968253374099731 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.7947667241096497 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 particle/accuracy_3_of_5‚1 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3047424852848053 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7329127192497253 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ,jet/accuracy_4_of_4‚0.013899499550461769 particle/accuracy_2_of_3‚1 8"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.685875654220581 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9945332407951355 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.04143347218632698 *jet/accuracy_3_of_3‚0.0703614130616188 jet/accuracy_1_of_5‚1 jet/accuracy_2_of_5‚1 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8082955479621887 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7160610556602478 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 +jet/accuracy_3_of_4‚0.09555235505104065 +particle/specificity‚0.9066706895828247 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.03846153989434242 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6040095090866089 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.07537725567817688 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 _step‚824±Ý0»=)µ{ 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9194869995117188 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.03846153989434242 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5879109501838684 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7101277709007263 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7731481194496155 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8595771789550781 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.2894499599933624 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.17446844279766083 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5354166626930237 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9984145164489746 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.044893279671669006 +jet/accuracy_2_of_2‚0.14055269956588745 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8888888955116272 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7111278176307678 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.34384772181510925 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3404027223587036 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.2212146520614624 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 particle/accuracy_4_of_5‚1 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7255368232727051 +particle/sensitivity‚0.6109967231750488 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6395161747932434 *jet/accuracy_1_of_4‚0.9659546613693237 7validation_average_jet_accuracy‚0.35258054733276367 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6360713243484497 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9941149950027466 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 , Purity/0FRt2SRqqt1FBt/FBt_purity‚0.96875 trainer/global_step‚9999 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 *jet/accuracy_1_of_2‚0.7510433197021484 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9807692170143127 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9863505363464355 6 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.644136905670166 8"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.587178647518158 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8000189661979675 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 jet/accuracy_3_of_5‚1 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9977208971977234 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.07475484162569046 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.6808780431747437 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.2864583134651184 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6223958730697632 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9949269890785217 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.3901388645172119 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5645931363105774 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.015115022659301758 _runtime‚16600.258593226 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.2222222238779068 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.21889731287956238 particle/accuracy_2_of_5‚1 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 6 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.983889639377594 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6499647498130798 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.17115171253681183 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.24021141231060028 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8111489415168762 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.17313158512115479 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7332388758659363 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.17982056736946106 6 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.988095223903656 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.18319837749004364 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7731481194496155 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.25531670451164246 particle/accuracy_1_of_4‚1 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5580130219459534 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6262549757957458 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5800457000732422 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.3784973919391632 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9968253374099731 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.7947667241096497 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 particle/accuracy_3_of_5‚1 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3047424852848053 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7329127192497253 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ,jet/accuracy_4_of_4‚0.013899499550461769 particle/accuracy_2_of_3‚1 8"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.685875654220581 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9945332407951355 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.04143347218632698 *jet/accuracy_3_of_3‚0.0703614130616188 jet/accuracy_1_of_5‚1 jet/accuracy_2_of_5‚1 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8082955479621887 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7160610556602478 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 +jet/accuracy_3_of_4‚0.09555235505104065 +particle/specificity‚0.9066706895828247 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.03846153989434242 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6040095090866089 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.07537725567817688 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 _step‚824 6 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.984049379825592 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.2871857285499573 epoch‚24 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6230626106262207 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8000189661979675 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.07407407462596893 8"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.636248767375946 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 6 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.995473325252533 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9992673993110657 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7857142686843872 8"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.623096764087677 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.22093741595745087 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4244142770767212 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.5739468932151794 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9946363568305969 8"Purity/0FRt1SRqqt0FBt/event_purity‚0.577727198600769 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.2677956819534302 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.20317871868610382 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9841469526290894 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2407269924879074 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.1937226504087448 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9814814329147339 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.1695120632648468 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 *jet/accuracy_2_of_4‚0.7627373933792114 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6435250043869019 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9111376404762268 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.07407407462596893 )loss/val_total_loss‚8.200541960496997 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8119425773620605 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9934239983558655 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 ) Purity/2FRt*SRqqt2FBt/FRt_purity‚0.75 jet/accuracy_4_of_5‚0 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8898369073867798 8"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.628451943397522 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7091548442840576 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.2193652093410492 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9992673993110657 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.4608497619628906 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.06880778074264526 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.22956469655036926 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.22895470261573792 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6577109694480896 particle/accuracy_3_of_3‚1 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4453125 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 particle/accuracy_2_of_2‚1 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.22956469655036926 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 (particle/accuracy‚0.8431687951087952 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.22096343338489532 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6592785716056824 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7142857313156128 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9814814329147339 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.030561288818717003 /particle/accuracy_4_of_4‚0.9554693102836609 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.46836066246032715 8"Purity/0FRt0SRqqt2FBt/event_purity‚0.967779278755188 8"Purity/0FRt1SRqqt1FBt/event_purity‚0.530422031879425 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.07728736102581024 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_3_of_4‚1 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 particle/accuracy_5_of_5‚0 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.2222222238779068 'particle/f_score‚0.6259215474128723 % _timestamp‚1.7843064160407379e+09 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9994588494300842 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.1829332709312439 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9960039854049683 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 9"Purity/2FRt*SRqqt1FBt/event_purity‚0.1274999976158142 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.7947667241096497 particle/accuracy_1_of_2‚1 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9629628658294678 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.20952801406383514 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6291900873184204 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5583664178848267 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.23831206560134888 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6586085557937622 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9944872260093689 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3499999940395355 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.625 particle/accuracy_1_of_5‚1 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.42824074625968933 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8572884798049927 8"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.577727198600769 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.24137987196445465 *jet/accuracy_2_of_3‚0.5250979661941528 particle/accuracy_1_of_1‚1 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.5897037982940674 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.06880778074264526 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6070290803909302 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 6 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.997705340385437 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9934527277946472 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6222434043884277 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.24138611555099487 , Purity/0FRt2SRqqt*FBt/FBt_purity‚0.96875 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7555729746818542 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.20883361995220184 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6145286560058594 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.02327420935034752 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.02922837808728218 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 particle/accuracy_1_of_3‚1 particle/accuracy_2_of_4‚1 jet/accuracy_5_of_5‚0 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.46220311522483826 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.07830582559108734 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.44212961196899414 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6849466562271118 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 *jet/accuracy_1_of_1‚0.3114287853240967 *jet/accuracy_1_of_3‚0.9062774181365967¸À4ÿË“Ž)‚(:ˆ ôµéҹȉ÷ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚287.615' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚57.523 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0>$“µ+)‚(:  ôµéÒ¶ÆÂù# memory_percent‚83.1735372543335) proc.memory.availableMB‚ 2756.84765625* disk./.usagePercent‚37.701644334344365 disk.rbd3.in‚0& disk.nvme0n1.out‚135186.9580078125( disk.nvme0c0n1.out‚135186.9580078125 disk.rbd0.out‚0 network.sent‚ 1.9215421e+07! disk.nvme0n1.in‚ 38372.6953125 disk.dm-1.in‚75.4375& disk.rbd0.in‚1.759218604441423e+13 disk.sda1.in‚0 disk.sda1.out‚0* proc.memory.percent‚17.107534408569336 disk.drbd2165.out‚0 disk.rbd3.out‚0# disk.nvme0c0n1.in‚ 38372.6953125 disk.sda2.out‚0 disk.dm-1.out‚ 819.515625 disk.sda.in‚75.4375 disk.sda.out‚ 819.515625 disk.dm-0.in‚ 38372.6953125# disk.dm-0.out‚135186.9580078125 disk.rbd1.out‚ 43.48046875 disk.sda3.in‚75.4375 cpu‚25.58459030236284 network.recv‚ 4.139886e+06" proc.memory.rssMB‚ 2802.8984375 proc.cpu.threads‚17 disk.sda3.out‚ 819.515625$ disk./.usageGB‚561.7817459106445 disk.drbd2165.in‚0 disk.sda2.in‚0 disk.rbd1.in‚ 0.51220703125B¬…™’)‚(:‡ ƒ¶éÒ”†ðö gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚339.274' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚67.8548 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0’–².4‘)‚(:© ƒ¶éÒìÙîø( disk./.usagePercent‚37.7011548479389 disk.sda.out‚ 819.90234375 disk.drbd2165.out‚0$ disk.nvme0c0n1.in‚38830.58203125" proc.memory.rssMB‚ 2802.8984375* proc.memory.percent‚17.107534408569336$ disk./.usageGB‚561.7744522094727 disk.rbd0.out‚0 disk.sda1.in‚0 disk.sda2.in‚0( disk.nvme0c0n1.out‚135198.7919921875" disk.nvme0n1.in‚38830.58203125 disk.sda2.out‚0 network.recv‚ 4.144216e+06 disk.dm-1.in‚75.4375 disk.dm-0.in‚38830.58203125 disk.sda3.in‚75.4375 disk.rbd3.in‚0 disk.rbd3.out‚0& disk.nvme0n1.out‚135198.7919921875$ memory_percent‚83.17325115203857) proc.memory.availableMB‚ 2756.89453125# disk.dm-0.out‚135198.7919921875 disk.rbd1.out‚ 43.48046875 proc.cpu.threads‚17 disk.drbd2165.in‚0 disk.rbd1.in‚ 0.51220703125 disk.sda.in‚75.4375& disk.rbd0.in‚1.759218604441423e+13 disk.sda3.out‚ 819.90234375 network.sent‚ 1.9273566e+07 disk.dm-1.out‚ 819.90234375 disk.sda1.out‚0 cpu‚25.584636347388447þ:Ïž’)‚(:“ ’¶éÒ¶Íåö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚264.47' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚52.894000000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ŒµTÉ9“)‚(:® ’¶éÒà…ù network.sent‚ 1.927866e+07 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda.in‚ 80.37109375 disk.dm-1.out‚ 827.66015625 disk.sda3.out‚ 827.66015625* proc.memory.percent‚17.107534408569336( disk./.usagePercent‚37.7016072132519 disk.dm-1.in‚ 80.37109375 disk.rbd0.out‚0 disk.rbd1.out‚ 43.54296875 disk.rbd3.out‚0) disk.nvme0c0n1.out‚135212.30029296875 disk.nvme0n1.in‚ 38866.640625' disk.nvme0n1.out‚135212.30029296875& disk.rbd0.in‚1.759218604441423e+13# disk./.usageGB‚561.781192779541 disk.sda1.out‚0 network.recv‚ 4.147492e+06" disk.nvme0c0n1.in‚ 38866.640625$ disk.dm-0.out‚135212.30029296875 disk.sda1.in‚0( proc.memory.availableMB‚ 2756.8828125 disk.sda.out‚ 827.66015625 disk.dm-0.in‚ 38866.640625 disk.drbd2165.in‚0 disk.drbd2165.out‚0 cpu‚25.584649416273546 proc.cpu.threads‚17 disk.sda3.in‚ 80.37109375 disk.rbd3.in‚0 disk.rbd1.in‚ 0.51220703125# memory_percent‚83.1733226776123" proc.memory.rssMB‚ 2802.8984375`PÌ””)‚(:‰ ¡¶éÒÌüÞö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚297.097' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚59.4194 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0«!7•)‚(:¬ ¡¶éÒ꧇ù disk.dm-1.in‚ 80.37890625* proc.memory.percent‚17.107534408569336 disk.rbd0.out‚0 disk.drbd2165.in‚0! disk.nvme0c0n1.in‚ 38866.84375 disk.rbd1.in‚ 0.51220703125 disk.sda.in‚ 80.37890625& disk.rbd0.in‚1.759218604441423e+13( proc.memory.availableMB‚ 2756.8984375 disk.drbd2165.out‚0 disk.rbd3.in‚0' disk.nvme0n1.out‚135222.27001953125 cpu‚25.58463338206933$ disk./.usageGB‚561.7856330871582 disk.sda.out‚ 864.7890625 network.recv‚ 4.151229e+06* disk./.usagePercent‚37.701905206021756 disk.sda1.in‚0) disk.nvme0c0n1.out‚135222.27001953125 disk.sda2.in‚0$ memory_percent‚83.17322731018066 disk.sda3.in‚ 80.37890625 disk.sda2.out‚0" proc.memory.rssMB‚ 2802.8984375 network.sent‚ 1.9283729e+07 disk.sda1.out‚0 disk.sda3.out‚ 864.7890625 disk.rbd1.out‚ 43.54296875 disk.dm-1.out‚ 864.7890625 disk.rbd3.out‚0 proc.cpu.threads‚17 disk.dm-0.in‚ 38866.84375$ disk.dm-0.out‚135222.27001953125 disk.nvme0n1.in‚ 38866.84375Á´R¼ž–)‚(:“ °¶éң͜ø gpu.0.gpu‚90.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚288.679' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚57.73579999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0im³C=—)‚(:² °¶éÒàß¿ù disk.rbd1.in‚ 0.51220703125 disk.dm-1.out‚ 1061.0390625 disk.rbd0.out‚0 disk.sda3.out‚ 1061.0390625& disk.rbd0.in‚1.759218604441423e+13' proc.memory.availableMB‚ 2756.890625* proc.memory.percent‚17.107534408569336 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.nvme0n1.in‚ 38867.265625' disk.nvme0n1.out‚135231.24755859375 disk.sda3.in‚ 106.2109375 disk.dm-1.in‚ 106.2109375$ disk.dm-0.out‚135231.24755859375 disk.rbd3.in‚0 disk.rbd3.out‚0$ memory_percent‚83.17327499389648" proc.memory.rssMB‚ 2802.8984375 proc.cpu.threads‚17 disk.dm-0.in‚ 38867.265625 cpu‚25.584585860205205 disk.sda2.out‚0 disk.sda.in‚ 106.2109375 disk.sda.out‚ 1061.0390625 disk.drbd2165.out‚0* disk./.usagePercent‚37.702533960525855$ disk./.usageGB‚561.7950019836426) disk.nvme0c0n1.out‚135231.24755859375 disk.rbd1.out‚ 43.60546875 disk.sda1.in‚0 network.sent‚ 1.9288793e+07 network.recv‚ 4.154813e+06" disk.nvme0c0n1.in‚ 38867.265625 disk.sda2.in‚0r@3ƒ4˜)/ _wandbruntime‚16678  _runtime‚16678¶s½âË™)Å % _timestamp‚1.7843064945078273e+09 _runtime‚16678.725233683 _step‚825 ' lr-AdamW/pg1‚0.00035429584243259685 ' lr-AdamW/pg2‚0.00035429584243259685 trainer/global_step‚10049Ûó>$К)Ê _step‚825 ' lr-AdamW/pg1‚0.00035429584243259685 ' lr-AdamW/pg2‚0.00035429584243259685 trainer/global_step‚10049 % _timestamp‚1.7843064945078273e+09 _runtime‚16678.725233683¹†R4›)/ _wandbruntime‚16678  _runtime‚16678@ÔC2Áœ)€P YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚85774815 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9456207888384 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚28498861 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2889694349565952 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2889694375739392 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9454862498816 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚25784299 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9440027400507 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚792333393334784 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2897886135216443 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2899149212064768 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 172831836 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚258 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12200070529536 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2899150583627776 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚87047042 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 G?DevuVôÙEiceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 $ _timestamp‚1.784306494509282e+09 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚85774815 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚28499119 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9438682800185 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚25784304 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2899149212064768 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9454862498816 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚54283423 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚87057015 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚87047042 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2899150583627776 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9456207888384 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2899149212064768 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 172831836 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚258 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12200070529536 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2899150583627776 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚87047042 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 $ _timestamp‚1.784306494509282e+09 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚85774815 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚28499119 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9438682800185 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚25784304 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2899149212064768 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9454862498816 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚54283423 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚87057015 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚87047042 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2899150583627776 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9456207888384 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6ºj4³è”ž)‚(:‰ ¿¶é҈ؕø gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚266.886' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.3772 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0OÀµ@Ÿ)‚(:µ ¿¶éÒò¡¹ø network.sent‚ 1.9294005e+07 network.recv‚ 4.158302e+06 disk.dm-1.out‚ 1063.22265625& disk.rbd0.in‚1.759218604441423e+13' disk.nvme0n1.out‚135246.25830078125 proc.cpu.threads‚17$ disk./.usageGB‚561.8148956298828$ disk.dm-0.out‚135246.25830078125 disk.sda3.in‚ 106.2109375 disk.sda2.out‚0$ memory_percent‚83.17575454711914* proc.memory.percent‚17.107534408569336" disk.nvme0c0n1.in‚ 38867.484375 disk.nvme0n1.in‚ 38867.484375' proc.memory.availableMB‚ 2756.484375 disk.rbd0.out‚0 disk.rbd3.out‚0 disk.sda1.out‚0) disk.nvme0c0n1.out‚135246.25830078125 disk.sda.out‚ 1063.22265625 disk.rbd1.out‚ 43.60546875 disk.sda.in‚ 106.2109375* disk./.usagePercent‚37.703869039816915 disk.dm-0.in‚ 38867.484375 disk.sda1.in‚0 disk.sda3.out‚ 1063.22265625 disk.sda2.in‚0 disk.rbd1.in‚ 0.51220703125 disk.dm-1.in‚ 106.2109375 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.rbd3.in‚0" proc.memory.rssMB‚ 2802.8984375 cpu‚25.584513468297903iDF*4 )/ _wandbruntime‚16680  _runtime‚16680¥KHÃ'¡)ŠN D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 172838541 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2899428137830400 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚87055520 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚54288718 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚32432128 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2899429509400576 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 400523264 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2888723964166144 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚85783021 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚54288466 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 172848534 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚25786847 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 172838541 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚780210809212928 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12201277434368 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2889972376469504 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚85783021 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2889972350296064 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9455787534336 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 172848534 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 _step‚827 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚247 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚28501624 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚792412086647296 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9439606266431 trainer/global_step‚10049 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12201245002240 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 _runtime‚16680.11315666Ø„±Îg¢)N QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12201245002240 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 _runtime‚16680.11315666 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 172838541 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2899428137830400 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚87055520 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚54288718 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚32432128 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 Sª DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚85783021 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚54288466 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 172848534 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚25786847 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 172838541 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚780210809212928 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12201277434368 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2889972376469504 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚85783021 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2889972350296064 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9455787534336 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 172848534 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 _step‚827 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚247 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚28501624 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚792412086647296 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9439606266431 trainer/global_step‚10049»Vt¯4£)/ _wandbruntime‚16680  _runtime‚16680]„IM ¤)„ 1loss/SRqqt1/detection_loss‚0.6474359582522831 1loss/SRqqt2/detection_loss‚0.5367665085629937 %loss/total_loss‚6.583933200437829 $ _timestamp‚1.784306495981449e+09 /loss/FBt2/detection_loss‚0.4644200604118903 trainer/global_step‚10049 0loss/FRt1/assignment_loss‚1.2517875903158344 0loss/FRt2/assignment_loss‚0.4598285748263982 .loss/FBt1/detection_loss‚0.654568253304266 epoch‚25 _step‚828 2loss/SRqqt1/assignment_loss‚0.3075617160621217 3loss/SRqqt2/assignment_loss‚0.31818886261802143 1loss/FBt1/assignment_loss‚0.02220117840128558 2loss/FBt2/assignment_loss‚0.006538866277117888 0loss/FRt1/detection_loss‚0.47027063822771453 _runtime‚16680.198407534 .loss/FRt2/detection_loss‚1.444364993177902Ër}*¥)‰ .loss/FRt2/detection_loss‚1.444364993177902 1loss/SRqqt1/detection_loss‚0.6474359582522831 1loss/SRqqt2/detection_loss‚0.5367665085629937 %loss/total_loss‚6.583933200437829 $ _timestamp‚1.784306495981449e+09 /loss/FBt2/detection_loss‚0.4644200604118903 trainer/global_step‚10049 0loss/FRt1/assignment_loss‚1.2517875903158344 0loss/FRt2/assignment_loss‚0.4598285748263982 .loss/FBt1/detection_loss‚0.654568253304266 epoch‚25 _step‚828 2loss/SRqqt1/assignment_loss‚0.3075617160621217 3loss/SRqqt2/assignment_loss‚0.31818886261802143 1loss/FBt1/assignment_loss‚0.02220117840128558 2loss/FBt2/assignment_loss‚0.006538866277117888 0loss/FRt1/detection_loss‚0.47027063822771453 _runtime‚16680.198407534¼–\àž¦)‚(:“ ζéÒïÖÜ÷ gpu.0.gpu‚83.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚341.161' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚68.23219999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¶P=§)‚(:² ζéÒ»€Ûø disk.dm-1.out‚ 1197.078125 disk.sda3.out‚ 1197.078125" proc.memory.rssMB‚ 2802.8984375 proc.cpu.threads‚17 network.recv‚ 4.162382e+06 disk.sda3.in‚ 106.21484375) proc.memory.availableMB‚ 2756.42578125' disk.nvme0n1.out‚135253.51123046875 disk.rbd1.in‚ 0.51220703125 disk.dm-1.in‚ 106.21484375) disk.nvme0c0n1.out‚135253.51123046875 disk.rbd3.in‚0 disk.sda2.out‚0 disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 39120.515625 disk.rbd1.out‚ 43.66796875* proc.memory.percent‚17.107534408569336& disk.rbd0.in‚1.759218604441423e+13 cpu‚25.584481468397716 disk.sda2.in‚0 disk.sda1.in‚0) disk./.usagePercent‚37.70430604467786$ disk./.usageGB‚561.8214073181152 disk.sda.in‚ 106.21484375 disk.sda.out‚ 1197.078125$ disk.dm-0.out‚135253.51123046875 disk.rbd3.out‚0 network.sent‚ 1.9346391e+07 disk.dm-0.in‚ 39120.515625 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.nvme0n1.in‚ 39120.515625# memory_percent‚83.17611217498782Æ¢Ÿ¨)‚(:” ݶéÒëãýö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚269.062' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚53.812400000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0&[=©)‚(:² ݶéÒ€þù' disk.nvme0n1.out‚135262.78369140625 disk.rbd1.out‚ 43.66796875 disk.sda.out‚ 1197.66796875 disk.sda1.in‚0 disk.sda3.in‚ 106.21484375 disk.drbd2165.in‚0 disk.rbd3.out‚0 disk.nvme0n1.in‚ 39120.9375$ disk.dm-0.out‚135262.78369140625) disk./.usagePercent‚37.70484750061278 disk.dm-1.out‚ 1197.66796875& disk.rbd0.in‚1.759218604441423e+13 disk.rbd0.out‚0* proc.memory.percent‚17.107534408569336# disk./.usageGB‚561.829475402832 disk.rbd1.in‚ 0.51220703125 disk.sda3.out‚ 1197.66796875 disk.sda2.in‚0 disk.dm-0.in‚ 39120.9375) proc.memory.availableMB‚ 2756.66796875 network.recv‚ 4.166748e+06 disk.drbd2165.out‚0 disk.rbd3.in‚0 disk.sda2.out‚0" proc.memory.rssMB‚ 2802.8984375 disk.nvme0c0n1.in‚ 39120.9375) disk.nvme0c0n1.out‚135262.78369140625 disk.sda.in‚ 106.21484375 disk.sda1.out‚0$ memory_percent‚83.17463397979736 cpu‚25.584434631530847 proc.cpu.threads‚17 network.sent‚ 1.9398562e+07 disk.dm-1.in‚ 106.21484375ì¥Û¥”ª)‚(:‰ ì¶éÒÞºíö gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚40.0 gpu.0.powerWatts‚145.233' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚29.0466 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚09Šù©@«)‚(:µ ì¶éÒž­¸ù network.recv‚ 4.170279e+06 disk.sda3.out‚ 1198.24609375 disk.sda2.out‚0 disk.sda.in‚ 106.21484375 disk.rbd0.out‚0* proc.memory.percent‚17.107534408569336 disk.sda1.in‚0! disk.nvme0c0n1.in‚ 39121.15625 disk.sda2.in‚0 disk.dm-1.in‚ 106.21484375 disk.sda1.out‚0" proc.memory.rssMB‚ 2802.8984375 proc.cpu.threads‚17 disk.sda3.in‚ 106.21484375 disk.drbd2165.in‚0 disk.sda.out‚ 1198.24609375 network.sent‚ 1.9404184e+07* disk./.usagePercent‚37.705143445322356 disk.nvme0n1.in‚ 39121.15625' disk.nvme0n1.out‚135275.72900390625$ disk.dm-0.out‚135275.72900390625# memory_percent‚83.1768274307251) disk.nvme0c0n1.out‚135275.72900390625& disk.rbd0.in‚1.759218604441423e+13) proc.memory.availableMB‚ 2756.30859375$ disk./.usageGB‚561.8338851928711 disk.drbd2165.out‚0 disk.rbd3.out‚0 disk.rbd1.in‚ 0.51220703125 disk.dm-1.out‚ 1198.24609375 disk.dm-0.in‚ 39121.15625 disk.rbd3.in‚0 disk.rbd1.out‚ 43.73046875 cpu‚25.58441738016714‡”Z~”¬)‚(:‰ û¶éÒ¼†ùö gpu.0.gpu‚90.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚240.261' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚48.0522 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0²¯ ªB­)‚(:· û¶éÒ«ð½ù disk.drbd2165.in‚0 disk.rbd3.in‚0" proc.memory.rssMB‚ 2802.8984375 network.sent‚ 1.9409295e+07 disk.rbd1.in‚ 0.51220703125 disk.dm-1.in‚ 106.21484375) disk.nvme0c0n1.out‚135285.70458984375 disk.dm-1.out‚ 1198.75390625 disk.sda2.in‚0 disk.sda2.out‚0$ memory_percent‚83.17704200744629 cpu‚25.584371506488456) disk./.usagePercent‚37.70558761839427 disk.sda.out‚ 1198.75390625 disk.sda1.out‚0 disk.sda3.out‚ 1198.75390625 disk.nvme0n1.in‚ 39121.359375 proc.cpu.threads‚17 disk.rbd3.out‚0* proc.memory.percent‚17.107534408569336 disk.dm-0.in‚ 39121.359375$ disk.dm-0.out‚135285.70458984375 disk.drbd2165.out‚0" disk.nvme0c0n1.in‚ 39121.359375( proc.memory.availableMB‚ 2756.2734375 disk.rbd0.out‚0& disk.rbd0.in‚1.759218604441423e+13 network.recv‚ 4.174034e+06# disk./.usageGB‚561.840503692627 disk.sda.in‚ 106.21484375' disk.nvme0n1.out‚135285.70458984375 disk.rbd1.out‚ 43.73046875 disk.sda1.in‚0 disk.sda3.in‚ 106.21484375çsËÕž®)‚(:“ Š·éÒŸ‹¸ø gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚281.143' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚56.22859999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Bš‡á>¯)‚(:³ Š·éÒ‰Áàû disk.rbd1.out‚ 43.85546875 disk.dm-1.in‚ 106.21484375 disk.sda1.in‚0" proc.memory.rssMB‚ 2802.8984375$ disk.nvme0c0n1.in‚39283.63671875 disk.rbd1.in‚ 0.51220703125 network.sent‚ 1.9414507e+07 disk.drbd2165.in‚0 disk.sda.in‚ 106.21484375 disk.sda.out‚ 1199.0390625 disk.rbd0.out‚0 disk.sda3.out‚ 1199.0390625' disk.nvme0c0n1.out‚135298.333984375 network.recv‚ 4.177544e+06* disk./.usagePercent‚37.706043567812756# disk./.usageGB‚561.847297668457" disk.nvme0n1.in‚39283.63671875 disk.sda2.in‚0 disk.sda2.out‚0& proc.memory.availableMB‚ 2756.28125" disk.dm-0.out‚135298.333984375 disk.sda1.out‚0 disk.sda3.in‚ 106.21484375 disk.drbd2165.out‚0 disk.rbd3.out‚0 cpu‚25.584303799522825 proc.cpu.threads‚17 disk.dm-0.in‚39283.63671875 disk.rbd3.in‚0$ memory_percent‚83.17699432373047 disk.dm-1.out‚ 1199.0390625% disk.nvme0n1.out‚135298.333984375& disk.rbd0.in‚1.759218604441423e+13* proc.memory.percent‚17.107534408569336¸ëNÉ4°)/ _wandbruntime‚16758  _runtime‚16758gÌ 8ʱ)Ä $ _timestamp‚1.784306574111782e+09 _runtime‚16758.329036112 _step‚829 ' lr-AdamW/pg1‚0.00035152408611580104 ' lr-AdamW/pg2‚0.00035152408611580104 trainer/global_step‚10099'‹˜Ï²)É trainer/global_step‚10099 $ _timestamp‚1.784306574111782e+09 _runtime‚16758.329036112 _step‚829 ' lr-AdamW/pg1‚0.00035152408611580104 ' lr-AdamW/pg2‚0.00035152408611580104½Lù|L4³)/ _wandbruntime‚16758  _runtime‚16758 DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚28636472 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚54547926 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚86185121 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚276 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚28636196 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚796267680637440 trainer/global_step‚10099 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404724736 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 % _timestamp‚1.7843065741185362e+09 _runtime‚16758.336217197 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2913095564500992 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚281 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚25911449 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2911826461238351 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚87472794 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚54547645 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _step‚830 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2903594386071552 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 173667888 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚25911454øóo4µ)†P KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚86185121 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9486264513615 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚86185121 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚276 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚28636196 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚796267680637440 trainer/global_step‚10099 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404724736 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 % _timestamp‚1.7843065741185362e+09 _runtime‚16758.336217197 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2913095564500992 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚281 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚25911449 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2911826461238351 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚87472794 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚54547645 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _step‚830 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2903594386071552 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 173667888 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚25911454 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚86185115 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚28636472 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚54547926 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 J8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 173674593 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 _step‚831 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 173684586 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚784084761505792 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9487187979973 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 trainer/global_step‚10099 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚796345946753536 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚86193321 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9502103464960 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 % _timestamp‚1.7843065755063891e+09 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2913375861836800 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚87491259 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2903872412975104 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2903872412975104 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚86193321 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 173684586 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9503448861696 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚25913997 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚87481272 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 _runtime‚16759.723816402 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2912105266482885 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2902618053074944 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚87481272 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12261185247744 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚87491259 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚784085129596928 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚86193327 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9503448861696 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚54552840 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚28638848 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚796346351471104 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2903872386801664 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚29026180785029128úxé'¸)N ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚275 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2912103896454483 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚86193327 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9503448861696 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚54552840 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚28638848 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚796346351471104 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2903872386801664 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2902618078502912 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404717568 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 173674593 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 _step‚831 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 173684586 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚784084761505792 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9487187979973 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 trainer/global_step‚10099 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚796345946753536 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚86193321 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9502103464960 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 % _timestamp‚1.7843065755063891e+09 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2913375861836800 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚87491259 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2903872412975104 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2903872412975104 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚86193321 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 173684586 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9503448861696 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚25913997 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚87481272 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 _runtime‚16759.723816402 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2912105266482885 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2902618053074944 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚87481272 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12261185247744 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚87491259 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚784085129596928 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 QDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9547494360064 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2925765417231969 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2927041916937216 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚787881595482112 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚787881963573248 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚86595415 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚28774431 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚87898546 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2927043288500224 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2917494422577152 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚267 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12319554168320 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 _runtime‚16838.173958607 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2917494448750592 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9548839749632 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404724736 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12319590801920 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9531157026401 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚26038599 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2925766787260259 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2916234285633536 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚800201554375168 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚272 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2927043288500224 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9547494360064 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9548839749632 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚87908519 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2917494422577152 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 % _timestamp‚1.7843066539565957e+09 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚54813302 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚54813030 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚87908519 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2917494448750592 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚28774698 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 _step‚834 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚86595421 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚86595421 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚26038604 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚10149 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2916234260205568 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9532501626723 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚86595415 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36633600 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 174493961 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0\¾ (Ë)†P WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9547494360064 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9548839749632 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚87908519 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2917494422577152 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 % _timestamp‚1.7843066539565957e+09 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚54813302 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚54813030 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚87908519 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2917494448750592 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚28774698 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 _step‚834 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚86595421 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚86595421 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚26038604 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚10149 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2916234260205568 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9532501626723 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚86595415 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36633600 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 174493961 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2927041916937216 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9547494360064 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2925765417231969 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2927041916937216 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚787881595482112 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚787881963573248 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚86595415 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚28774431 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚87898546 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2927043288500224 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2917494422577152 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚267 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12319554168320 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 _runtime‚16838.173958607 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2917494448750592 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9548839749632 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404724736 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12319590801920 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9531157026401 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚26038599 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2925766787260259 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2916234285633536 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚800201554375168 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚272 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2927043288500224 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0Âñ}óh4Ì)/ _wandbruntime‚16839  _runtime‚16839‚wí'Í)‹N A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12320778684928 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚87917011 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚28777171 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2916512167411712 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 % _timestamp‚1.7843066553452184e+09 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚86603621 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚26041147 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404717568 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9532080492647 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚54818313 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚787959081889792 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚255 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2917772449480704 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚800280228665856 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚800279823948288 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2926045592504793 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2917772423307264 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9549764792320 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 174520638 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚260 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 174520638 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12320742058496 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9549764792320 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9548419395584 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚86603627 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2917772423307264 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚86603627 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 174510645 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚87917011 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 _runtime‚16839.562604129 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _step‚835 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚787959449980928 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚86603621 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚86603621 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚26041147 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_eõµåènd/allocation.small_pool.peak‚12097 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404717568 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9532080492647 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚54818313 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚787959081889792 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚255 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2917772449480704 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚800280228665856 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚800279823948288 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2926045592504793 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2917772423307264 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9549764792320 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 174520638 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚260 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 174520638 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12320742058496 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9549764792320 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9548419395584 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚86603627 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2917772423307264 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚86603627 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 174510645 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚87917011 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 _runtime‚16839.562604129 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _step‚835 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚787959449980928 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚86603621 FDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚87005715 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2931394459082752 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚791756283957248 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2940988269373440 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2939705743253877 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚26165754 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚28912910 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2940989640936448 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2939707113282167 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 175330013 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9593810290688 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2930128349114368 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2931394485256192 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚791755915866112 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2931394459082752 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚263 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9577394139509 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12379127788032 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 % _timestamp‚1.7843067334874618e+09íØ5 êá)†P WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9593810290688 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚838 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2931394485256192 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 175339992 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12379127788032 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 % _timestamp‚1.7843067334874618e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚258 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚804135043654144 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 _runtime‚16917.704713486 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9595155680256 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚26165749 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚88334271 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚55078659 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 trainer/global_step‚10199 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚88324298 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2940989640936448 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2940988269373440 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚88324298 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 ^LDeviceStatsMon†\º€"itor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9595155680256 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚87005715 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚88334271 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚28913168 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2930128374542336 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9578738739831 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚87005715 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2931394459082752 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚791756283957248 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2940988269373440 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2939705743253877 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚26165754 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚28912910 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2940989640936448 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2939707113282167 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 175330013 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9593810290688 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2930128349114368 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2931394485256192 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚791755915866112 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2931394459082752 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚263 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9577394139509 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2931672459812864 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 175356690 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 175346697 trainer/global_step‚10199 _runtime‚16919.09061892 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚87013921 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2931672459812864 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 175356690 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2941267195139072 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2930406230892544 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚839 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 175346697 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚55083953 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2941268566709248 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚87013927 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2941268566709248 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚87013927 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2931672459812864 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 175356690 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 175346697 trainer/global_step‚10199 _runtime‚16919.09061892 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚87013921 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2931672459812864 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 175356690 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2941267195139072 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2930406230892544 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚839 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 175346697 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚55083953 FDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 _step‚842 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12439033393664 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2954934621809664 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2953647439304075 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9641471610880 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 _runtime‚16997.539284076 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 176176044 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9624975852939 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚55344532 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚29051377 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚808069672271360 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12439067930112 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚55344276 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚795630236250112 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚256 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9623631252617 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 trainer/global_step‚10249 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2944022438023168 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚88760023 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚88750050 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚251 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚87416015 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚88750050 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚29051628 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2945294495588352 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚795630604341248 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2953646069275785 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2944022463451136 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9640126221312 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9641471610880 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚808069269643776 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2954934621809664 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2953647439304075 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9641471610880 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 _runtime‚16997.539284076 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 176176044 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9624975852939 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚55344532 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚29051377 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚808069672271360 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12439067930112 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚55344276 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚795630236250112 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚256 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9623631252617 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 trainer/global_step‚10249 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2944022438023168 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚88760023 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚88750050 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚251 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚87416015 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚88750050 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚29051628 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2945294495588352 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚795630604341248 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2953646069275785 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2944022463451136 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9640126221312 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9641471610880 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚808069269643776 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 _step‚842 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12439033393664 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 J8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9642396653568 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚243 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2955214919145472 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚795707722657792 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2955213547575296 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 _runtime‚16998.927952201 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚795708090748928 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12440231917056 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚29054382 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 176182749 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚55349829 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚26295442 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2945572496318464 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚26295447 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2944300345229312 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9625899319297 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9642396653568 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2944300319801344 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 trainer/global_step‚10249 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 176192742 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 176192742 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚87424221 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚88758528 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12440266446336 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2953924874520207 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9624554718863 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚87424227 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚88768515 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚55349581 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2945572522491904žÇ_'ü)N H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 176192742 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚87424221 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚88758528 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12440266446336 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2953924874520207 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9624554718863 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚87424227 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚88768515 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚55349581 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2945572522491904 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9642396653568 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚243 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2955214919145472 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚795707722657792 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2955213547575296 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 _runtime‚16998.927952201 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚795708090748928 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12440231917056 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚29054382 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 176182749 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚55349829 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚26295442 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2945572496318464 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚26295447 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2944300345229312 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9625899319297 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9642396653568 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2944300319801344 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 trainer/global_step‚10249 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 176192742 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995ˆ œk4ý)/ _wandbruntime‚16999  _runtime‚16999…!¥nþ)Œ 3loss/SRqqt2/assignment_loss‚0.32609917332757316 /loss/FRt2/detection_loss‚1.3186787267285398 %loss/total_loss‚6.285679599866997 3loss/SRqqt1/assignment_loss‚0.33924240550771656 1loss/SRqqt2/detection_loss‚0.5232102144411981 /loss/FBt2/detection_loss‚0.3653067410121219 1loss/FRt2/assignment_loss‚0.34968368396162985 /loss/FRt1/detection_loss‚0.4739102128719216 /loss/FBt1/detection_loss‚0.6251616754840749 epoch‚25 % _timestamp‚1.7843068147947862e+09 3loss/FBt1/assignment_loss‚2.921054030914484e-05 4loss/FBt2/assignment_loss‚2.1420680439415247e-05 1loss/SRqqt1/detection_loss‚0.7081575525398675 trainer/global_step‚10249 _runtime‚16999.011710081 _step‚844 0loss/FRt1/assignment_loss‚1.2561785827716045Òÿ)‘ %loss/total_loss‚6.285679599866997 3loss/SRqqt1/assignment_loss‚0.33924240550771656 1loss/SRqqt2/detection_loss‚0.5232102144411981 /loss/FBt2/detection_loss‚0.3653067410121219 1loss/FRt2/assignment_loss‚0.34968368396162985 /loss/FRt1/detection_loss‚0.4739102128719216 /loss/FBt1/detection_loss‚0.6251616754840749 epoch‚25 % _timestamp‚1.7843068147947862e+09 3loss/FBt1/assignment_loss‚2.921054030914484e-05 4loss/FBt2/assignment_loss‚2.1420680439415247e-05 1loss/SRqqt1/detection_loss‚0.7081575525398675 trainer/global_step‚10249 _runtime‚16999.011710081 _step‚844 0loss/FRt1/assignment_loss‚1.2561785827716045 3loss/SRqqt2/assignment_loss‚0.32609917332757316 /loss/FRt2/detection_loss‚1.3186787267285398ÌÜJ1¶”€*‚(:‰ ‰¹éÒËß«ø gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚277.374' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.4748 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¼«^rM*‚(: ‰¹éÒ¬äªú) proc.memory.availableMB‚ 2756.08203125 disk.sda3.in‚ 106.21484375 disk.sda3.out‚ 1206.16015625 disk.rbd0.out‚0* proc.memory.percent‚17.107534408569336 disk.dm-0.in‚39856.67578125 disk.sda.out‚ 1206.16015625 proc.cpu.threads‚17% disk.nvme0n1.out‚135932.111328125 disk.sda2.in‚0& disk.rbd0.in‚1.759218604441423e+13$ memory_percent‚83.17821025848389" disk.dm-0.out‚135932.111328125 disk.rbd3.in‚0$ disk.nvme0c0n1.in‚39856.67578125 disk.rbd1.in‚ 0.51220703125 network.recv‚ 4.238605e+06" disk.nvme0n1.in‚39856.67578125 disk.sda.in‚ 106.21484375 cpu‚25.58400013555868 disk.rbd3.out‚ 0.07421875$ disk./.usageGB‚562.8550148010254 disk.sda1.in‚0 disk.drbd2165.in‚0 network.sent‚ 1.9674994e+07* disk./.usagePercent‚37.773672310110406 disk.dm-1.in‚ 106.21484375" proc.memory.rssMB‚ 2802.8984375 disk.sda1.out‚0 disk.drbd2165.out‚0' disk.nvme0c0n1.out‚135932.111328125 disk.sda2.out‚0 disk.rbd1.out‚ 44.36328125 disk.dm-1.out‚ 1206.16015625 ¢-ž‚*‚(:“ ˜¹éÒ…»âù gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚337.681' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚67.53620000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‘[ªHƒ*‚(:½ ˜¹éÒ—žóû disk.rbd0.out‚0 disk.sda2.out‚0' proc.memory.availableMB‚ 2815.328125 disk.sda1.in‚0 disk.drbd2165.out‚0$ disk.nvme0c0n1.in‚39856.87890625' disk.nvme0c0n1.out‚137648.267578125 proc.cpu.threads‚17 disk.sda2.in‚0 disk.sda.out‚ 1206.7109375 disk.drbd2165.in‚0 disk.rbd3.in‚0" disk.dm-0.out‚137648.267578125 network.sent‚ 1.9737604e+07* disk./.usagePercent‚37.868237909153294 disk.sda.in‚ 106.21484375* proc.memory.percent‚17.107534408569336 cpu‚25.58409730462155 disk.sda3.out‚ 1206.7109375" disk.nvme0n1.in‚39856.87890625$ memory_percent‚82.81660079956055" proc.memory.rssMB‚ 2802.8984375 disk.rbd1.in‚ 0.51220703125 disk.rbd1.out‚ 44.42578125& disk.rbd0.in‚1.759218604441423e+13 network.recv‚ 4.243123e+06 disk.dm-1.in‚ 106.21484375 disk.dm-0.in‚39856.87890625 disk.sda1.out‚0 disk.rbd3.out‚ 0.07421875% disk.nvme0n1.out‚137648.267578125$ disk./.usageGB‚564.2641105651855 disk.dm-1.out‚ 1206.7109375 disk.sda3.in‚ 106.21484375­å^ì’„*‚(:‡ §¹éÒîЦ÷ gpu.0.gpu‚90.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚256.11' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚51.222 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ó(ùV…*‚(:Ë §¹éÒü“ù$ disk.dm-0.out‚138157.89111328125" disk.nvme0n1.in‚39857.30078125" proc.memory.rssMB‚ 2802.8984375 disk.rbd1.in‚ 0.51220703125 disk.rbd1.out‚ 44.42578125 network.sent‚ 1.9742731e+07 network.recv‚ 4.246533e+06 disk.dm-1.in‚ 106.21484375 disk.sda1.out‚0 disk.sda3.out‚ 1207.04296875 disk.drbd2165.in‚0$ disk.nvme0c0n1.in‚39857.30078125 disk.sda2.in‚0 proc.cpu.threads‚17 disk.dm-1.out‚ 1207.04296875 disk.rbd0.out‚0 disk.rbd3.in‚0 disk.sda.in‚ 106.21484375 disk.sda3.in‚ 106.21484375* proc.memory.percent‚17.107534408569336 cpu‚25.584119875190165 disk.sda.out‚ 1207.04296875& disk.rbd0.in‚1.759218604441423e+13 disk.drbd2165.out‚0) disk.nvme0c0n1.out‚138157.89111328125$ memory_percent‚82.81807899475098 disk.sda2.out‚0* disk./.usagePercent‚37.746153804247385 disk.sda1.in‚0 disk.rbd3.out‚ 0.09814453125' disk.nvme0n1.out‚138157.89111328125( proc.memory.availableMB‚ 2815.0859375$ disk./.usageGB‚562.4449691772461 disk.dm-0.in‚39857.300781253“ƒŸ’†*‚(:‡ ¶¹éÒØð±÷ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚280.04' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚56.008 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0-Y(NT‡*‚(:É ¶¹éÒçÖÑú* proc.memory.percent‚17.107534408569336 disk.rbd3.in‚0) disk.nvme0c0n1.out‚138165.65380859375 disk.sda.out‚ 1207.34765625 disk.sda3.out‚ 1207.34765625' disk.nvme0n1.out‚138165.65380859375 disk.sda3.in‚ 106.21484375# disk./.usageGB‚562.451602935791 disk.rbd1.in‚ 0.51220703125 disk.rbd1.out‚ 44.53515625" proc.memory.rssMB‚ 2802.8984375 cpu‚25.584056012075603 network.recv‚ 4.24966e+06 disk.sda2.in‚0 disk.rbd0.out‚0 disk.dm-0.in‚39857.51953125 disk.drbd2165.out‚0) proc.memory.availableMB‚ 2815.33984375 disk.rbd3.out‚ 0.09814453125 disk.sda2.out‚0 disk.sda.in‚ 106.21484375$ disk.nvme0c0n1.in‚39857.51953125 disk.dm-1.out‚ 1207.34765625 disk.sda1.in‚0 disk.drbd2165.in‚0$ memory_percent‚82.81652927398682 network.sent‚ 1.9748353e+07 disk.dm-1.in‚ 106.21484375$ disk.dm-0.out‚138165.65380859375 proc.cpu.threads‚17) disk./.usagePercent‚37.74659900134942" disk.nvme0n1.in‚39857.51953125& disk.rbd0.in‚1.759218604441423e+13 disk.sda1.out‚0GzŒžˆ*‚(:“ ŹéÒ®Œžø gpu.0.gpu‚63.0 gpu.0.memory‚20, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚339.941' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚67.98819999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0˜y9ÍJ‰*‚(:¿ ŹéÒÙéøø network.recv‚ 4.253332e+06) disk./.usagePercent‚37.74708208756653 disk.rbd3.in‚0 disk.sda.in‚ 106.21484375& disk.rbd0.in‚1.759218604441423e+13 disk.rbd0.out‚0 cpu‚25.58406066511507( disk.nvme0c0n1.out‚138181.9423828125 disk.sda3.out‚ 1207.671875* proc.memory.percent‚17.107534408569336 proc.cpu.threads‚17 disk.drbd2165.in‚0 network.sent‚ 1.975296e+07 disk.rbd3.out‚ 0.1611328125" disk.nvme0n1.in‚40007.40234375$ disk.nvme0c0n1.in‚40007.40234375 disk.dm-1.out‚ 1207.671875$ memory_percent‚82.81805515289307& disk.nvme0n1.out‚138181.9423828125 disk.sda2.out‚0 disk.rbd1.out‚ 44.53515625 disk.sda3.in‚ 106.21484375" proc.memory.rssMB‚ 2802.8984375$ disk./.usageGB‚562.4588012695312 disk.rbd1.in‚ 0.51220703125 disk.dm-1.in‚ 106.21484375 disk.sda1.in‚0 disk.drbd2165.out‚0) proc.memory.availableMB‚ 2815.08984375 disk.sda2.in‚0 disk.sda.out‚ 1207.671875 disk.dm-0.in‚40007.40234375# disk.dm-0.out‚138181.9423828125 disk.sda1.out‚0‰‚4Š*/ _wandbruntime‚17077  _runtime‚17077wÓ³YÊ‹*Ä $ _timestamp‚1.784306893361785e+09 _runtime‚17077.581060953 _step‚845 ' lr-AdamW/pg1‚0.00034043524468825646 ' lr-AdamW/pg2‚0.00034043524468825646 trainer/global_step‚10299 K+ÏŒ*É $ _timestamp‚1.784306893361785e+09 _runtime‚17077.581060953 _step‚845 ' lr-AdamW/pg1‚0.00034043524468825646 ' lr-AdamW/pg2‚0.00034043524468825646 trainer/global_step‚10299ͳ¶™4*/ _wandbruntime‚17077  _runtime‚17077ëË(Ž*€P SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _step‚846 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚812003935903232 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚812003533275648 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚799504924725248 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2968880974245888 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2957916552359936 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9686442151936 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚256 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚89185775 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 177002117 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2959194532093952 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚26420049 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚87826321 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9686442151936 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚29189281 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚89185775 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚87826315 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9671212966047 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 177012096 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚799504556634112 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚26420054 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12499011177984 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2967587765325983 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 % _timestamp‚1.7843068933685114e+09 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2959194558267392 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚55609330 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚261 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2959194532093952 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9669868365725 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚87826315 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2968882345808896 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12498976641536 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2957916526931968 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2968880974245888 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚812003533275648 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚799504924725248 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2968880974245888 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2957916552359936 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9686442151936 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚256 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚89185775 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 177002117 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2959194532093952 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚26420049 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚87826321 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9686442151936 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚29189281 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚89185775 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚87826315 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9671212966047 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 177012096 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚799504556634112 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚26420054 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12499011177984 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2967587765325983 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 % _timestamp‚1.7843068933685114e+09 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2959194558267392 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚55609330 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚261 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚2959194532093952 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9669868365725 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚87826315 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2968882345808896 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12498976641536 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2957916526931968 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚2968880974245888 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2958194408710144 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚87834521 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 trainer/global_step‚10299 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚812082616870400 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9688712584192 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9688712584192 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 177028794 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚55614856 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚89194267 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚26422592 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚246 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12500171208192 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 _runtime‚17078.971853497 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 _step‚847 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚55614605 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚26422597 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 Tú'’*N XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2958194434138112 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2967865200542115 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚89184280 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚251 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚29192259 % _timestamp‚1.7843068947544513e+09 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2969159900011520 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚26422592 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚246 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12500171208192 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 _runtime‚17078.971853497 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 _step‚847 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚55614605 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚26422597 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 177018801 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2967866570570517 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚89184280 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9687367187456 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚87834527 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2958194408710144 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚87834521 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 trainer/global_step‚10299 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚812082616870400 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9688712584192 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9688712584192 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 177028794 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚55614856 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚89194267 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 _step‚850 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚815937766657536 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2982828698245120 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2982827326682112 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚89601554 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12558889639424 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2971810641268736 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 177848148 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚803379245109248 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2981528091347891 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9732758082560 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚815938169285120 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚803378877018112 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 % _timestamp‚1.7843069728989265e+09 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚26547199 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2973094594772992 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2981526721319601 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚88236621 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚88236615 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 177838169 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚89601554 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9717450079155 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9716105478833 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2973094594772992 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚10349 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973´M_ (¥*†P ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚815938169285120 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚803378877018112 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 % _timestamp‚1.7843069728989265e+09 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚26547199 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2973094594772992 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2981526721319601 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚88236621 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚88236615 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 177838169 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚89601554 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9717450079155 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9716105478833 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2973094594772992 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚10349 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2973094568599552 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 _step‚850 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚815937766657536 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2982828698245120 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2982827326682112 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚89601554 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12558889639424 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2971810641268736 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 177848148 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚803379245109248 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚2981528091347891 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9732758082560 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440Ò2éSŠ4¦*/ _wandbruntime‚17158  _runtime‚17158ð髧*‹N cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚803456731516928 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚248 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2983106252447744 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚89620019 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2981806896592425 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9718373545513 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2973372569329664 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚89610032 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚803456363425792 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12560083681792 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚26549742 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2973372595503104 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 177854853 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9717028945079 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚10349 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 177864846 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚88244827 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚55880506 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚816016849728000 I@DeviceStatsMïJX÷onitor.on_train_batch_end/segment.large_pool.current‚1175 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9733683118080 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚26549747 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚29330764 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 _runtime‚17158.506022039 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 177854853 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚29331007 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚88244827 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2983107624017920 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2973372595503104 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 177864846 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9733683118080 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚55880754 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2972088497618944 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9735028514816 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0ãÊ£'¨*N HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚803456363425792 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12560083681792 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚26549742 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚2973372595503104 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 177854853 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9717028945079 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚10349 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 177864846 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚88244827 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚55880506 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚816016849728000 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9733683118080 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚26549747 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚29330764 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 _runtime‚17158.506022039 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 177854853 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚29331007 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚88244827 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2983107624017920 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2973372595503104 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 177864846 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9733683118080 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚55880754 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚2972088497618944 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9735028514816 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚803456731516928 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚248 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2983106252447744 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚89620019 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2981806896592425 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9718373545513 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2973372569329664 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚89610032ÓÕº:v4©*/ _wandbruntime‚17158  _runtime‚17158R@ª*Œ /loss/FBt2/detection_loss‚0.5358512761539996 3loss/FBt1/assignment_loss‚0.0002409346513152716 /loss/FRt1/detection_loss‚0.4570376479622295 /loss/FRt2/detection_loss‚1.3654990934782887 /loss/FBt1/detection_loss‚0.6052111602138456 _runtime‚17158.589864583 _step‚852 1loss/FRt2/assignment_loss‚0.45193300120936236 2loss/SRqqt1/assignment_loss‚0.2468840066636794 4loss/FBt2/assignment_loss‚0.00010971206451452334 &loss/total_loss‚6.4718223686097005 epoch‚25 0loss/FRt1/assignment_loss‚1.2206155807741226 1loss/SRqqt2/detection_loss‚0.5337439250794935 trainer/global_step‚10349 % _timestamp‚1.7843069743728836e+09 3loss/SRqqt2/assignment_loss‚0.38738516311418075 1loss/SRqqt1/detection_loss‚0.6673108672446688}"ÐÍ«*‘ 2loss/SRqqt1/assignment_loss‚0.2468840066636794 4loss/FBt2/assignment_loss‚0.00010971206451452334 &loss/total_loss‚6.4718223686097005 epoch‚25 0loss/FRt1/assignment_loss‚1.2206155807741226 1loss/SRqqt2/detection_loss‚0.5337439250794935 trainer/global_step‚10349 % _timestamp‚1.7843069743728836e+09 3loss/SRqqt2/assignment_loss‚0.38738516311418075 1loss/SRqqt1/detection_loss‚0.6673108672446688 /loss/FBt2/detection_loss‚0.5358512761539996 3loss/FBt1/assignment_loss‚0.0002409346513152716 /loss/FRt1/detection_loss‚0.4570376479622295 /loss/FRt2/detection_loss‚1.3654990934782887 /loss/FBt1/detection_loss‚0.6052111602138456 _runtime‚17158.589864583 _step‚852 1loss/FRt2/assignment_loss‚0.45193300120936236Ô*çÈ•”¬*‚(:‰ ŸºéÒÝŽìö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚287.327' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚57.4654 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ìW‡ýR­*‚(:Ç ŸºéÒü¸êù network.sent‚ 1.9842758e+07 disk.sda3.in‚ 106.21484375 disk.drbd2165.in‚0 disk.rbd3.in‚0 disk.rbd1.out‚ 44.80078125 disk.drbd2165.out‚0 disk.sda.out‚ 1212.58984375# disk.dm-0.out‚138510.3994140625( proc.memory.availableMB‚ 2815.0546875$ disk./.usageGB‚563.2819023132324 disk.sda2.out‚0 disk.rbd0.out‚0 network.recv‚ 4.275567e+06 disk.sda1.out‚0 disk.sda3.out‚ 1212.58984375" disk.nvme0n1.in‚40309.68359375 cpu‚25.58390859823581 disk.dm-1.in‚ 106.21484375& disk.rbd0.in‚1.759218604441423e+13 disk.dm-0.in‚40309.68359375 disk.rbd3.out‚ 0.22412109375& disk.nvme0n1.out‚138510.3994140625 disk.sda2.in‚0 disk.rbd1.in‚ 0.51220703125* proc.memory.percent‚17.107534408569336$ disk.nvme0c0n1.in‚40309.68359375( disk.nvme0c0n1.out‚138510.3994140625" proc.memory.rssMB‚ 2802.8984375 proc.cpu.threads‚17* disk./.usagePercent‚37.802321089237104 disk.sda1.in‚0 disk.sda.in‚ 106.21484375$ memory_percent‚82.81826972961426 disk.dm-1.out‚ 1212.58984375|‰@mž®*‚(:“ ®ºéÒäЀø gpu.0.gpu‚82.0 gpu.0.memory‚25, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚46.0 gpu.0.powerWatts‚337.288' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚67.45760000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0x¦F¯*‚(:» ®ºéÒËÆÛù& disk.rbd0.in‚1.759218604441423e+13 disk.drbd2165.out‚0! disk.nvme0c0n1.in‚ 40310.09375 disk.nvme0n1.in‚ 40310.09375* proc.memory.percent‚17.107534408569336$ disk./.usageGB‚562.5740737915039 disk.drbd2165.in‚0" proc.memory.rssMB‚ 2802.8984375 network.recv‚ 4.28019e+06 disk.sda.in‚ 106.21484375 disk.sda.out‚ 1212.75390625$ memory_percent‚82.81784057617188 disk.rbd1.in‚ 0.51220703125 disk.rbd1.out‚ 44.87890625 disk.rbd0.out‚0 disk.sda3.out‚ 1212.75390625) disk./.usagePercent‚37.75481812323621 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd3.out‚ 0.22412109375 disk.sda2.in‚0$ proc.memory.availableMB‚2815.125' disk.nvme0n1.out‚139085.31298828125 disk.dm-0.in‚ 40310.09375 disk.dm-1.in‚ 106.21484375 disk.dm-1.out‚ 1212.75390625 disk.sda3.in‚ 106.21484375) disk.nvme0c0n1.out‚139085.31298828125 proc.cpu.threads‚17 network.sent‚ 1.9905384e+07$ disk.dm-0.out‚139085.31298828125 disk.rbd3.in‚0 disk.sda2.out‚0 cpu‚25.58390847663693ó{~.ž°*‚(:“ ½ºéÒ÷¨Ûö gpu.0.gpu‚90.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚297.575' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚59.51499999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0+$«œL±*‚(:Á ½ºéÒŠ‡Þø& disk.rbd0.in‚1.759218604441423e+13 disk.rbd0.out‚0% disk.nvme0n1.out‚139093.458984375 disk.rbd1.out‚ 44.87890625 disk.dm-1.in‚ 106.21484375 disk.dm-0.in‚ 40637.8046875" disk.dm-0.out‚139093.458984375 disk.rbd3.out‚ 0.22412109375# disk.nvme0c0n1.in‚ 40637.8046875 proc.cpu.threads‚17 network.sent‚ 1.9911113e+07 disk.sda.in‚ 106.21484375 disk.dm-1.out‚ 1213.13671875 disk.sda3.out‚ 1213.13671875 disk.rbd1.in‚ 0.51220703125$ disk./.usageGB‚562.5797424316406 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.rbd3.in‚0) proc.memory.availableMB‚ 2814.76953125 network.recv‚ 4.283875e+06 disk.sda1.in‚0' disk.nvme0c0n1.out‚139093.458984375 disk.sda2.out‚0" proc.memory.rssMB‚ 2802.8984375) disk./.usagePercent‚37.75519855043209 disk.sda3.in‚ 106.21484375 disk.drbd2165.out‚0 disk.sda.out‚ 1213.13671875! disk.nvme0n1.in‚ 40637.8046875 disk.sda2.in‚0# memory_percent‚82.8200101852417 cpu‚25.583852911323294* proc.memory.percent‚17.107534408569336Ló8Ÿ²*‚(:” ̺éÒ îåö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚302.936' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚60.587199999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¨w­bK³*‚(:À ̺éÒŠ˜ù% disk.nvme0n1.out‚139101.619140625 cpu‚25.583851031416994& disk.rbd0.in‚1.759218604441423e+13" disk.dm-0.out‚139101.619140625 disk.sda1.in‚0 disk.drbd2165.in‚0 disk.sda2.in‚0( proc.memory.availableMB‚ 2814.8828125 network.sent‚ 1.9915328e+07 disk.sda.in‚ 106.21484375 disk.rbd3.in‚0 disk.rbd3.out‚ 0.22412109375" proc.memory.rssMB‚ 2802.8984375 disk.sda.out‚ 1213.75390625 disk.sda1.out‚0# disk.nvme0c0n1.in‚ 40637.8046875) disk./.usagePercent‚37.75353603750474 disk.rbd0.out‚0 disk.dm-0.in‚ 40637.8046875 disk.sda3.in‚ 106.21484375! disk.nvme0n1.in‚ 40637.8046875* proc.memory.percent‚17.107534408569336 disk.rbd1.out‚ 44.94140625 network.recv‚ 4.286532e+06$ disk./.usageGB‚562.5549697875977 disk.drbd2165.out‚0# memory_percent‚82.8193187713623 disk.dm-1.in‚ 106.21484375 disk.dm-1.out‚ 1213.75390625' disk.nvme0c0n1.out‚139101.619140625 disk.rbd1.in‚ 0.51220703125 proc.cpu.threads‚17 disk.sda3.out‚ 1213.75390625 disk.sda2.out‚0‚ùŸ´*‚(:” ÛºéÒ”Ùçö gpu.0.gpu‚90.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚292.448' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚58.489599999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0’o°±Oµ*‚(:Ä ÛºéÒû«•ù disk.drbd2165.in‚0* proc.memory.percent‚17.107534408569336 disk.rbd0.out‚0 disk.sda3.in‚ 106.21484375 disk.rbd3.in‚0" proc.memory.rssMB‚ 2802.8984375 network.sent‚ 1.9920466e+07& disk.rbd0.in‚1.759218604441423e+13 disk.rbd1.out‚ 44.94140625 disk.dm-1.out‚ 1214.3359375 disk.sda3.out‚ 1214.3359375 cpu‚25.58376366222313 disk.sda.out‚ 1214.3359375 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd2165.out‚0 disk.rbd3.out‚ 0.22412109375$ memory_percent‚82.81924724578857 network.recv‚ 4.290138e+06) disk./.usagePercent‚37.75403576421157 disk.dm-1.in‚ 106.21484375' disk.nvme0n1.out‚139110.60498046875 proc.cpu.threads‚17$ disk.dm-0.out‚139110.60498046875# disk.nvme0c0n1.in‚ 40638.2265625) disk.nvme0c0n1.out‚139110.60498046875! disk.nvme0n1.in‚ 40638.2265625 disk.rbd1.in‚ 0.51220703125) proc.memory.availableMB‚ 2814.89453125 disk.sda2.in‚0 disk.sda2.out‚0$ disk./.usageGB‚562.5624160766602 disk.sda.in‚ 106.21484375 disk.dm-0.in‚ 40638.2265625u*‚(:” êºéÒ“¿­÷ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚267.174' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚53.434799999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÃÔTM·*‚(: êºéÒ¶¥±ù disk.rbd1.out‚ 45.01953125 disk.rbd0.out‚0 disk.drbd2165.in‚0" proc.memory.rssMB‚ 2802.8984375 cpu‚25.58367568672444 disk.sda2.out‚0 network.recv‚ 4.293247e+06# disk./.usageGB‚562.566520690918 disk.rbd1.in‚ 0.51220703125 disk.drbd2165.out‚0 disk.rbd3.out‚ 0.22412109375 proc.cpu.threads‚17 disk.sda1.out‚0# disk.nvme0c0n1.in‚ 40638.4296875& disk.nvme0n1.out‚139124.2509765625* proc.memory.percent‚17.107534408569336 network.sent‚ 1.9925563e+07 disk.dm-0.in‚ 40638.4296875 disk.sda1.in‚0 disk.sda3.in‚ 106.21484375 disk.sda3.out‚ 1214.48828125 disk.sda2.in‚0 disk.rbd3.in‚0( disk.nvme0c0n1.out‚139124.2509765625! disk.nvme0n1.in‚ 40638.4296875* disk./.usagePercent‚37.754311228318414 disk.sda.in‚ 106.21484375 disk.sda.out‚ 1214.48828125 disk.dm-1.in‚ 106.21484375& disk.rbd0.in‚1.759218604441423e+13# disk.dm-0.out‚139124.2509765625( proc.memory.availableMB‚ 2814.8828125 disk.dm-1.out‚ 1214.48828125# memory_percent‚82.8193187713623hÍ4¸*/ _wandbruntime‚17236  _runtime‚17236˜¤0¥É¹*à & lr-AdamW/pg1‚0.0003348922270175304 & lr-AdamW/pg2‚0.0003348922270175304 trainer/global_step‚10399 % _timestamp‚1.7843070527057328e+09 _runtime‚17236.922937279 _step‚853§–Këκ*È & lr-AdamW/pg1‚0.0003348922270175304 & lr-AdamW/pg2‚0.0003348922270175304 trainer/global_step‚10399 % _timestamp‚1.7843070527057328e+09 _runtime‚17236.922937279 _step‚853Õœ½H4»*/ _wandbruntime‚17236  _runtime‚17236“Õÿ …¼*P GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚90037279 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2985704704749568 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9779074013184 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2995467047341509 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚17236.924129941 _step‚854 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9779074013184 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 178684200 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12618702515712 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 trainer/global_step‚10399 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2996775050681344 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2985704730177536 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚88646921 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚88646915 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2986994631278592 % _timestamp‚1.7843070527068043e+09 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 178674221 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚90037279 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚26674349 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12618739149312 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9762342591941 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2986994605105152 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚29466176 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404724736 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚56140258 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚90027306 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚29465909 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2986994631278592 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9780419402752 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9763687192263 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚88646915 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚807253197402112 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2996773679118336 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚56140530 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚26674354 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚88646921 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2996775050681344 …Ó (½*†P ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚819872304642560 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36633600 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚2996773679118336 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚56140530 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚26674354 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚88646921 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚2996775050681344 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚90037279 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚2985704704749568 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9779074013184 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚2995467047341509 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚17236.924129941 _step‚854 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9779074013184 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 178684200 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12618702515712 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 trainer/global_step‚10399 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚2996775050681344 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚2985704730177536 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚88646921 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚88646915 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚2986994631278592 % _timestamp‚1.7843070527068043e+09 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 178674221 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚90037279 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚26674349 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12618739149312 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9762342591941 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚2986994605105152 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚29466176 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404724736 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚56140258 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚90027306 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚29465909 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚2986994631278592 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9780419402752 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9763687192263 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚88646915 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚807253197402112 JDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 178700898 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2995747222614333 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2997052604883968 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2987272605835264 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 trainer/global_step‚10399 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚88655121 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2997053976454144 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚56145527 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2995745852585931 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚254 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 % _timestamp‚1.7843070540935578e+09 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚807330683809792 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36626432 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2985982611955712 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9779999048704 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚88655121 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚56145786 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚259 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚29468635 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9781344445440 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404717568 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 _runtime‚17238.311052114 _step‚855 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2987272605835264 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 178690905 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2997053976454144 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9781344445440 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚88655127 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9779999048704 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚88655127 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9764610658621 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2987272632008704 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚807331051900928 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12619923689984 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0¡a‡ $À*N _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚2987272632008704 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚807331051900928 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12619923689984 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12619887063552 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 178700898 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚2995747222614333 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚2997052604883968 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚2987272605835264 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 trainer/global_step‚10399 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚88655121 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚2997053976454144 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚56145527 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚2995745852585931 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚254 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 % _timestamp‚1.7843070540935578e+09 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚807330683809792 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36626432 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚2985982611955712 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9779999048704 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚88655121 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚56145786 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚259 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚29468635 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9781344445440 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404717568 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 _runtime‚17238.311052114 _step‚855 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚2987272605835264 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 178690905 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚2997053976454144 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9781344445440 REDeviceStatsMonitoUyK¡ r.on_train_batch_end/allocation.large_pool.allocated‚88655127 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9779999048704 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚88655127 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9764610658621 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6×pGH½4Á*/ _wandbruntime‚17238  _runtime‚17238þ62Â*‰ 3loss/SRqqt2/assignment_loss‚0.35676640895619827 1loss/SRqqt2/detection_loss‚0.5567379196115392 0loss/FBt2/detection_loss‚0.24978211216488475 trainer/global_step‚10399 $ _timestamp‚1.784307054179148e+09 _step‚856 0loss/FRt1/assignment_loss‚1.3105639295445548 3loss/FBt1/assignment_loss‚7.086959756183873e-05 /loss/FBt1/detection_loss‚0.5471395711188741 3loss/SRqqt1/assignment_loss‚0.35132651680866206 4loss/FBt2/assignment_loss‚0.00011100864292368589 1loss/SRqqt1/detection_loss‚0.6988017420627992 epoch‚25 _runtime‚17238.396047609 0loss/FRt2/assignment_loss‚0.3768380371872912 /loss/FRt1/detection_loss‚0.4689101205629817 /loss/FRt2/detection_loss‚1.3699655900373287 #loss/total_loss‚6.2870138262956ó*öÃ*Ž 3loss/SRqqt1/assignment_loss‚0.35132651680866206 4loss/FBt2/assignment_loss‚0.00011100864292368589 1loss/SRqqt1/detection_loss‚0.6988017420627992 epoch‚25 _runtime‚17238.396047609 0loss/FRt2/assignment_loss‚0.3768380371872912 /loss/FRt1/detection_loss‚0.4689101205629817 /loss/FRt2/detection_loss‚1.3699655900373287 #loss/total_loss‚6.2870138262956 3loss/SRqqt2/assignment_loss‚0.35676640895619827 1loss/SRqqt2/detection_loss‚0.5567379196115392 0loss/FBt2/detection_loss‚0.24978211216488475 trainer/global_step‚10399 $ _timestamp‚1.784307054179148e+09 _step‚856 0loss/FRt1/assignment_loss‚1.3105639295445548 3loss/FBt1/assignment_loss‚7.086959756183873e-05 /loss/FBt1/detection_loss‚0.5471395711188741ØÍŠ@,”Ä*‚(:‰ ùºéÒŽÎÕö gpu.0.gpu‚97.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚340.764' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚68.1528 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0y•FVFÅ*‚(:» ùºéÒÜÏÄù$ disk./.usageGB‚562.5691299438477 disk.sda3.out‚ 1214.65625* proc.memory.percent‚17.107534408569336 cpu‚25.58484730472201 proc.cpu.threads‚17) disk./.usagePercent‚37.75448633747184" disk.dm-0.out‚139136.580078125 disk.drbd2165.in‚0 disk.dm-0.in‚40745.11328125% disk.nvme0n1.out‚139136.580078125 disk.sda.in‚ 106.21484375 disk.dm-1.in‚ 106.21484375 disk.dm-1.out‚ 1214.65625 disk.rbd1.in‚ 0.51220703125 disk.sda1.out‚0 disk.rbd3.in‚0 disk.rbd3.out‚ 0.22412109375' disk.nvme0c0n1.out‚139136.580078125 disk.sda2.out‚0 disk.sda.out‚ 1214.65625& disk.rbd0.in‚1.759218604441423e+13 disk.rbd0.out‚0 disk.sda3.in‚ 106.21484375$ disk.nvme0c0n1.in‚40745.11328125) proc.memory.availableMB‚ 2814.66796875 network.sent‚ 1.9930705e+07 disk.sda1.in‚0" disk.nvme0n1.in‚40745.11328125 disk.rbd1.out‚ 45.01953125" proc.memory.rssMB‚ 2802.8984375 disk.drbd2165.out‚0 disk.sda2.in‚0$ memory_percent‚82.82063007354736 network.recv‚ 4.296812e+06]n•Æ*‚(:Š ˆ»éÒ×Âöö gpu.0.gpu‚100.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚40.0 gpu.0.powerWatts‚177.792' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.5584 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0û8OÇ*‚(:Ä ˆ»éÒâñòù$ disk.nvme0c0n1.in‚40754.33203125) disk./.usagePercent‚37.75500449672114$ disk.dm-0.out‚139144.51416015625 disk.sda2.out‚0) proc.memory.availableMB‚ 2763.54296875 cpu‚25.586551201409215 proc.cpu.threads‚17 network.sent‚ 1.9993859e+07 disk.sda.out‚ 1214.9921875 disk.dm-1.out‚ 1214.9921875" disk.nvme0n1.in‚40754.33203125 disk.sda.in‚ 106.21484375 disk.rbd3.out‚ 0.22412109375 disk.drbd2165.out‚0& disk.rbd0.in‚1.759218604441423e+13 disk.dm-0.in‚40754.33203125 disk.sda3.in‚ 106.21484375 disk.drbd2165.in‚0) disk.nvme0c0n1.out‚139144.51416015625 disk.rbd1.in‚ 0.51220703125$ memory_percent‚83.13267230987549) proc.memory.percent‚17.42105484008789 disk.sda1.out‚0' disk.nvme0n1.out‚139144.51416015625$ disk./.usageGB‚562.5768508911133 disk.sda1.in‚0 disk.sda3.out‚ 1214.9921875 disk.rbd3.in‚0 disk.sda2.in‚0 disk.rbd1.out‚ 45.08203125! proc.memory.rssMB‚ 2854.265625 network.recv‚ 4.3016e+06 disk.dm-1.in‚ 106.21484375 disk.rbd0.out‚0 ½’4È*/ _wandbruntime‚17266  _runtime‚172667÷‰,¼=É*¶{ 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9988215565681458 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9147982597351074 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7512282729148865 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.2056104838848114 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.19525115191936493 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.013518563471734524 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9992673993110657 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6741088032722473 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9983081817626953 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6826176643371582 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.21845391392707825 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8262849450111389 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 .particle/accuracy_4_of_4‚0.941375732421875 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9951749444007874 particle/accuracy_4_of_5‚1 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7361111044883728 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8823962211608887 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9990078806877136 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.43460118770599365 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.20284734666347504 *jet/accuracy_1_of_2‚0.7500908374786377 particle/accuracy_1_of_4‚1 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.26183173060417175 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.02153250388801098 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 particle/accuracy_5_of_5‚0 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.5909048914909363 , Purity/0FRt2SRqqt1FBt/FBt_purity‚0.96875 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 trainer/global_step‚10399 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.1441306620836258 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6672642230987549 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5049999952316284 particle/accuracy_2_of_4‚1 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.29883548617362976 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7361111044883728 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6451838612556458 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.05000000074505806 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 jet/accuracy_3_of_5‚1 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.05000000074505806 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7251063585281372 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7391070127487183 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3499999940395355 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.11538461595773697 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8439531922340393 6 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.194583922624588 particle/accuracy_1_of_2‚1 ("Purity/2FRt0SRqqt2FBt/event_purity‚1 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9724019169807434 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.21634097397327423 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7338294982910156 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7251497507095337 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9862009882926941 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6306288242340088 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9825584888458252 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9841357469558716 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 7 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.0413781963288784 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.025754496455192566 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.9375 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.44733062386512756 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.10666666179895401 particle/accuracy_2_of_3‚1 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.04511726647615433 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.44071656465530396 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7334538102149963 *jet/accuracy_1_of_4‚0.9661109447479248 particle/accuracy_2_of_5‚1 particle/accuracy_3_of_5‚1 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 7validation_average_jet_accuracy‚0.35012245178222656 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 particle/accuracy_2_of_2‚1 +particle/specificity‚0.9073724150657654 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.875 jet/accuracy_4_of_5‚0 _runtime‚17266.339714888 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6445277333259583 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8703200221061707 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6259775757789612 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5505216121673584 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.15634630620479584 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.7954918742179871 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 8"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.645481526851654 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6089051365852356 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.21782563626766205 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5400312542915344 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3541826903820038 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.22333453595638275 +jet/accuracy_4_of_4‚0.01908157207071781 +jet/accuracy_1_of_1‚0.30719760060310364 *jet/accuracy_1_of_3‚0.9137127995491028 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6080199480056763 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03275872766971588 6 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.159016415476799 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8551613092422485 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 6 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.942307710647583 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.5620495676994324 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7538265585899353 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6737676858901978 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.7954918742179871 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9994588494300842 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 *jet/accuracy_2_of_4‚0.7255133390426636 'particle/f_score‚0.6301673650741577 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.40509259700775146 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.20552591979503632 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.19524028897285461 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4496309757232666 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6375929713249207 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 +particle/sensitivity‚0.6158620119094849 epoch‚25 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9957830309867859 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 +jet/accuracy_3_of_3‚0.06747541576623917 +jet/accuracy_3_of_4‚0.08862504363059998 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.19468212127685547 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.19895058870315552 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.04865577071905136 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.5911458730697632 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.45122119784355164 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6097974181175232 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.17878596484661102 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7065620422363281 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 +jet/accuracy_2_of_2‚0.14678235352039337 particle/accuracy_3_of_3‚1 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.11538461595773697 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9314160943031311 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 particle/accuracy_3_of_4‚1 % _timestamp‚1.7843070821219437e+09 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6274847388267517 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.27996453642845154 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5962303280830383 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5215704441070557 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 jet/accuracy_2_of_5‚1 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2167550027370453 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6322098970413208 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.14905674755573273 particle/accuracy_1_of_5‚1 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9929356575012207 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 _step‚857 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5962303280830383 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.2056104838848114 particle/accuracy_1_of_1‚1 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.04098972678184509 jet/accuracy_5_of_5‚0 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7126690745353699 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9814814329147339 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.04480673372745514 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 )loss/val_total_loss‚8.226883576974434 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8453136682510376 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6731105446815491 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.2683126926422119 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9814814329147339 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9992673993110657 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.3984375 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.18232743442058563 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 jet/accuracy_1_of_5‚1 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9878116846084595 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 /"Purity/2FRt0SRqqt1FBt/event_purity‚0.234375 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9981684684753418 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9940118193626404 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.19784609973430634 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6646218299865723 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 (particle/accuracy‚0.8447692394256592 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 & Purity/2FRt0SRqqt2FBt/FRt_purity‚1 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.41530176997184753 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8204458951950073 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7319350242614746 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9942954182624817 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.35171598196029663 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.4004629850387573 8"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.658886194229126 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8204458951950073 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6821813583374023 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.043264348059892654 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.041376277804374695 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6180062294006348 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5803964734077454 8"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.597186803817749 , Purity/0FRt2SRqqt*FBt/FBt_purity‚0.96875 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.1343165785074234 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2004156857728958 particle/accuracy_1_of_3‚1 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9940482378005981 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.16617915034294128 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.18194998800754547 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9950305223464966 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6666476726531982 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.04865577071905136 *jet/accuracy_2_of_3‚0.5236248970031738"ò3¢,Ê*»{ =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3541826903820038 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.22333453595638275 +jet/accuracy_4_of_4‚0.01908157207071781 +jet/accuracy_1_of_1‚0.30719760060310364 *jet/accuracy_1_of_3‚0.9137127995491028 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6080199480056763 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03275872766971588 6 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.159016415476799 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8551613092422485 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 6 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.942307710647583 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.5620495676994324 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7538265585899353 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6737676858901978 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.7954918742179871 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9994588494300842 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 *jet/accuracy_2_of_4‚0.7255133390426636 'particle/f_score‚0.6301673650741577 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.40509259700775146 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.20552591979503632 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.19524028897285461 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4496309757232666 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6375929713249207 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 +particle/sensitivity‚0.6158620119094849 epoch‚25 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9957830309867859 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 +jet/accuracy_3_of_3‚0.06747541576623917 +jet/accuracy_3_of_4‚0.08862504363059998 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.19468212127685547 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.19895058870315552 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.04865577071905136 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.5911458730697632 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.45122119784355164 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6097974181175232 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.17878596484661102 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7065620422363281 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 +jet/accuracy_2_of_2‚0.14678235352039337 particle/accuracy_3_of_3‚1 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.11538461595773697 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9314160943031311 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 particle/accuracy_3_of_4‚1 % _timestamp‚1.7843070821219437e+09 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6274847388267517 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.27996453642845154 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5962303280830383 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5215704441070557 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 jet/accuracy_2_of_5‚1 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2167550027370453 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6322098970413208 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.14905674755573273 particle/accuracy_1_of_5‚1 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9929356575012207 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 _step‚857 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5962303280830383 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.2056104838848114 particle/accuracy_1_of_1‚1 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.04098972678184509 jet/accuracy_5_of_5‚0 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7126690745353699 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9814814329147339 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.04480673372745514 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9978835582733154 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 )loss/val_total_loss‚8.226883576974434 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8453136682510376 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6731105446815491 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.2683126926422119 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9814814329147339 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9992673993110657 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.3984375 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.18232743442058563 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 jet/accuracy_1_of_5‚1 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9878116846084595 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 /"Purity/2FRt0SRqqt1FBt/event_purity‚0.234375 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9981684684753418 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9940118193626404 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.19784609973430634 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6646218299865723 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 (particle/accuracy‚0.8447692394256592 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 & Purity/2FRt0SRqqt2FBt/FRt_purity‚1 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.41530176997184753 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8204458951950073 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7319350242614746 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9942954182624817 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.35171598196029663 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.4004629850387573 8"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.658886194229126 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8204458951950073 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6821813583374023 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.043264348059892654 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.041376277804374695 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6180062294006348 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5803964734077454 8"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.597186803817749 , Purity/0FRt2SRqqt*FBt/FBt_purity‚0.96875 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.1343165785074234 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.2004156857728958 particle/accuracy_1_of_3‚1 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9940482378005981 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.16617915034294128 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.18194998800754547 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9950305223464966 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6666476726531982 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.04865577071905136 *jet/accuracy_2_of_3‚0.5236248970031738 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9988215565681458 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9147982597351074 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7512282729148865 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.2056104838848114 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.19525115191936493 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.013518563471734524 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9992673993110657 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6741088032722473 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9983081817626953 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6826176643371582 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.21845391392707825 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8262849450111389 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 .particle/accuracy_4_of_4‚0.941375732421875 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9951749444007874 particle/accuracy_4_of_5‚1 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7361111044883728 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8823962211608887 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9990078806877136 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.43460118770599365 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.20284734666347504 *jet/accuracy_1_of_2‚0.7500908374786377 particle/accuracy_1_of_4‚1 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.26183173060417175 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9965986609458923 :"Purity/2FRt*ùlE@SRqqt*FBt/event_purity‚0.02153250388801098 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 particle/accuracy_5_of_5‚0 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.5909048914909363 , Purity/0FRt2SRqqt1FBt/FBt_purity‚0.96875 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 trainer/global_step‚10399 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.1441306620836258 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6672642230987549 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5049999952316284 particle/accuracy_2_of_4‚1 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.29883548617362976 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7361111044883728 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6451838612556458 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.05000000074505806 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 jet/accuracy_3_of_5‚1 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.05000000074505806 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7251063585281372 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7391070127487183 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3499999940395355 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.11538461595773697 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8439531922340393 6 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.194583922624588 particle/accuracy_1_of_2‚1 ("Purity/2FRt0SRqqt2FBt/event_purity‚1 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9724019169807434 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.21634097397327423 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7338294982910156 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7251497507095337 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9862009882926941 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6306288242340088 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9825584888458252 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9841357469558716 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 7 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.0413781963288784 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.025754496455192566 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.9375 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.44733062386512756 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.10666666179895401 particle/accuracy_2_of_3‚1 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.04511726647615433 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.44071656465530396 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7334538102149963 *jet/accuracy_1_of_4‚0.9661109447479248 particle/accuracy_2_of_5‚1 particle/accuracy_3_of_5‚1 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 7validation_average_jet_accuracy‚0.35012245178222656 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 particle/accuracy_2_of_2‚1 +particle/specificity‚0.9073724150657654 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.875 jet/accuracy_4_of_5‚0 _runtime‚17266.339714888 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6445277333259583 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8703200221061707 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6259775757789612 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5505216121673584 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.15634630620479584 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.7954918742179871 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 8"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.645481526851654 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6089051365852356 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.21782563626766205 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5400312542915344ÙG"(”Ë*‚(:‰ —»éÒóÂ÷ gpu.0.gpu‚90.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚292.207' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚58.4414 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0.K»ËTÌ*‚(:É —»éÒžß·ù disk.rbd3.out‚ 0.22412109375 disk.sda.in‚ 106.21484375 network.sent‚ 1.9999184e+07) disk./.usagePercent‚37.75470010376293 disk.dm-1.in‚ 106.21484375$ disk.dm-0.out‚139159.23486328125* proc.memory.percent‚17.107534408569336 disk.sda1.in‚0 disk.rbd3.in‚0) disk.nvme0c0n1.out‚139159.23486328125 disk.sda2.in‚0$ memory_percent‚82.81960487365723" proc.memory.rssMB‚ 2802.8984375 disk.drbd2165.in‚0$ disk.nvme0c0n1.in‚40754.75390625" disk.nvme0n1.in‚40754.75390625 disk.rbd0.out‚0( proc.memory.availableMB‚ 2814.8359375 network.recv‚ 4.305316e+06 disk.drbd2165.out‚0' disk.nvme0n1.out‚139159.23486328125 disk.sda.out‚ 1215.16015625 disk.dm-1.out‚ 1215.16015625& disk.rbd0.in‚1.759218604441423e+13$ disk./.usageGB‚562.5723152160645 disk.sda3.in‚ 106.21484375 cpu‚25.58672319866216 proc.cpu.threads‚17 disk.sda1.out‚0 disk.sda3.out‚ 1215.16015625 disk.sda2.out‚0 disk.rbd1.out‚ 45.08203125 disk.dm-0.in‚40754.75390625 disk.rbd1.in‚ 0.51220703125Ñòâ^”Í*‚(:‰ ¦»éÒÐòÚö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚277.374' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.4748 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0DØ ÁMÎ*‚(: ¦»éÒù¤Ýø disk.sda2.in‚0" proc.memory.rssMB‚ 2802.8984375 disk.sda3.in‚ 106.21484375 disk.drbd2165.out‚0 disk.rbd3.out‚ 0.22412109375( disk.nvme0c0n1.out‚139168.0673828125 disk.sda.out‚ 1215.421875 disk.rbd3.in‚0 disk.sda2.out‚0 cpu‚25.586664720002865$ disk.nvme0c0n1.in‚40754.97265625 network.sent‚ 2.005691e+07 network.recv‚ 4.309646e+06& disk.rbd0.in‚1.759218604441423e+13# disk.dm-0.out‚139168.0673828125$ memory_percent‚82.82010555267334 disk.dm-1.out‚ 1215.421875 disk.sda1.out‚0 disk.rbd1.in‚ 0.51220703125 disk.rbd1.out‚ 45.16015625 proc.cpu.threads‚17 disk.rbd0.out‚0 disk.dm-0.in‚40754.97265625 disk.sda3.out‚ 1215.421875 disk.sda.in‚ 106.21484375) proc.memory.availableMB‚ 2814.75390625& disk.nvme0n1.out‚139168.0673828125 disk.sda1.in‚0* disk./.usagePercent‚37.851241568955075$ disk./.usageGB‚564.0108528137207 disk.drbd2165.in‚0* proc.memory.percent‚17.107534408569336 disk.dm-1.in‚ 106.21484375" disk.nvme0n1.in‚40754.97265625!žÓ”Ï*‚(:‰ µ»éÒÄ‘þö gpu.0.gpu‚89.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚333.089' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚66.6178 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚09yËJÐ*‚(:¿ µ»éÒ‹—šù network.sent‚ 2.0062041e+07 disk.dm-1.in‚ 106.21484375& disk.rbd0.in‚1.759218604441423e+13 disk.rbd0.out‚0 disk.rbd3.in‚0' disk.nvme0n1.out‚140803.75048828125" proc.memory.rssMB‚ 2802.8984375 network.recv‚ 4.313031e+06 disk.sda.out‚ 1215.82421875 disk.dm-0.in‚ 40755.1875 disk.sda1.out‚0 disk.drbd2165.out‚0 disk.sda2.out‚0) proc.memory.availableMB‚ 2814.76953125$ disk.dm-0.out‚140803.75048828125 disk.sda3.in‚ 106.21484375 disk.rbd3.out‚ 0.22412109375 disk.sda2.in‚0* disk./.usagePercent‚37.766281116589354$ disk./.usageGB‚562.7448806762695 disk.rbd1.out‚ 45.16015625 disk.dm-1.out‚ 1215.82421875 disk.sda1.in‚0 disk.sda3.out‚ 1215.82421875 disk.nvme0n1.in‚ 40755.1875# memory_percent‚82.8200101852417 disk.rbd1.in‚ 0.51220703125) disk.nvme0c0n1.out‚140803.75048828125* proc.memory.percent‚17.107534408569336 cpu‚25.586689200689833 disk.sda.in‚ 106.21484375 disk.drbd2165.in‚0 disk.nvme0c0n1.in‚ 40755.1875 proc.cpu.threads‚17ñtˆ°žÑ*‚(:“ Ä»éÒúŠ÷ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚262.054' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚52.41079999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0à-JÒ*‚(:¿ Ä»éÒ¦ñºù! disk.dm-0.out‚140816.51171875 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd1.in‚ 0.51220703125 disk.dm-1.out‚ 1216.12890625 disk.rbd0.out‚0 disk.rbd3.in‚0& disk.nvme0c0n1.out‚140816.51171875$ disk.nvme0n1.out‚140816.51171875 network.recv‚ 4.315758e+06$ disk.nvme0c0n1.in‚40892.05859375 cpu‚25.58662788192825& disk.rbd0.in‚1.759218604441423e+13 disk.drbd2165.in‚0 disk.drbd2165.out‚0" disk.nvme0n1.in‚40892.05859375$ memory_percent‚82.82079696655273) disk./.usagePercent‚37.76671172126195 disk.sda.out‚ 1216.12890625 disk.rbd1.out‚ 45.22265625 disk.sda3.in‚ 106.21484375 disk.rbd3.out‚ 0.22412109375" proc.memory.rssMB‚ 2802.8984375 proc.cpu.threads‚17 disk.dm-1.in‚ 106.21484375 disk.dm-0.in‚40892.05859375 disk.sda1.out‚0' proc.memory.availableMB‚ 2814.640625 network.sent‚ 2.0066954e+07 disk.sda.in‚ 106.21484375 disk.sda1.in‚0 disk.sda3.out‚ 1216.12890625* proc.memory.percent‚17.107534408569336$ disk./.usageGB‚562.7512969970703ÄCˆ3“Ó*‚(:ˆ Ó»éÒõú“÷ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚38.0 gpu.0.powerWatts‚110.125' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.025 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚01ÔàÊRÔ*‚(:Ç Ó»éÒ‘Ìàø disk.sda.in‚ 106.21484375) disk.nvme0c0n1.out‚140826.55517578125 disk.sda2.out‚0 disk.rbd1.in‚ 0.51220703125 disk.dm-1.in‚ 106.21484375 disk.sda1.out‚0 disk.sda2.in‚0$ memory_percent‚82.82098770141602" proc.memory.rssMB‚ 2802.8984375 disk.sda3.in‚ 106.21484375 disk.dm-1.out‚ 1216.62890625& disk.rbd0.in‚1.759218604441423e+13 disk.drbd2165.in‚0" disk.nvme0n1.in‚40892.27734375 disk.sda.out‚ 1216.62890625$ disk.dm-0.out‚140826.55517578125' proc.memory.availableMB‚ 2814.609375* proc.memory.percent‚17.107534408569336$ disk.nvme0c0n1.in‚40892.27734375' disk.nvme0n1.out‚140826.55517578125 disk.dm-0.in‚40892.27734375 proc.cpu.threads‚17 network.recv‚ 4.320066e+06$ disk./.usageGB‚562.7164764404297 disk.rbd3.out‚ 0.22412109375 disk.rbd1.out‚ 45.22265625 disk.rbd0.out‚0 disk.sda1.in‚0 cpu‚25.58661281748781 network.sent‚ 2.0073756e+07 disk.sda3.out‚ 1216.62890625 disk.rbd3.in‚0( disk./.usagePercent‚37.7643748844894 disk.drbd2165.out‚0Çí1¸4Õ*/ _wandbruntime‚17344  _runtime‚17344Ì?ŒÓËÖ*Å ' lr-AdamW/pg1‚0.00033212169203328403 ' lr-AdamW/pg2‚0.00033212169203328403 trainer/global_step‚10449 % _timestamp‚1.7843071605579898e+09 _runtime‚17344.778092778 _step‚858Á†úâÐ×*Ê _runtime‚17344.778092778 _step‚858 ' lr-AdamW/pg1‚0.00033212169203328403 ' lr-AdamW/pg2‚0.00033212169203328403 trainer/global_step‚10449 % _timestamp‚1.7843071605579898e+09ÚG_T 4Ø*/ _wandbruntime‚17344  _runtime‚17344)´÷¦(Ù*P O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 179701261 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3014553855601152 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 179711240 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚3003424728018944 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚89189227 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚811175998124032 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9832577945088 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚56448726 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚26810422 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 _runtime‚17344.782920305 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚3013239111427462 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 _step‚859 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚3014553855601152 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12686124353024 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3014555227164160 % _timestamp‚1.7843071605654554e+09 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3004722649219072 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚3004722649219072 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚823862122477056 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9815753436808 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚89189221 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3014555227164160 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9832577945088 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚29638304 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚29638049 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚90522013 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚26810417 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚255 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚811175630032896 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚823861719849472 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9814408836486 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚89189227 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820pªrÂì&Ú*†P XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚26810422 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 _runtime‚17344.782920305 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚3013239111427462 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 _step‚859 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚3014553855601152 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12686124353024 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3014555227164160 % _timestamp‚1.7843071605654554e+09 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3004722649219072 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚3004722649219072 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚823862122477056 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9815753436808 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚89189221 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3014555227164160 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9832577945088 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚29638304 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚29638049 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚90522013 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚26810417 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚255 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚811175630032896 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚823861719849472 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9814408836486 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚89189227 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 179701261 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3014553855601152 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 179711240 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚3003424728018944 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚89189227 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚811175998124032 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_p{²B ool.current‚ 47351595008 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9832577945088 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚56448726Ûâ-¬„4Û*/ _wandbruntime‚17346  _runtime‚17346Ó—™_'Ü*‹N T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3014832781366784 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3014834152936960 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3003702584369152 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9832157591040 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚90530505 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚26812960 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚29641037 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9816676903166 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚90530505 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚90520518 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚823940404691456 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚26812965 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚860 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚89197427 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚90520518 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚250 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3014834152936960 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚89197427 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚17346.170032192 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚811253116440576 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚10449 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚56453752 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚811253484531712 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12687322780160 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 179727938 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3005000623775744 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚56454002 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚823940807311872 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9833502987776 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572)k 'Ý*N K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9833502987776 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3014832781366784 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3014834152936960 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3003702584369152 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9832157591040 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚90530505 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚26812960 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚29641037 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9816676903166 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚90530505 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚90520518 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚823940404691456 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚26812965 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚860 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚89197427 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚90520518 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚250 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3014834152936960 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚89197427 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚17346.170032192 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚811253116440576 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚10449 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚56453752 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚811253484531712 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12687322780160 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 179727938 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3005000623775744 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚56454002 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚823940807311872 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995Ü¿‹Ð4Þ*/ _wandbruntime‚17346  _runtime‚17346š¯Íß*ˆ 2loss/SRqqt1/assignment_loss‚0.2787507006282352 2loss/SRqqt2/assignment_loss‚0.2808086685756845 /loss/FRt1/detection_loss‚0.4228971802323908 /loss/FBt1/detection_loss‚0.4623384429791639 0loss/FRt2/assignment_loss‚0.3663879944483439 3loss/FBt1/assignment_loss‚9.568067352874292e-05 %loss/total_loss‚6.114151437592971 trainer/global_step‚10449 _runtime‚17346.254569631 _step‚861 /loss/FRt2/detection_loss‚1.3821945746410638 2loss/FBt2/assignment_loss‚0.001421482065152434 1loss/SRqqt1/detection_loss‚0.6214665887900483 1loss/SRqqt2/detection_loss‚0.5669778390376082 0loss/FBt2/detection_loss‚0.48932224542358166 epoch‚26 % _timestamp‚1.7843071620376177e+09 0loss/FRt1/assignment_loss‚1.2414900400981692)]ÙÔà* /loss/FRt2/detection_loss‚1.3821945746410638 2loss/FBt2/assignment_loss‚0.001421482065152434 1loss/SRqqt1/detection_loss‚0.6214665887900483 1loss/SRqqt2/detection_loss‚0.5669778390376082 0loss/FBt2/detection_loss‚0.48932224542358166 epoch‚26 % _timestamp‚1.7843071620376177e+09 0loss/FRt1/assignment_loss‚1.2414900400981692 2loss/SRqqt1/assignment_loss‚0.2787507006282352 2loss/SRqqt2/assignment_loss‚0.2808086685756845 /loss/FRt1/detection_loss‚0.4228971802323908 /loss/FBt1/detection_loss‚0.4623384429791639 0loss/FRt2/assignment_loss‚0.3663879944483439 3loss/FBt1/assignment_loss‚9.568067352874292e-05 %loss/total_loss‚6.114151437592971 trainer/global_step‚10449 _runtime‚17346.254569631 _step‚861Ý+Cðxžá*‚(:“ â»éҼ߮÷ gpu.0.gpu‚90.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚283.228' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚56.64560000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Qd"<â*‚(:± â»éÒ˽‡ù disk.sda.out‚ 1216.8125# disk.nvme0n1.out‚140834.3359375" disk.nvme0n1.in‚40892.69921875) disk./.usagePercent‚37.76486795500035 disk.dm-0.in‚40892.69921875 disk.drbd2165.out‚0 network.sent‚ 2.0077518e+07 disk.sda1.in‚0 disk.rbd3.in‚0% disk.nvme0c0n1.out‚140834.3359375) proc.memory.availableMB‚ 2814.34765625 proc.cpu.threads‚17 disk.sda3.out‚ 1216.8125 disk.drbd2165.in‚0 disk.rbd1.in‚ 0.51220703125 disk.rbd1.out‚ 45.30078125* proc.memory.percent‚17.107534408569336 cpu‚25.586580273813468 network.recv‚ 4.323504e+06$ disk./.usageGB‚562.7238235473633& disk.rbd0.in‚1.759218604441423e+13 disk.sda2.in‚0$ disk.nvme0c0n1.in‚40892.69921875 disk.sda.in‚ 106.21484375 disk.dm-1.in‚ 106.21484375 disk.rbd0.out‚0 disk.sda1.out‚0 disk.sda3.in‚ 106.21484375 disk.dm-1.out‚ 1216.8125 disk.dm-0.out‚140834.3359375 disk.rbd3.out‚ 0.22412109375 disk.sda2.out‚0" memory_percent‚82.822585105896" proc.memory.rssMB‚ 2802.8984375—!Øžã*‚(:“ ñ»éÒÍìÏö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚279.79' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚55.958000000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¿ÓšÀUä*‚(:Ê ñ»éÒÆšÇø disk.rbd3.out‚ 0.22412109375" disk.nvme0n1.in‚40892.90234375 disk.sda2.out‚0$ memory_percent‚82.82065391540527 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda3.in‚ 106.21484375' disk.nvme0n1.out‚140846.84814453125 disk.dm-0.in‚40892.90234375$ disk.dm-0.out‚140846.84814453125( proc.memory.availableMB‚ 2814.6640625* proc.memory.percent‚17.107534408569336 proc.cpu.threads‚17 network.recv‚ 4.328132e+06$ disk./.usageGB‚562.7280921936035 disk.drbd2165.in‚0 disk.rbd3.in‚0 disk.sda.in‚ 106.21484375& disk.rbd0.in‚1.759218604441423e+13 disk.rbd0.out‚0 disk.sda2.in‚0 disk.sda.out‚ 1216.98046875 disk.rbd1.out‚ 45.30078125" proc.memory.rssMB‚ 2802.8984375 cpu‚25.586507177069674 network.sent‚ 2.0140696e+07) disk./.usagePercent‚37.76515442743116 disk.drbd2165.out‚0$ disk.nvme0c0n1.in‚40892.90234375 disk.rbd1.in‚ 0.51220703125 disk.sda3.out‚ 1216.98046875) disk.nvme0c0n1.out‚140846.84814453125 disk.dm-1.in‚ 106.21484375 disk.dm-1.out‚ 1216.980468752ŸËF“å*‚(:ˆ €¼éÒºá÷ gpu.0.gpu‚13.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚331.202' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚66.2404 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚01ïRæ*‚(:Ç €¼éÒ¯¦Ýù disk.sda2.in‚0 disk.dm-1.in‚ 106.21484375( disk.nvme0c0n1.out‚140854.7470703125 disk.rbd0.out‚0 disk.sda1.in‚0$ disk.nvme0c0n1.in‚41140.76171875 disk.dm-1.out‚ 1217.44140625 network.recv‚ 4.331582e+06) disk./.usagePercent‚37.76562701733937 disk.rbd1.out‚ 45.36328125* proc.memory.percent‚17.107534408569336 disk.sda1.out‚0 disk.sda3.out‚ 1217.44140625 network.sent‚ 2.014582e+07$ disk./.usageGB‚562.7351341247559 proc.cpu.threads‚17 disk.dm-0.in‚41140.76171875 disk.rbd3.in‚0 disk.rbd3.out‚ 0.22412109375 disk.sda.out‚ 1217.44140625" disk.nvme0n1.in‚41140.76171875& disk.nvme0n1.out‚140854.7470703125 disk.sda2.out‚0) proc.memory.availableMB‚ 2814.41796875 disk.sda.in‚ 106.21484375# disk.dm-0.out‚140854.7470703125 disk.sda3.in‚ 106.21484375 disk.drbd2165.in‚0$ memory_percent‚82.82215595245361 cpu‚25.586529525416008 disk.rbd1.in‚ 0.51220703125& disk.rbd0.in‚1.759218604441423e+13 disk.drbd2165.out‚0" proc.memory.rssMB‚ 2802.8984375Ûó»6Ÿç*‚(:” ¼éÒʈ÷ gpu.0.gpu‚94.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚295.159' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚59.031800000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0˜ng9Kè*‚(:À ¼éÒÓ¥áù$ disk./.usageGB‚562.7418670654297' disk.nvme0c0n1.out‚140869.416015625 network.sent‚ 2.0151019e+07 disk.rbd1.in‚ 0.51220703125 cpu‚25.586497969650814 disk.rbd3.out‚ 0.22412109375 disk.sda.in‚ 106.21484375 disk.dm-1.out‚ 1217.7890625& disk.rbd0.in‚1.759218604441423e+13 disk.rbd0.out‚0 disk.sda1.in‚0( proc.memory.availableMB‚ 2814.4140625* proc.memory.percent‚17.107534408569336 disk.sda2.in‚0 disk.sda.out‚ 1217.7890625 disk.dm-1.in‚ 106.21484375 disk.sda3.out‚ 1217.7890625 disk.drbd2165.out‚0 disk.rbd3.in‚0( disk./.usagePercent‚37.7660788706373 disk.drbd2165.in‚0% disk.nvme0n1.out‚140869.416015625 disk.dm-0.in‚41141.18359375 disk.sda1.out‚0" proc.memory.rssMB‚ 2802.8984375 disk.sda2.out‚0 disk.rbd1.out‚ 45.36328125$ disk.nvme0c0n1.in‚41141.18359375" disk.nvme0n1.in‚41141.18359375" disk.dm-0.out‚140869.416015625 disk.sda3.in‚ 106.21484375$ memory_percent‚82.82217979431152 proc.cpu.threads‚17 network.recv‚ 4.335086e+06Üâ/Z”é*‚(:‰ ž¼éÒÔ´åö gpu.0.gpu‚57.0 gpu.0.memory‚17, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚37.0 gpu.0.powerWatts‚110.012' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.0024 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ã¥‘8Vê*‚(:Ë ž¼éÒ¡Ñ°ù$ disk./.usageGB‚562.7463073730469 disk.rbd0.out‚0$ disk.nvme0c0n1.in‚41141.38671875 network.recv‚ 4.338287e+06) disk./.usagePercent‚37.76637686340716" disk.nvme0n1.in‚41141.38671875 disk.sda.in‚ 106.21484375 disk.dm-1.in‚ 106.21484375 disk.dm-0.in‚41141.38671875 disk.sda3.in‚ 106.21484375 disk.rbd3.out‚ 0.22412109375 network.sent‚ 2.0156168e+07 disk.rbd1.in‚ 0.51220703125 cpu‚25.586496440183087 disk.dm-1.out‚ 1217.94140625 disk.drbd2165.in‚0" proc.memory.rssMB‚ 2802.8984375 disk.sda.out‚ 1217.94140625& disk.rbd0.in‚1.759218604441423e+13 disk.sda1.in‚0) disk.nvme0c0n1.out‚140877.25537109375 proc.cpu.threads‚17 disk.rbd1.out‚ 45.44140625 disk.sda1.out‚0 disk.sda3.out‚ 1217.94140625 disk.drbd2165.out‚0$ memory_percent‚82.82215595245361) proc.memory.availableMB‚ 2814.41796875 disk.sda2.in‚0 disk.sda2.out‚0$ disk.dm-0.out‚140877.25537109375' disk.nvme0n1.out‚140877.25537109375 disk.rbd3.in‚0* proc.memory.percent‚17.107534408569336“<'ç4ë*/ _wandbruntime‚17424  _runtime‚17424ÞË2Èì* & lr-AdamW/pg1‚0.0003293520435599983 & lr-AdamW/pg2‚0.0003293520435599983 trainer/global_step‚10499 $ _timestamp‚1.784307240194735e+09 _runtime‚17424.411845691 _step‚862&½ËÍí*Ç & lr-AdamW/pg2‚0.0003293520435599983 trainer/global_step‚10499 $ _timestamp‚1.784307240194735e+09 _runtime‚17424.411845691 _step‚862 & lr-AdamW/pg1‚0.0003293520435599983ÞÎ ³4î*/ _wandbruntime‚17424  _runtime‚17424äEH ï*€P JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 180547292 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404724736 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12745816669696 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚56713238 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12745780036096 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚89599527 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚3028500208037376 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚56713517 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚274 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚3018622685724672 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3027180807477660 _step‚863 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚89599527 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚279 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9878893875712 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚827795730452992 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚3017318816927744 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚90947765 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚90937792 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚26937567 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚3017318791499776 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9861990549916 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚815050318508032 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3028500208037376 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚3018622659551232 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚29775945 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚29775671 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚827796135177728 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3028501579600384 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216¢´¯¸ (ð*…P @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3018622659551232 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚90937792 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36633600 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _runtime‚17424.412741061 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3028501579600384 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚89599521 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚90947765 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚815049950416896 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 180537313 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 180547292 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404724736 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12745816669696 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚56713238 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12745780036096 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚89599527 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚3028500208037376 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚56713517 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚274 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚3018622685724672 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3027180807477660 _step‚863 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚89599527 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚279 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9878893875712 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚827795730452992 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚3017318816927744 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚90947765 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚90937792 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚26937567 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚3017318791499776 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9861990549916 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚815050318508032 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3028500208037376 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚3018622659551232 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚29775945 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚29775671 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚827796135177728 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3028501579600384 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚89607733 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚89607727 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 _step‚864 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚90956257 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 trainer/global_step‚10499 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 180563990 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚3018900686454784 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚90956257 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12747000227328 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3028779133803008 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3027459612722194 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9862914016274 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚26940110 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 180553997 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚89607733 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9879818918400 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9878473521664 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚89607727 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3018900686454784 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404717568 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3028780505373184 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9878473521664 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚29778682 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚3027458242693792 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚815127436824576 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9861569415840 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 180563990 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚17425.806288553 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3017596673277952 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚56718526 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚815127804915712 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36626432 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 180553997 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3028780505373184 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9879818918400 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0a–Õšó*N H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 180563990 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚3018900686454784 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚90956257 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12747000227328 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3028779133803008 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3027459612722194 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9862914016274 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚26940110 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 180553997 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚89607733 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9879818918400 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9878473521664 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚89607727 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3018900686454784 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404717568 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3028780505373184 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9878473521664 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚29778682 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚3027458242693792 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.curre§OQ´|nt‚1175 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚815127436824576 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9861569415840 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 180563990 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚17425.806288553 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3017596673277952 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚56718526 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚815127804915712 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36626432 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 180553997 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3028780505373184 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9879818918400 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚271 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚89607733 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚89607727 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 _step‚864 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚90956257 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 trainer/global_step‚10499à‡Ýgæ4ô*/ _wandbruntime‚17425  _runtime‚17425é¨ây õ*‡ 2loss/FBt1/assignment_loss‚0.000458343150612743 2loss/FBt2/assignment_loss‚0.003195348984075939 trainer/global_step‚10499 /loss/FRt1/assignment_loss‚1.284671111811547 0loss/FRt2/assignment_loss‚0.3645051790678755 1loss/SRqqt2/detection_loss‚0.5316954430710297 %loss/total_loss‚6.269054369365564 % _timestamp‚1.7843072416705961e+09 _runtime‚17425.887496313 /loss/FRt2/detection_loss‚1.3714807534397164 1loss/SRqqt1/detection_loss‚0.6937540447022887 /loss/FBt1/detection_loss‚0.5146380105656326 epoch‚26 _step‚865 2loss/SRqqt1/assignment_loss‚0.3448419332935842 2loss/SRqqt2/assignment_loss‚0.2721615005279634 0loss/FRt1/detection_loss‚0.47428484985938235 0loss/FBt2/detection_loss‚0.41336785089185446O;ö*Œ /loss/FRt1/assignment_loss‚1.284671111811547 0loss/FRt2/assignment_loss‚0.3645051790678755 1loss/SRqqt2/detection_loss‚0.5316954430710297 %loss/total_loss‚6.269054369365564 % _timestamp‚1.7843072416705961e+09 _runtime‚17425.887496313 /loss/FRt2/detection_loss‚1.3714807534397164 1loss/SRqqt1/detection_loss‚0.6937540447022887 /loss/FBt1/detection_loss‚0.5146380105656326 epoch‚26 _step‚865 2loss/SRqqt1/assignment_loss‚0.3448419332935842 2loss/SRqqt2/assignment_loss‚0.2721615005279634 0loss/FRt1/detection_loss‚0.47428484985938235 0loss/FBt2/detection_loss‚0.41336785089185446 2loss/FBt1/assignment_loss‚0.000458343150612743 2loss/FBt2/assignment_loss‚0.003195348984075939 trainer/global_step‚10499áae ÷*‚(:• ­¼éÒѵûö gpu.0.gpu‚100.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚287.087' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚57.417399999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0× mMø*‚(: ­¼éÒ‚ÍÕù disk.drbd2165.in‚0 disk.sda2.in‚0& disk.rbd0.in‚1.759218604441423e+13 disk.rbd0.out‚0 disk.sda1.in‚0' disk.nvme0c0n1.out‚140889.009765625" disk.nvme0n1.in‚41141.80859375 disk.sda3.out‚ 1218.1953125* proc.memory.percent‚17.107534408569336 proc.cpu.threads‚17 network.recv‚ 4.341585e+06% disk.nvme0n1.out‚140889.009765625 disk.rbd1.in‚ 0.51220703125 disk.rbd1.out‚ 45.44140625 disk.dm-1.in‚ 106.21484375 disk.rbd3.in‚0 disk.rbd3.out‚ 0.22412109375 disk.sda.out‚ 1218.1953125* disk./.usagePercent‚37.767003057835915 disk.dm-1.out‚ 1218.1953125$ disk./.usageGB‚562.7556381225586" disk.dm-0.out‚140889.009765625 disk.sda1.out‚0 disk.sda3.in‚ 106.21484375" proc.memory.rssMB‚ 2802.8984375 cpu‚25.58650714043401 disk.dm-0.in‚41141.80859375 network.sent‚ 2.0161297e+07 disk.drbd2165.out‚0 disk.sda2.out‚0) proc.memory.availableMB‚ 2813.90234375$ disk.nvme0c0n1.in‚41141.80859375 disk.sda.in‚ 106.21484375$ memory_percent‚82.82530307769775Gá°c”ù*‚(:‰ ¼¼éÒþ¤÷ö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚256.168' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚51.2336 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0;º<¸Nú*‚(:à ¼¼éÒ¦”Äù disk.sda3.in‚ 106.21484375( disk.nvme0c0n1.out‚140902.2333984375! disk.nvme0n1.in‚ 41706.8046875 disk.rbd1.in‚ 0.51220703125 disk.rbd1.out‚ 45.50390625 disk.rbd0.out‚0 network.sent‚ 2.0224575e+07 network.recv‚ 4.346093e+06) disk./.usagePercent‚37.76730309866604 disk.dm-1.in‚ 106.21484375# disk.dm-0.out‚140902.2333984375 disk.sda2.out‚0* proc.memory.percent‚17.107534408569336 disk.sda1.in‚0 proc.cpu.threads‚17$ disk./.usageGB‚562.7601089477539 disk.rbd3.out‚ 0.22412109375& disk.nvme0n1.out‚140902.2333984375$ memory_percent‚82.82532691955566 cpu‚25.58644758397742 disk.sda3.out‚ 1218.42578125# disk.nvme0c0n1.in‚ 41706.8046875" proc.memory.rssMB‚ 2802.8984375 disk.sda.in‚ 106.21484375 disk.dm-1.out‚ 1218.42578125& disk.rbd0.in‚1.759218604441423e+13 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.rbd3.in‚0 disk.sda2.in‚0( proc.memory.availableMB‚ 2813.8984375 disk.sda1.out‚0 disk.sda.out‚ 1218.42578125 disk.dm-0.in‚ 41706.8046875ž‘•û*‚(:Š ˼éÒÉð’÷ gpu.0.gpu‚100.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚320.587' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚64.1174 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0&Ì™=Mü*‚(: ˼éÒ¤ÛËù) disk./.usagePercent‚37.76788218770849 disk.rbd3.in‚0$ memory_percent‚82.82535076141357 proc.cpu.threads‚17 disk.rbd1.in‚ 0.51220703125 disk.sda.out‚ 1218.73046875 disk.dm-1.out‚ 1218.73046875" proc.memory.rssMB‚ 2802.8984375 cpu‚25.586430437847635 disk.sda3.out‚ 1218.73046875 disk.drbd2165.out‚0 disk.rbd0.out‚0" disk.dm-0.out‚140914.294921875 network.sent‚ 2.0229709e+07$ disk./.usageGB‚562.7687377929688 disk.sda3.in‚ 106.21484375 disk.rbd1.out‚ 45.50390625 disk.sda.in‚ 106.21484375 disk.dm-0.in‚ 41707.0078125 network.recv‚ 4.349614e+06 disk.sda1.out‚0 disk.drbd2165.in‚0! disk.nvme0n1.in‚ 41707.0078125 disk.sda2.out‚0* proc.memory.percent‚17.107534408569336# disk.nvme0c0n1.in‚ 41707.0078125' disk.nvme0c0n1.out‚140914.294921875 disk.sda1.in‚0 disk.rbd3.out‚ 0.22412109375 disk.sda2.in‚0 disk.dm-1.in‚ 106.21484375& disk.rbd0.in‚1.759218604441423e+13% disk.nvme0n1.out‚140914.294921875) proc.memory.availableMB‚ 2813.89453125\nUžý*‚(:“ Ú¼éÒȇ¥÷ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚280.367' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚56.07340000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ú—;Kþ*‚(:À Ú¼éÒ—³ðú disk.rbd1.in‚ 0.51220703125 disk.rbd0.out‚0 disk.sda3.in‚ 106.21484375! disk.nvme0n1.in‚ 41707.4296875 disk.rbd1.out‚ 45.58203125 cpu‚25.586352499063768 network.sent‚ 2.0234804e+07$ disk./.usageGB‚562.7753410339355 disk.sda.in‚ 106.21484375 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.rbd3.out‚ 0.22412109375 disk.sda2.in‚0$ memory_percent‚82.82535076141357* disk./.usagePercent‚37.768325336750266 disk.sda2.out‚0* proc.memory.percent‚17.107534408569336 proc.cpu.threads‚17 disk.sda3.out‚ 1219.046875 disk.rbd3.in‚0" proc.memory.rssMB‚ 2802.8984375 disk.dm-1.out‚ 1219.046875& disk.rbd0.in‚1.759218604441423e+13) proc.memory.availableMB‚ 2813.89453125 disk.sda1.in‚0 disk.drbd2165.out‚0# disk.nvme0c0n1.in‚ 41707.4296875( disk.nvme0c0n1.out‚140926.8759765625& disk.nvme0n1.out‚140926.8759765625 network.recv‚ 4.352726e+06 disk.sda.out‚ 1219.046875 disk.dm-1.in‚ 106.21484375 disk.dm-0.in‚ 41707.4296875# disk.dm-0.out‚140926.8759765625j»ÇtŸÿ*‚(:” é¼éÒîöÿö gpu.0.gpu‚89.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚40.0 gpu.0.powerWatts‚188.822' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚37.764399999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0jö=ÉO€+‚(:Ä é¼éÒÔ–Âù network.recv‚ 4.356547e+06 disk.sda1.in‚0 disk.rbd1.out‚ 45.58203125 proc.cpu.threads‚17 disk.drbd2165.in‚0& disk.nvme0n1.out‚140938.4482421875* proc.memory.percent‚17.107534408569336& disk.rbd0.in‚1.759218604441423e+13 disk.sda1.out‚0( disk.nvme0c0n1.out‚140938.4482421875! disk.nvme0n1.in‚ 41707.6484375 disk.sda2.out‚0( proc.memory.availableMB‚ 2813.8984375 network.sent‚ 2.023995e+07$ disk./.usageGB‚562.7822761535645# disk.dm-0.out‚140938.4482421875 disk.sda2.in‚0 disk.rbd1.in‚ 0.51220703125 disk.sda.in‚ 106.21484375 disk.dm-0.in‚ 41707.6484375 disk.drbd2165.out‚0 disk.rbd3.in‚0 disk.sda.out‚ 1219.27734375 disk.sda3.in‚ 106.21484375 disk.rbd0.out‚0 disk.rbd3.out‚ 0.22412109375 disk.dm-1.in‚ 106.21484375$ memory_percent‚82.82532691955566" proc.memory.rssMB‚ 2802.8984375 cpu‚25.586382537732447* disk./.usagePercent‚37.768790758447516 disk.sda3.out‚ 1219.27734375# disk.nvme0c0n1.in‚ 41707.6484375 disk.dm-1.out‚ 1219.27734375#h·14+/ _wandbruntime‚17504  _runtime‚17504·‹¡ÔË‚+Å ' lr-AdamW/pg2‚0.00032658345948701964 trainer/global_step‚10549 % _timestamp‚1.7843073200367923e+09 _runtime‚17504.253972772 _step‚866 ' lr-AdamW/pg1‚0.00032658345948701964"æõ¶Ðƒ+Ê ' lr-AdamW/pg2‚0.00032658345948701964 trainer/global_step‚10549 % _timestamp‚1.7843073200367923e+09 _runtime‚17504.253972772 _step‚866 ' lr-AdamW/pg1‚0.00032658345948701964â~Ye4„+/ _wandbruntime‚17504  _runtime‚17504Å< å(…+P ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9923864416768 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚90009827 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚3031212880408576 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 _step‚867 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3042446560473600 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3042447932036608 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚3032522722230272 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404724736 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12805045403136 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 % _timestamp‚1.7843073200379086e+09 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚3042446560473600 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9925209806336 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚56976366 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9925209806336 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚831729720928768 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 trainer/global_step‚10549 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚91363544 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚27064717 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚3041119763471278 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9908227663024 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚29911649 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚831729316204032 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3041121133499568 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 _runtime‚17504.255243892 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚90009827 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚91373517 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚818924638892032 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚3032522696056832 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9906883062702 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 181373365 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚91363544 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚3031212905836544 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36633600 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚90009821 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚56976640 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3032522696056832 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3042447932036608 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3032522722230272\Ý †+†P VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚90009827 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚3031212880408576 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 _step‚867 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3042446560473600 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3042447932036608 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚3032522722230272 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404724736 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12805045403136 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 % _timestamp‚1.7843073200379086e+09 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚3042446560473600 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9925209806336 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚56976366 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9925209806336 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚831729720928768 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 trainer/global_step‚10549 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚91363544 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚27064717 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚3041119763471278 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9908227663024 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚29911649 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚831729316204032 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3041121133499568 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 _runti`Ü»ù me‚17504.255243892 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚90009827 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚91373517 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚818924638892032 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚3032522696056832 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9906883062702 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 181373365 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚91363544 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚3031212905836544 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36633600 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚90009821 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚56976640 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3032522696056832 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3042447932036608 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3032522722230272 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9923864416768ãtÏvJ”‡+‚(:‰ ø¼éÒïã÷ö gpu.0.gpu‚90.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚249.149' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚49.8298 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0š¿Jˆ+‚(:¿ ø¼éÒü¸ñù) disk./.usagePercent‚37.76919704240435 disk.sda.in‚ 106.21484375 disk.sda.out‚ 1219.78515625& disk.rbd0.in‚1.759218604441423e+13 disk.sda3.out‚ 1219.78515625" disk.nvme0c0n1.in‚ 41734.140625* proc.memory.percent‚17.107534408569336 disk.sda1.out‚0 disk.rbd3.in‚0 network.sent‚ 2.024519e+07# disk./.usageGB‚562.788330078125& disk.nvme0n1.out‚140946.4560546875 disk.rbd1.out‚ 45.64453125 disk.dm-1.in‚ 106.21484375 disk.dm-0.in‚ 41734.140625 disk.sda1.in‚0 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.rbd3.out‚ 0.22412109375( proc.memory.availableMB‚ 2813.6484375 proc.cpu.threads‚17 disk.sda3.in‚ 106.21484375 disk.sda2.in‚0# disk.dm-0.out‚140946.4560546875( disk.nvme0c0n1.out‚140946.4560546875 disk.nvme0n1.in‚ 41734.140625 cpu‚25.586407315776214 disk.sda2.out‚0 disk.rbd1.in‚ 0.51220703125 network.recv‚ 4.359957e+06 disk.dm-1.out‚ 1219.78515625 disk.rbd0.out‚0$ memory_percent‚82.82685279846191" proc.memory.rssMB‚ 2802.8984375öX(>4‰+/ _wandbruntime‚17505  _runtime‚17505ßzÿ'Š+‹N TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚3032800722960384 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚27067260 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 181390049 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12806270132736 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3041399938744102 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9909151129382 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 _runtime‚17505.642143951 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3042725486239232 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 181400042 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚27067265 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 181400042 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9926134849024 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12806233506304 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 % _timestamp‚1.7843073214247892e+09 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚90018033 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚56981665 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404717568 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3031490762186752 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚258 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚819001757208576 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚29914663 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚831807990714880 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9924789452288 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3042726857809408 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚91372022 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 181390049 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3031490787614720 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9926134849024 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3042726857809408 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚91372022 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 181390049 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3031490787614720 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9926134849024 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚3032800722960384 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚27067260 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 181390049 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12806270132736 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3041399938744102 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9909151129382 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 _runtime‚17505.642143951 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3042725486239232 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 181400042 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚27067265 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 181400042 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9926134849024 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12806233506304 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 % _timestamp‚1.7843073214247892e+09 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚90018033 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚56981665 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404717568 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3031490762186752 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚258 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚819001757208576 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚29914663 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚831807990714880 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9924789452288 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192ä«ij4Œ+/ _wandbruntime‚17505  _runtime‚17505¨ŽÕ+‰ _runtime‚17505.726991257 1loss/FRt2/assignment_loss‚0.39746850880269474 0loss/SRqqt1/detection_loss‚0.662285257629089 0loss/FBt2/detection_loss‚0.43514331398531797 _step‚869 /loss/FRt2/detection_loss‚1.4292726249376142 1loss/SRqqt2/detection_loss‚0.5398180129697108 1loss/SRqqt1/assignment_loss‚0.285920447510203 3loss/SRqqt2/assignment_loss‚0.33518668855917394 2loss/FBt1/assignment_loss‚0.011564949982939313 &loss/total_loss‚6.4813895277722455 epoch‚26 % _timestamp‚1.7843073215100348e+09 0loss/FRt1/assignment_loss‚1.1964444489732606 2loss/FBt2/assignment_loss‚0.009152824690719229 0loss/FRt1/detection_loss‚0.46610357901039295 /loss/FBt1/detection_loss‚0.7130288707211296 trainer/global_step‚10549.0ZÿŽ+Ž &loss/total_loss‚6.4813895277722455 epoch‚26 % _timestamp‚1.7843073215100348e+09 0loss/FRt1/assignment_loss‚1.1964444489732606 2loss/FBt2/assignment_loss‚0.009152824690719229 0loss/FRt1/detection_loss‚0.46610357901039295 /loss/FBt1/detection_loss‚0.7130288707211296 trainer/global_step‚10549 _runtime‚17505.726991257 1loss/FRt2/assignment_loss‚0.39746850880269474 0loss/SRqqt1/detection_loss‚0.662285257629089 0loss/FBt2/detection_loss‚0.43514331398531797 _step‚869 /loss/FRt2/detection_loss‚1.4292726249376142 1loss/SRqqt2/detection_loss‚0.5398180129697108 1loss/SRqqt1/assignment_loss‚0.285920447510203 3loss/SRqqt2/assignment_loss‚0.33518668855917394 2loss/FBt1/assignment_loss‚0.011564949982939313å€Fû¥Ÿ+‚(:” ‡½éÒ‰‹×ö gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚284.382' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚56.876400000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0mâLUP+‚(:Å ‡½éÒǾóø" memory_percent‚82.826828956604 disk.dm-1.in‚ 106.21484375 disk.rbd0.out‚0 disk.sda1.out‚0) proc.memory.availableMB‚ 2813.65234375 disk.rbd1.in‚ 0.51220703125 disk.sda1.in‚0 network.recv‚ 4.363968e+06! disk.nvme0n1.in‚ 41827.0390625 disk.rbd1.out‚ 45.64453125) disk./.usagePercent‚37.76947941871462) disk.nvme0c0n1.out‚140964.12060546875' disk.nvme0n1.out‚140964.12060546875 disk.dm-0.in‚ 41827.0390625$ disk.dm-0.out‚140964.12060546875 disk.sda3.in‚ 106.21484375 disk.drbd2165.out‚0 cpu‚25.586336461560084 disk.sda.out‚ 1220.02734375 disk.rbd3.in‚0# disk./.usageGB‚562.792537689209 disk.sda2.out‚0 disk.sda.in‚ 106.21484375& disk.rbd0.in‚1.759218604441423e+13* proc.memory.percent‚17.107534408569336 proc.cpu.threads‚17# disk.nvme0c0n1.in‚ 41827.0390625 disk.drbd2165.in‚0 disk.rbd3.out‚ 0.22412109375" proc.memory.rssMB‚ 2802.8984375 network.sent‚ 2.0296887e+07 disk.sda2.in‚0 disk.dm-1.out‚ 1220.02734375 disk.sda3.out‚ 1220.027343755ýI”‘+‚(:‰ –½éÒÆöëö gpu.0.gpu‚84.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚335.642' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚67.1284 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0—:EíM’+‚(: –½éÒµ¦£ù cpu‚25.58633273850909 network.sent‚ 2.034941e+07 disk.dm-1.in‚ 106.21484375 disk.dm-1.out‚ 1220.18359375 disk.sda1.in‚0# disk.nvme0c0n1.in‚ 41827.2578125 proc.cpu.threads‚17 disk.rbd1.in‚ 0.51220703125 network.recv‚ 4.368307e+06 disk.drbd2165.out‚0 disk.sda2.out‚0( proc.memory.availableMB‚ 2813.8203125$ disk./.usageGB‚562.7995491027832 disk.sda1.out‚0! disk.nvme0n1.in‚ 41827.2578125 disk.sda.out‚ 1220.18359375( disk.nvme0c0n1.out‚140973.4951171875& disk.nvme0n1.out‚140973.4951171875 disk.sda2.in‚0$ memory_percent‚82.82580375671387" proc.memory.rssMB‚ 2802.8984375) disk./.usagePercent‚37.76994996056255 disk.sda.in‚ 106.21484375& disk.rbd0.in‚1.759218604441423e+13# disk.dm-0.out‚140973.4951171875 disk.rbd3.in‚0* proc.memory.percent‚17.107534408569336 disk.rbd0.out‚0 disk.dm-0.in‚ 41827.2578125 disk.drbd2165.in‚0 disk.rbd3.out‚ 0.22412109375 disk.sda3.in‚ 106.21484375 disk.sda3.out‚ 1220.18359375 disk.rbd1.out‚ 45.72265625{A›^Ÿ“+‚(:” ¥½éÒÔåîö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚258.586' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚51.717200000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ÿr O”+‚(:Ä ¥½éÒ¸ÑÈù# disk.dm-0.out‚140988.5634765625) disk./.usagePercent‚37.77040283789062 disk.rbd3.out‚ 0.22412109375 disk.sda2.out‚0& disk.rbd0.in‚1.759218604441423e+13 disk.dm-0.in‚ 41827.6796875 disk.sda3.in‚ 106.21484375" proc.memory.rssMB‚ 2802.8984375 disk.rbd3.in‚0 disk.rbd1.in‚ 0.51220703125 disk.rbd1.out‚ 45.72265625$ disk./.usageGB‚562.8062973022461 disk.sda2.in‚0 disk.sda1.out‚0 cpu‚25.586329310649052 disk.sda1.in‚0! disk.nvme0n1.in‚ 41827.6796875 disk.sda.in‚ 106.21484375 disk.drbd2165.in‚0 disk.drbd2165.out‚0 network.recv‚ 4.37193e+06( disk.nvme0c0n1.out‚140988.5634765625* proc.memory.percent‚17.107534408569336 proc.cpu.threads‚17 disk.dm-1.in‚ 106.21484375$ memory_percent‚82.82592296600342) proc.memory.availableMB‚ 2813.80078125 disk.sda3.out‚ 1220.47265625 network.sent‚ 2.0354501e+07# disk.nvme0c0n1.in‚ 41827.6796875& disk.nvme0n1.out‚140988.5634765625 disk.sda.out‚ 1220.47265625 disk.dm-1.out‚ 1220.47265625 disk.rbd0.out‚0Pÿ\¤”•+‚(:‰ ´½éÒ“«÷ gpu.0.gpu‚83.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚339.166' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚67.8332 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ΪL–+‚(:Á ´½éÒ“·Œú disk.sda1.in‚0# memory_percent‚82.8226089477539' disk.nvme0n1.out‚140995.96826171875 disk.dm-1.in‚ 106.21484375 disk.dm-1.out‚ 1220.6640625 disk.rbd0.out‚0$ disk.dm-0.out‚140995.96826171875 network.recv‚ 4.375317e+06) disk./.usagePercent‚37.77070518278856$ disk./.usageGB‚562.8108024597168 disk.sda2.in‚0 disk.dm-0.in‚ 41827.8828125 disk.sda1.out‚0 disk.rbd3.in‚0& disk.rbd0.in‚1.759218604441423e+13& proc.memory.availableMB‚ 2814.34375" proc.memory.rssMB‚ 2802.8984375* proc.memory.percent‚17.107534408569336) disk.nvme0c0n1.out‚140995.96826171875 disk.sda.out‚ 1220.6640625 disk.sda3.in‚ 106.21484375 disk.sda3.out‚ 1220.6640625# disk.nvme0c0n1.in‚ 41827.8828125 disk.sda2.out‚0 cpu‚25.586353568908287 proc.cpu.threads‚17 network.sent‚ 2.0359642e+07 disk.drbd2165.in‚0! disk.nvme0n1.in‚ 41827.8828125 disk.rbd1.out‚ 45.84765625 disk.sda.in‚ 106.21484375 disk.drbd2165.out‚0 disk.rbd3.out‚ 0.22412109375 disk.rbd1.in‚ 0.51220703125¡ Ý»Ÿ—+‚(:” ýéÒÆŸÉ÷ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚297.863' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚59.572599999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¦â9ĵ˜+‚(:Ä ýéÒ¢ý³û disk.sda3.out‚ 1220.97265625" disk.nvme0c0n1.in‚ 41942.203125 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-1.in‚ 106.21484375 disk.dm-0.i<‘Éšn‚ 41942.203125 disk.rbd3.in‚0 disk.rbd3.out‚ 0.22412109375 disk.nvme0n1.in‚ 41942.203125 disk.rbd1.in‚ 0.51220703125 disk.sda.in‚ 106.21484375 disk.sda.out‚ 1220.97265625 disk.dm-1.out‚ 1220.97265625) disk./.usagePercent‚37.77137925062621& disk.rbd0.in‚1.759218604441423e+13$ memory_percent‚82.82492160797119 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.rbd1.out‚ 45.84765625 disk.rbd0.out‚0$ disk.dm-0.out‚141007.16650390625 disk.sda1.in‚0 network.recv‚ 4.379022e+06" proc.memory.rssMB‚ 2802.8984375$ disk./.usageGB‚562.8208465576172 disk.drbd2165.out‚0' disk.nvme0n1.out‚141007.16650390625* proc.memory.percent‚17.107534408569336 cpu‚25.58631865381787 proc.cpu.threads‚17 network.sent‚ 2.0364734e+07 disk.sda3.in‚ 106.21484375) disk.nvme0c0n1.out‚141007.16650390625) proc.memory.availableMB‚ 2813.96484375 F›²4™+/ _wandbruntime‚17584  _runtime‚17584ô^³¼Éš+à _step‚870 & lr-AdamW/pg1‚0.0003238161176353292 & lr-AdamW/pg2‚0.0003238161176353292 trainer/global_step‚10599 % _timestamp‚1.7843073998288696e+09 _runtime‚17584.047174572,‹®¯Î›+È & lr-AdamW/pg2‚0.0003238161176353292 trainer/global_step‚10599 % _timestamp‚1.7843073998288696e+09 _runtime‚17584.047174572 _step‚870 & lr-AdamW/pg1‚0.0003238161176353292æPw¼æ4œ+/ _wandbruntime‚17584  _runtime‚17584­=+¿(+P A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚91799269 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 % _timestamp‚1.7843073998307989e+09 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9970180347392 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚57240297 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚3046422732562432 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9971525736960 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚3045106994745344 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚91799269 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3046422758735872 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚3046422758735872 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9953120175810 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9970180347392 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚90420121 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 182219396 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚3045106969317376 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12864270951936 trainer/global_step‚10599 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3056392912909824 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3046422732562432 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚3055060089493186 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36633600 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3055061459521476 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 _runtime‚17584.048159401 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 Í (ž+†P eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚822798959276032 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚30048158 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚822798591184896 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9954464776132 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚3056392912909824 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚272 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚835663266861568 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚835662862136832 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12864307585536 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _step‚871 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3056392912909824 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3046422732562432 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚3055060089493186 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36633600 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3055061459521476 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 _runtime‚17584.048159401 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚91799269 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 % _timestamp‚1.7843073998307989e+09 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9970180347392 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚57240297 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚3046422732562432 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9971525736960 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚3045106994745344 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚91799269 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3046422758735872 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚3046422758735872 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9953120175810 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9970180347392 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚90420121 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 182219396 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚3045106969317376 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12864270951936 trainer/global_step‚10599 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9954043642056 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚822876077592576 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚261 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404717568 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 182226101 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚90428333 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚27194410 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 % _timestamp‚1.7843074012195432e+09 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚57245357 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚91797774 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9972450779648 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9971105382912 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12865455992320 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3056673210245632 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9971105382912 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3045384851095552 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚91797774 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚3055338894737608 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚835741938302464 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36626432 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _runtime‚17585.437106561 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 182226101 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 trainer/global_step‚10599 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚835741533584896 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9972450779648 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚90428327 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12865492618752 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 182236094 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 182236094 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚30051208 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚3046700759465984 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 p\ãÉ¡+N PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚261 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 TFDeviceStatsMonitor.on_train_bat!¸ÏsM$ch_end/inactive_split_bytes.all.current‚ 404717568 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 182226101 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚90428333 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚27194410 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 % _timestamp‚1.7843074012195432e+09 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚57245357 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚91797774 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9972450779648 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9971105382912 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12865455992320 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3056673210245632 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9971105382912 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3045384851095552 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚91797774 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚3055338894737608 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚835741938302464 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36626432 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _runtime‚17585.437106561 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 182226101 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 trainer/global_step‚10599 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚835741533584896 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9972450779648 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚90428327 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12865492618752 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 182236094 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 182236094 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚30051208 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚3046700759465984 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3055340264766010 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 _step‚872 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚90428327 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3046700733292544 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚3046700733292544 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9954043642056 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚822876077592576 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0è!f*4¢+/ _wandbruntime‚17585  _runtime‚17585ºl̳£+‹ 3loss/SRqqt2/assignment_loss‚0.25720171045396173 /loss/FRt2/detection_loss‚1.3116966637136602 1loss/SRqqt1/detection_loss‚0.6441245780250017 1loss/SRqqt2/detection_loss‚0.5477368880084557 trainer/global_step‚10599 %loss/total_loss‚6.129454327388931 epoch‚26 _runtime‚17585.520694753 0loss/FRt1/assignment_loss‚1.1773279155845995 1loss/FRt2/assignment_loss‚0.36238084313789387 4loss/FBt1/assignment_loss‚1.8717156763971488e-05 _step‚873 2loss/SRqqt1/assignment_loss‚0.3712130061843823 2loss/FBt2/assignment_loss‚0.020825968341885853 0loss/FRt1/detection_loss‚0.43914360623954996 /loss/FBt1/detection_loss‚0.7302511586807668 /loss/FBt2/detection_loss‚0.2675332718620097 % _timestamp‚1.7843074013037498e+09Á ¤+ %loss/total_loss‚6.129454327388931 epoch‚26 _runtime‚17585.520694753 0loss/FRt1/assignment_loss‚1.1773279155845995 1loss/FRt2/assignment_loss‚0.36238084313789387 4loss/FBt1/assignment_loss‚1.8717156763971488e-05 _step‚873 2loss/SRqqt1/assignment_loss‚0.3712130061843823 2loss/FBt2/assignment_loss‚0.020825968341885853 0loss/FRt1/detection_loss‚0.43914360623954996 /loss/FBt1/detection_loss‚0.7302511586807668 /loss/FBt2/detection_loss‚0.2675332718620097 % _timestamp‚1.7843074013037498e+09 3loss/SRqqt2/assignment_loss‚0.25720171045396173 /loss/FRt2/detection_loss‚1.3116966637136602 1loss/SRqqt1/detection_loss‚0.6441245780250017 1loss/SRqqt2/detection_loss‚0.5477368880084557 trainer/global_step‚10599éQ ö韥+‚(:” Ò½éÒ€šö÷ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚284.382' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚56.876400000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ñÔt…:¦+‚(:¯ Ò½éÒ¼öÔù disk.dm-1.in‚ 106.21484375 disk.rbd3.in‚0 disk.rbd1.out‚ 45.92578125& disk.rbd0.in‚1.759218604441423e+13 disk.rbd0.out‚0 disk.sda1.out‚0 proc.cpu.threads‚17 disk.sda.out‚ 1221.15625 disk.sda3.in‚ 106.21484375 disk.sda3.out‚ 1221.15625' proc.memory.availableMB‚ 2813.921875 network.sent‚ 2.0369437e+07 network.recv‚ 4.382423e+06$ disk.nvme0c0n1.out‚ 141020.265625 disk.sda2.in‚0" proc.memory.rssMB‚ 2802.8984375 cpu‚25.58619364189002$ disk.nvme0c0n1.in‚41942.60546875" disk.nvme0n1.out‚ 141020.265625 disk.dm-1.out‚ 1221.15625 disk.dm-0.in‚41942.60546875 disk.rbd1.in‚ 0.51220703125# memory_percent‚82.8251838684082* proc.memory.percent‚17.107534408569336$ disk./.usageGB‚562.8255615234375" disk.nvme0n1.in‚41942.60546875 disk.dm-0.out‚ 141020.265625 disk.sda1.in‚0 disk.drbd2165.out‚0 disk.rbd3.out‚ 0.22412109375 disk.drbd2165.in‚0) disk./.usagePercent‚37.77169567593854 disk.sda2.out‚0 disk.sda.in‚ 106.21484375°OÁ¼Ÿ§+‚(:” á½éÒݨæö gpu.0.gpu‚100.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚339.413' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚67.88260000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0kP¨+‚(:Å á½éÒÀØù disk.sda1.in‚0$ disk./.usageGB‚562.8180618286133 disk.dm-1.in‚ 106.21484375 disk.rbd3.out‚ 0.22412109375" proc.memory.rssMB‚ 2802.8984375# disk.dm-0.out‚141031.9521484375 disk.sda3.in‚ 106.21484375 disk.drbd2165.in‚0 disk.rbd0.out‚0" disk.nvme0n1.in‚41942.80859375* proc.memory.percent‚17.107534408569336 cpu‚25.586176997362884 disk.rbd3.in‚0( disk.nvme0c0n1.out‚141031.9521484375 disk.sda2.out‚0 disk.rbd1.in‚ 0.51220703125$ memory_percent‚82.82520771026611 disk.sda.in‚ 106.21484375 disk.dm-0.in‚41942.80859375 disk.drbd2165.out‚0 disk.sda2.in‚0 proc.cpu.threads‚17 disk.dm-1.out‚ 1221.5078125 disk.sda1.out‚0) proc.memory.availableMB‚ 2813.91796875 network.recv‚ 4.387103e+06) disk./.usagePercent‚37.77119236512622& disk.rbd0.in‚1.759218604441423e+13 disk.sda3.out‚ 1221.5078125$ disk.nvme0c0n1.in‚41942.80859375& disk.nvme0n1.out‚141031.9521484375 network.sent‚ 2.0433202e+07 disk.sda.out‚ 1221.5078125 disk.rbd1.out‚ 45.92578125ý|jꔩ+‚(:‰ ð½éÒç±÷ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚293.271' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚58.6542 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0©©äbUª+‚(:Ê ð½éÒÞ„ùù" proc.memory.rssMB‚ 2802.8984375 cpu‚25.586145457688687 network.sent‚ 2.043855e+07$ disk./.usageGB‚562.8276977539062 disk.sda3.in‚ 106.21484375 disk.rbd3.out‚ 0.22412109375$ disk.nvme0c0n1.in‚41943.23046875 proc.cpu.threads‚17 disk.sda1.in‚0 disk.rbd3.in‚0) disk.nvme0c0n1.out‚141039.91943359375 disk.sda2.out‚0 disk.dm-1.out‚ 1221.58203125& disk.rbd0.in‚1.759218604441423e+13 disk.dm-0.in‚41943.23046875( proc.memory.availableMB‚ 2813.9140625 disk.sda1.out‚0 disk.sda3.out‚ 1221.58203125 disk.drbd2165.in‚0" disk.nvme0n1.in‚41943.23046875' disk.nvme0n1.out‚141039.91943359375* proc.memory.percent‚17.107534408569336 network.recv‚ 4.390442e+06* disk./.usagePercent‚37.771839040157715 disk.sda2.in‚0$ memory_percent‚82.82523155212402 disk.rbd1.in‚ 0.51220703125 disk.rbd1.out‚ 45.98828125$ disk.dm-0.out‚141039.91943359375 disk.drbd2165.out‚0 disk.sda.out‚ 1221.58203125 disk.rbd0.out‚0 disk.sda.in‚ 106.21484375 disk.dm-1.in‚ 106.21484375h1"Ÿœ«+‚(:‘ ÿ½éÒù¹þö gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚333.618' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚66.72359999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0$k–×U¬+‚(:Ê ÿ½éÒÞÎù network.sent‚ 2.0443637e+07$ disk./.usageGB‚562.8469085693359) disk.nvme0c0n1.out‚141054.20458984375 disk.rbd3.out‚ 0.22412109375 disk.sda2.out‚0$ disk.dm-0.out‚141054.20458984375* proc.memory.percent‚17.107534408569336 cpu‚25.586101572090417 network.recv‚ 4.393959e+06" disk.nvme0n1.in‚42062.05078125 disk.sda.out‚ 1222.11328125 disk.dm-1.out‚ 1222.11328125 disk.sda1.in‚0 disk.sda3.in‚ 106.21484375 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.sda2.in‚0 disk.dm-1.in‚ 106.21484375 disk.rbd0.out‚0$ memory_percent‚82.82523155212402" proc.memory.rssMB‚ 2802.8984375 disk.rbd3.in‚0 disk.rbd1.in‚ 0.51220703125 disk.sda3.out‚ 1222.11328125$ disk.nvme0c0n1.in‚42062.05078125' disk.nvme0n1.out‚141054.20458984375 disk.rbd1.out‚ 45.98828125 disk.sda1.out‚0( proc.memory.availableMB‚ 2813.9140625) disk./.usagePercent‚37.77312829410015 disk.sda.in‚ 106.21484375& disk.rbd0.in‚1.759218604441423e+13 disk.dm-0.in‚42062.05078125 proc.cpu.threads‚17ú(ž­+‚(:“ Ž¾éÒè¨èö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚275.487' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚55.09740000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0kÄŽ7K®+‚(:À Ž¾éÒîæÍù& disk.rbd0.in‚1.759218604441423e+13 disk.sda3.in‚ 106.21484375) disk./.usagePercent‚37.77358654758643$ disk.nvme0c0n1.in‚42062.25390625 disk.rbd1.in‚ 0.51220703125 disk.sda.out‚ 1222.28125 disk.dm-0.in‚42062.25390625$ disk.dm-0.out‚141061.72216796875 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.rbd3.in‚0* proc.memory.percent‚17.107534408569336' disk.nvme0n1.out‚141061.72216796875 disk.sda2.in‚0 disk.rbd1.out‚ 46.05078125" disk.nvme0n1.in‚42062.25390625 disk.sda2.out‚0 disk.sda.in‚ 106.21484375 cpu‚25.58605682580501 proc.cpu.threads‚17 disk.sda1.out‚0 disk.sda3.out‚ 1222.28125 disk.rbd3.out‚ 0.22412109375$ memory_percent‚82.82675743103027" proc.memory.rssMB‚ 2802.8984375$ disk./.usageGB‚562.8537368774414 disk.rbd0.out‚0 network.sent‚ 2.0448743e+07 network.recv‚ 4.397094e+06 disk.dm-1.out‚ 1222.28125 disk.sda1.in‚0( proc.memory.availableMB‚ 2813.6640625) disk.nvme0c0n1.out‚141061.72216796875 disk.dm-1.in‚ 106.214843758¢ƒ‹4¯+/ _wandbruntime‚17663  _runtime‚17663¸»É°+à _step‚874 & lr-AdamW/pg1‚0.0003210501957461229 & lr-AdamW/pg2‚0.0003210501957461229 trainer/global_step‚10649 % _timestamp‚1.7843074794098773e+09 _runtime‚17663.630258661ØDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚92225021 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚57504000 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚3059001083654144 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚10649 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3069001785543384 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10000701889240 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚839596391716864 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚27319022 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚3070339265346048 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚90830427 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10017841667584 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10016496278016 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 183045469 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 183055448 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚271 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚27319017 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚92225021 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚826672911568896 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚839596794344448 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3060322795241472 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3070340636909056 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚90830427 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9999357288918 _runtime‚17663.63132244 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3070339265346048 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚30185249 % _timestamp‚1.7843074794140148e+09 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚266 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12923480147968 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚3060322769068032 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12923514684416 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚30184983 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973A™„™7 ´+ŠP OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚3059001058226176 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 _step‚875 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3070339265346048 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚30185249 % _timestamp‚1.7843074794140148e+09 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚266 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_(%SûÙbytes.small_pool.freed‚12923480147968 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚3060322769068032 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12923514684416 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚30184983 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚90830421 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10017841667584 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚92225021 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚57504000 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚3059001083654144 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚10649 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3069001785543384 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10000701889240 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚839596391716864 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚27319022 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚3070339265346048 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚90830427 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10017841667584 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10016496278016 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 183045469 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 183055448 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚271 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚27319017 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚92225021 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚826672911568896 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚839596794344448 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3060322795241472 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3070340636909056 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚90830427 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9999357288918 _runtime‚17663.63132244 JDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚27321560 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10018766710272 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚3060600769798144 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚90838633 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3060600769798144 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3069280590787918 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚265 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12924703878144 % _timestamp‚1.7843074808021774e+09 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚839675469945856 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3059278940004352 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3059278965432320 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚92233513 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚57509280 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10000280755164 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚10649 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚92223526 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404717568 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚839675065228288 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚3069279220759516 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚90838633 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3070619562681856 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚57509550 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 _runtime‚17665.01955689 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3060600795971584 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10017421313536 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚30187985 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3070618191111680 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚27321565 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 183072146 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3069280590787918 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚265 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12924703878144 % _timestamp‚1.7843074808021774e+09 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚839675469945856 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3059278940004352 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3059278965432320 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚92233513 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚57509280 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10000280755164 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚10649 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚92223526 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404717568 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚839675065228288 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚3069279220759516 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚90838633 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3070619562681856 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚57509550 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 _runtime‚17665.01955689 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3060600795971584 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10017421313536 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚30187985 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3070618191111680 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚27321565 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 183072146 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚27321560 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10018766710272 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚3060600769798144 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚90838633 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3060600769798144 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 QDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚91240727 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚830547231952896 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10046939002348 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10064157598208 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10045594402026 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3074222805573632 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚843530121549824 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3082942111565292 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚3072895147134976 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚843530524177408 _runtime‚17743.370235969 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚830547600044032 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 _step‚879 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 183891500 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚253 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚3084285617782272 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 183881521 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚3074222831747072 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10062812208640 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10064157598208 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3074222831747072 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3084286989345280 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚92640800 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚27446167 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚30320901 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12982924133376 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚91240721 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3084286989345280 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚92650773 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚57767321 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚3082940741537002 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3084285617782272 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚30321149 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10062812208640 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚57767068 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚91240721 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12982889596928 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚3072895172562944 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 % _timestamp‚1.7843075591529055e+09 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚3074222805573632 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚91240727 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 183891500 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚253 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚3084285617782272 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 183881521 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚3074222831747072 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10062812208640 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10064157598208 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3074222831747072 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3084286989345280 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚92640800 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚27446167 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚30320901 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12982924133376 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚91240721 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3084286989345280 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚92650773 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚57767321 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚3082940741537002 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3084285617782272 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚30321149 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10062812208640 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚57767068 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚91240721 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12982889596928 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚3072895172562944 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 % _timestamp‚1.7843075591529055e+09 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚3074222805573632 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚91240727 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚91240727 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚830547231952896 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10046939002348 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10064157598208 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10045594402026 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0ïö\ö4Ë+/ _wandbruntime‚17744  _runtime‚17744ØO Ì+N L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 183908198 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚843608810864640 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚236 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7843075605407503e+09 _runtime‚17744.75814421 _step‚880 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3084565915118080 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚57772613 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10047862468706 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3083220916809826 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚91248933 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10065082640896 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 183898205 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10046517868272 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚92649278 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3074500832477184 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚92649278 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3073173054341120 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚241 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚30323662 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚830625086451712 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 183908198 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3074500806303744 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 B‚DeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10063737244160 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 trainer/global_step‚10699 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚92659265 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10063737244160 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚57772372 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚91248927 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3084565915118080 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚57772613 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10047862468706 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3083220916809826 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚91248933 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10065082640896 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 183898205 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10046517868272 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚92649278 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3074500832477184 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚92649278 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3073173054341120 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚241 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚30323662 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚830625086451712 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 183908198 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3074500806303744 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚57772372 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚91248927 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 183908198 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚843608810864640 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚236 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7843075605407503e+09 _runtime‚17744.75814421 _step‚880 FDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3098231970218496 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚30457406 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚27573322 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13042596954624 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚27573317 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚3088122842079232 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚254 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚834421552336896 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚847464149291520 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10091831515134 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚58030728 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚91651027 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚3086789261471744 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10110473528832 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚58030474 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 % _timestamp‚1.7843076390514805e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3088122842079232 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚3088122868252672 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚30457157 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 184717573 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚91651027 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚249 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 trainer/global_step‚10749 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 JDDeviceStatXø‚ŒsMonitor.on_train_batch_start/oversize_allocations.current‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚3086789236043776 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175ÒU(â+ŒP D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 % _timestamp‚1.7843076390514805e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3088122842079232 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚3088122868252672 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚30457157 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 184717573 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚91651027 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚249 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 trainer/global_step‚10749 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚3086789236043776 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚3098231970218496 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10109128139264 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3098231970218496 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚30457406 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚27573322 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13042596954624 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚27573317 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚3088122842079232 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚254 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚834421552336896 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚847464149291520 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10091831515134 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚58030728 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚91651027 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚3086789261471744 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10110473528832 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚58030474 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504ón½«4ã+/ _wandbruntime‚17824  _runtime‚17824é ÉÝ'ä+N BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚58035713 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚834499406835712 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3088400868982784 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10110053174784 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _runtime‚17824.656055849 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚91659227 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3088400842809344 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10111398571520 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚847542830209536 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 184744250 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚58035966 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚884 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10111398571520 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3098512267554304 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚3088400842809344 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 184734257 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚30459853 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚3097159872803332 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10110053174784 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3098512267554304 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3087067117821952 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚93075030 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚93075030 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 184744250 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚27575860 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚91659233 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175Ö¤2'å+•N UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 184744250 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚58035966 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚884 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10111398571520 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3098512267554304 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚3088400842809344 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 184734257 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚30459853 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚3097159872803332 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10110053174784 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3098512267554304 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3087067117821952 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚93075030 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚93075030 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 184744250 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚27575860 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚91659233 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚58035713 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚834499406835712 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3088400868982784 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10110053174784 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _runtime‚17824.656055849 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚91659227 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3088400842809344 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10111398571520 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚847542830209536 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0ô3ŒŒÙ4æ+/ _wandbruntime‚17824  _runtime‚17824É&çç+‹ 2loss/SRqqt2/assignment_loss‚0.3038534819383741 3loss/FBt2/assignment_loss‚0.0031612511446922342 /loss/FRt2/detection_loss‚1.4216537657922181 1loss/SRqqt1/detection_loss‚0.6466244078969504 0loss/FBt2/detection_loss‚0.45642171971667267 %loss/total_loss‚6.119672646024761 epoch‚26 trainer/global_step‚10749 0loss/FRt1/assignment_loss‚1.1886619478052871 4loss/FBt1/assignment_loss‚2.6668473993677357e-06 _runtime‚17824.741155178 _step‚885 0loss/FRt2/assignment_loss‚0.3520083364767906 3loss/SRqqt1/assignment_loss‚0.28264564726818564 0loss/FBt1/detection_loss‚0.46919431839528125 $ _timestamp‚1.784307640524241e+09 /loss/FRt1/detection_loss‚0.4455342068629438 1loss/SRqqt2/detection_loss‚0.5499108958799668¡¨ÒÆè+ 2loss/SRqqt2/assignment_loss‚0.3038534819383741 3loss/FBt2/assignment_loss‚0.0031612511446922342 /loss/FRt2/detection_loss‚1.4216537657922181 1loss/SRqqt1/detection_loss‚0.6466244078969504 0loss/FBt2/detection_loss‚0.45642171971667267 %loss/total_loss‚6.119672646024761 epoch‚26 trainer/global_step‚10749 0loss/FRt1/assignment_loss‚1.1886619478052871 4loss/FBt1/assignment_loss‚2.6668473993677357e-06 _runtime‚17824.741155178 _step‚885 0loss/FRt2/assignment_loss‚0.3520083364767906 3loss/SRqqt1/assignment_loss‚0.28264564726818564 0loss/FBt1/detection_loss‚0.46919431839528125 $ _timestamp‚1.784307640524241e+09 /loss/FRt1/detection_loss‚0.4455342068629438 1loss/SRqqt2/detection_loss‚0.5499108958799668õeÓø›pé+‚(:‰ ¿éÒÅÐö gpu.0.gpu‚90.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚57.761306762695305ÙC&$* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚290.327' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚58.0654 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0m:ÊAê+‚(:¶ ¿éÒ‡À¸ù disk.sda3.in‚ 106.21875 disk.sda2.in‚0 disk.sda.in‚ 106.21875 network.recv‚ 4.442104e+06 disk.nvme0c0n1.in‚ 42402.1875) disk.nvme0c0n1.out‚141200.17236328125& disk.rbd0.in‚1.759218604441423e+13 disk.rbd0.out‚0* proc.memory.percent‚17.107534408569336 cpu‚25.585833906122357 disk.nvme0n1.in‚ 42402.1875 disk.sda2.out‚0 disk.rbd1.in‚ 0.52001953125 disk.dm-0.in‚ 42402.1875 proc.cpu.threads‚17 disk.drbd2165.out‚0 disk.rbd3.in‚0) disk./.usagePercent‚37.77782296026304 disk.sda1.out‚0 disk.sda3.out‚ 1226.01171875 disk.rbd3.out‚ 0.22412109375' disk.nvme0n1.out‚141200.17236328125) proc.memory.availableMB‚ 2814.74609375 disk.drbd2165.in‚0 disk.dm-1.in‚ 106.21875" proc.memory.rssMB‚ 2802.8984375 network.sent‚ 2.0626663e+07 disk.sda1.in‚0 disk.dm-1.out‚ 1226.01171875$ memory_percent‚82.82015323638916 disk.rbd1.out‚ 46.724609375 disk.sda.out‚ 1226.01171875$ disk.dm-0.out‚141200.17236328125# disk./.usageGB‚562.916862487793Øu˜é”ë+‚(:‰ Ñ¿éÒÇÂùö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚280.902' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚56.1804 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ò(»&Cì+‚(:¸ Ñ¿éÒ€£ûø disk.sda3.in‚ 106.21875" disk.nvme0c0n1.in‚ 42402.390625 disk.rbd1.in‚ 0.52001953125* disk./.usagePercent‚37.778102008475365 disk.sda.out‚ 1226.33203125 disk.dm-1.out‚ 1226.33203125 disk.nvme0n1.in‚ 42402.390625 disk.rbd1.out‚ 46.724609375* proc.memory.percent‚17.107534408569336 disk.rbd0.out‚0 disk.drbd2165.in‚0 disk.rbd3.out‚ 0.22412109375( disk.nvme0c0n1.out‚141217.3486328125 disk.sda1.out‚0 disk.sda3.out‚ 1226.33203125 disk.sda2.out‚0$ memory_percent‚82.81748294830322" proc.memory.rssMB‚ 2802.8984375 disk.drbd2165.out‚0 disk.dm-0.in‚ 42402.390625 cpu‚25.58577728603881 network.recv‚ 4.4466e+06 disk.sda.in‚ 106.21875# disk.dm-0.out‚141217.3486328125 disk.rbd3.in‚0) proc.memory.availableMB‚ 2815.18359375 proc.cpu.threads‚17 network.sent‚ 2.0690043e+07$ disk./.usageGB‚562.9210205078125 disk.dm-1.in‚ 106.21875& disk.rbd0.in‚1.759218604441423e+13 disk.sda1.in‚0& disk.nvme0n1.out‚141217.3486328125 disk.sda2.in‚0TV¤”í+‚(:‰ à¿éÒã­ãö gpu.0.gpu‚86.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚332.413' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚66.4826 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 3I4Cî+‚(:¸ à¿éÒ“……ù disk.rbd0.out‚0 disk.sda3.out‚ 1227.8828125 disk.nvme0n1.in‚ 42402.671875& disk.nvme0n1.out‚141231.8232421875 disk.sda2.out‚0* proc.memory.percent‚17.107534408569336 disk.sda.in‚ 106.21875 disk.drbd2165.in‚0" proc.memory.rssMB‚ 2802.8984375 disk.dm-1.in‚ 106.21875 disk.dm-1.out‚ 1227.8828125( disk.nvme0c0n1.out‚141231.8232421875 disk.rbd1.in‚ 0.52001953125) proc.memory.availableMB‚ 2815.16015625 network.sent‚ 2.0694988e+07$ disk./.usageGB‚562.9335708618164 disk.rbd3.in‚0* disk./.usagePercent‚37.778944273263015# disk.dm-0.out‚141231.8232421875 disk.sda1.in‚0 disk.drbd2165.out‚0 disk.sda2.in‚0$ memory_percent‚82.81762599945068 proc.cpu.threads‚17& disk.rbd0.in‚1.759218604441423e+13 disk.sda1.out‚0 disk.sda3.in‚ 106.21875 network.recv‚ 4.449078e+06 disk.dm-0.in‚ 42402.671875 disk.rbd3.out‚ 0.22412109375 cpu‚25.585802971414367" disk.nvme0c0n1.in‚ 42402.671875 disk.rbd1.out‚ 46.802734375 disk.sda.out‚ 1227.8828125Ê!t”žï+‚(:“ ï¿éÒ‘»÷ö gpu.0.gpu‚90.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚275.782' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚55.15639999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0@4rêAð+‚(:¶ ï¿éÒâÜ€ù disk.rbd3.in‚0$ disk.nvme0n1.out‚141254.90234375 disk.sda2.in‚0 disk.rbd1.out‚ 46.802734375 network.sent‚ 2.069932e+07 disk.sda1.in‚0 disk.rbd1.in‚ 0.52001953125 disk.sda.out‚ 1229.12890625 disk.dm-1.in‚ 106.21875 disk.sda1.out‚0 disk.drbd2165.out‚0# disk.nvme0c0n1.in‚ 42508.5859375$ disk./.usageGB‚562.9387702941895 disk.dm-0.in‚ 42508.5859375 disk.rbd3.out‚ 0.22412109375! disk.nvme0n1.in‚ 42508.5859375 disk.sda2.out‚0 cpu‚25.58575907900292* disk./.usagePercent‚37.779293211532185 disk.sda3.out‚ 1229.12890625* proc.memory.percent‚17.107534408569336 network.recv‚ 4.452387e+06 disk.sda3.in‚ 106.21875) proc.memory.availableMB‚ 2815.30078125 proc.cpu.threads‚17& disk.rbd0.in‚1.759218604441423e+13 disk.rbd0.out‚0! disk.dm-0.out‚141254.90234375 disk.drbd2165.in‚0& disk.nvme0c0n1.out‚141254.90234375 disk.sda.in‚ 106.21875 disk.dm-1.out‚ 1229.12890625$ memory_percent‚82.81676769256592" proc.memory.rssMB‚ 2802.8984375W…ìŸñ+‚(:” þ¿éÒìÄëö gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚39.0 gpu.0.powerWatts‚110.286' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.057199999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ì´âˆ=ò+‚(:² þ¿éÒŒðÐù disk.dm-0.out‚141262.5078125 disk.sda.in‚ 106.21875 disk.rbd0.out‚0# disk.nvme0c0n1.in‚ 42508.7890625 network.recv‚ 4.455703e+06) disk./.usagePercent‚37.78342005298414$ disk./.usageGB‚563.0002632141113 disk.drbd2165.in‚0 disk.dm-0.in‚ 42508.7890625 disk.sda3.out‚ 1229.57421875 disk.drbd2165.out‚0* proc.memory.percent‚17.107534408569336 cpu‚25.585713323620094 disk.dm-1.in‚ 106.21875 disk.dm-1.out‚ 1229.57421875& disk.rbd0.in‚1.759218604441423e+13 disk.rbd3.in‚0# disk.nvme0n1.out‚141262.5078125$ memory_percent‚82.81755447387695" proc.memory.rssMB‚ 2802.8984375 proc.cpu.threads‚17 network.sent‚ 2.0704504e+07 disk.sda3.in‚ 106.21875! disk.nvme0n1.in‚ 42508.7890625 disk.rbd1.in‚ 0.52001953125 disk.rbd1.out‚ 46.865234375' proc.memory.availableMB‚ 2815.171875 disk.sda2.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda.out‚ 1229.57421875% disk.nvme0c0n1.out‚141262.5078125 disk.rbd3.out‚ 0.22412109375 disk.sda2.out‚0‡7Ö4ó+/ _wandbruntime‚17902  _runtime‚17902õƒÊô+Ä _step‚886 ' lr-AdamW/pg1‚0.00031276272582889136 ' lr-AdamW/pg2‚0.00031276272582889136 trainer/global_step‚10799 $ _timestamp‚1.784307718727798e+09 _runtime‚17902.945158572ý ©vÏõ+É ' lr-AdamW/pg1‚0.00031276272582889136 ' lr-AdamW/pg2‚0.00031276272582889136 trainer/global_step‚10799 $ _timestamp‚1.784307718727798e+09 _runtime‚17902.945158572 _step‚886öÚñU4ö+/ _wandbruntime‚17902  _runtime‚17902 õH; (÷+†P M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚27700472 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚851397959861760 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13102123774464 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3112179694217728 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚851398364586496 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚93492304 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13102087140864 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚92061327 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚92061327 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10139413228564 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10155444069888 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚3100683324952576 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10156789459456 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚3102022904758272 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36633600 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚887 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚93502277 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚271 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 _runtime‚17902.94640763 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3112178322654720 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10155444069888 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚838296240812032 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10156789459456 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚30593477 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚3112178322654720 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 185563604 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚93502277 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚58293944 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 % _timestamp‚1.7843077187290645e+09 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 185553625 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3102022878584832 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10138068628242 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚3110821393580818 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚3100683350380544 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚92061321 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚58294220 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3102022904758272 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3112179694217728 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚276 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3110822763609108 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0õ«ÛÑ(ø+‹P PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚92061327 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚92061327 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10139413228564 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10155444069888 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚3100683324952576 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10156789459456 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚3102022904758272 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36633600 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚887 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚93502277 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚271 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 _runtime‚17902.94640763 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3112178322654720 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10155444069888 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚838296240812032 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10156789459456 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚30593477 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚3112178322654720 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 185563604 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚93502277 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚58293944 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 % _timestamp‚1.7843077187290645e+09 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 185553625 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3102022878584832 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10138068628242 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚3110821393580818 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚3100683350380544 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚92061321 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚58294220 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3102022904758272 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3112179694217728 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚276 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3110822763609108 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚27700472 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚851397959861760 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13102123774464 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3112179694217728 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚851398364586496 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚93492304 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13102087140864 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0÷Îõƶ4ù+/ _wandbruntime‚17904  _runtime‚17904Uybú+‘N HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚93510769 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10156369105408 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚259 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚851476624689664 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3100961206730752 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 185580302 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚92069527 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10138992094488 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 185570309 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3112458619990528 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10157714502144 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚93500782 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚58299116 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10140336694922 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 YFDeviceSñÞ?†µtatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10156369105408 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3112458619990528 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚30596365 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚92069533 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 % _timestamp‚1.7843077201149857e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 185580302 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 185570309 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚838373359128576 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚3102300879314944 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3102300905488384 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚3111100198825240 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3111101568853642 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404717568 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚27703010 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13103302187520 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚92069527 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 _step‚888 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚93500782 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚92069533>jè'û+–N M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 185570309 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚838373359128576 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚3102300879314944 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3102300905488384 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚3111100198825240 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3111101568853642 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404717568 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚27703010 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13103302187520 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚92069527 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 _step‚888 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚93500782 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚92069533 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚93510769 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10156369105408 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚259 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚851476624689664 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3100961206730752 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 185580302 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚92069527 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10138992094488 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 185570309 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3112458619990528 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10157714502144 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚93500782 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚58299116 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10140336694922 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10156369105408 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3112458619990528 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚30596365 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚92069533 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 % _timestamp‚1.7843077201149857e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 185580302 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640ø“Äáâ4ü+/ _wandbruntime‚17904  _runtime‚17904çJÁ3ý+Œ 0loss/FRt1/assignment_loss‚1.2930540957441732 3loss/FBt2/assignment_loss‚0.0002974820457069207 0loss/SRqqt2/detection_loss‚0.516762136166825 /loss/FBt2/detection_loss‚0.3016169800471818 %loss/total_loss‚6.013783016751266 epoch‚26 % _timestamp‚1.7843077202002873e+09 _runtime‚17904.417228089 1loss/FRt2/assignment_loss‚0.35581444353578195 /loss/FRt1/detection_loss‚0.4620854754322682 _step‚889 3loss/SRqqt2/assignment_loss‚0.32544294718473393 /loss/FRt2/detection_loss‚1.3147998435799317 1loss/SRqqt1/detection_loss‚0.7359615753743045 3loss/SRqqt1/assignment_loss‚0.30672975637998284 4loss/FBt1/assignment_loss‚2.4604001421988795e-05 0loss/FBt1/detection_loss‚0.40119367725895466 trainer/global_step‚10799­Øþ+‘ 0loss/SRqqt2/detection_loss‚0.516762136166825 /loss/FBt2/detection_loss‚0.3016169800471818 %loss/total_loss‚6.013783016751266 epoch‚26 % _timestamp‚1.7843077202002873e+09 _runtime‚17904.417228089 1loss/FRt2/assignment_loss‚0.35581444353578195 /loss/FRt1/detection_loss‚0.4620854754322682 _step‚889 3loss/SRqqt2/assignment_loss‚0.32544294718473393 /loss/FRt2/detection_loss‚1.3147998435799317 1loss/SRqqt1/detection_loss‚0.7359615753743045 3loss/SRqqt1/assignment_loss‚0.30672975637998284 4loss/FBt1/assignment_loss‚2.4604001421988795e-05 0loss/FBt1/detection_loss‚0.40119367725895466 trainer/global_step‚10799 0loss/FRt1/assignment_loss‚1.2930540957441732 3loss/FBt2/assignment_loss‚0.0002974820457069207ùóP’ÿ+‚(:‡ ÀéÒ—œ€÷ gpu.0.gpu‚1.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚37.0 gpu.0.powerWatts‚109.461' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.8922 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0åØEqD€,‚(:¹ ÀéÒ˜žÀú disk.sda.in‚ 106.21875* proc.memory.percent‚17.107534408569336 disk.dm-1.out‚ 1230.109375$ disk.dm-0.out‚141330.15478515625 disk.sda3.in‚ 106.21875# disk.nvme0c0n1.in‚ 42509.3515625 disk.rbd1.out‚ 46.865234375 disk.dm-0.in‚ 42509.3515625 disk.sda2.out‚0& disk.rbd0.in‚1.759218604441423e+13 disk.sda1.out‚0) disk.nvme0c0n1.out‚141330.15478515625 proc.cpu.threads‚17* disk./.usagePercent‚37.780304185284905 disk.sda.out‚ 1230.109375 disk.rbd3.in‚0 disk.rbd1.in‚ 0.52001953125$ memory_percent‚82.82308578491211' proc.memory.availableMB‚ 2814.265625 network.sent‚ 2.0709587e+07 disk.dm-1.in‚ 106.21875 disk.rbd0.out‚0 disk.sda3.out‚ 1230.109375 disk.rbd3.out‚ 0.22412109375" proc.memory.rssMB‚ 2802.8984375 cpu‚25.586336086943962 disk.sda1.in‚0 disk.drbd2165.out‚0' disk.nvme0n1.out‚141330.15478515625 disk.drbd2165.in‚0! disk.nvme0n1.in‚ 42509.3515625 disk.sda2.in‚0 network.recv‚ 4.459195e+06$ disk./.usageGB‚562.9538345336914®ŒB…’,‚(:‡ œÀéÒô†ðö gpu.0.gpu‚1.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚336.329' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚67.2658 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0kr“„G‚,‚(:¼ œÀéÒûÔ¢ù# disk.nvme0c0n1.in‚ 42509.5703125 disk.sda3.in‚ 106.21875 disk.drbd2165.in‚0' disk.nvme0n1.out‚141347.21630859375 disk.rbd1.out‚ 46.943359375" proc.memory.rssMB‚ 2802.8984375& disk.rbd0.in‚1.759218604441423e+13$ disk.dm-0.out‚141347.21630859375) disk.nvme0c0n1.out‚141347.21630859375 network.sent‚ 2.0772803e+07) disk./.usagePercent‚37.78038277959791 disk.dm-1.in‚ 106.21875 disk.dm-1.out‚ 1231.2421875 disk.sda2.in‚0 disk.sda2.out‚0) proc.memory.availableMB‚ 2813.98046875 disk.sda.in‚ 106.21875 disk.drbd2165.out‚0 disk.rbd3.out‚ 0.22412109375 disk.rbd1.in‚ 0.52001953125 cpu‚25.587939052683367 network.recv‚ 4.463922e+06# disk./.usageGB‚562.955005645752 disk.dm-0.in‚ 42509.5703125 proc.cpu.threads‚17 disk.sda1.in‚0 disk.sda3.out‚ 1231.2421875 disk.rbd3.in‚0* proc.memory.percent‚17.107534408569336 disk.sda.out‚ 1231.2421875 disk.rbd0.out‚0 disk.sda1.out‚0! disk.nvme0n1.in‚ 42509.5703125$ memory_percent‚82.82482624053955°:P4ƒ,/ _wandbruntime‚17932  _runtime‚179324Ì›¢/„,Ú{ ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.030235735699534416 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.11342839151620865 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.03461211174726486 particle/accuracy_1_of_5‚1 particle/accuracy_3_of_5‚1 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.6709372401237488 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.19433094561100006 particle/accuracy_5_of_5‚0 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9974743723869324 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.4505263864994049 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 6 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.995943546295166 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.19790281355381012 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9893755316734314 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6832459568977356 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.19790281355381012 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 jet/accuracy_2_of_5‚1 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5189473628997803 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9906229376792908 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 +jet/accuracy_4_of_4‚0.01628045178949833 particle/accuracy_2_of_2‚1 particle/accuracy_1_of_3‚1 +particle/specificity‚0.9064048528671265 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 + Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9375 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.17142921686172485 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.26221776008605957 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.05000000074505806 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6356741189956665 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2954733371734619 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.17049098014831543 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6654788255691528 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7870370149612427 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.1976618766784668 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9134849309921265 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 7"Purity/0FRt1SRqqt0FBt/event_purity‚0.60039222240448 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7252722978591919 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.2222222238779068 +jet/accuracy_3_of_3‚0.06356717646121979 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7389953136444092 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.16306404769420624 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6597391963005066 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 +jet/accuracy_3_of_4‚0.08418873697519302 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 6 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.998015820980072 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9629629850387573 particle/accuracy_3_of_3‚1 (particle/accuracy‚0.8452187776565552 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6294320225715637 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.35497936606407166 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6578702926635742 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 particle/accuracy_1_of_2‚1 particle/accuracy_2_of_3‚1 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.11538461595773697 . Purity/2FRt0SRqqt1FBt/FRt_purity‚ 0.5859375 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9937442541122437 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.04007149115204811 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4771360158920288 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7486168146133423 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.021208828315138817 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5610405802726746 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8189654350280762 8"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.647849977016449 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.43737053871154785 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.15019948780536652 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.2222222238779068 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.028038041666150093 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.13982953131198883 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.011354722082614899 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.19680257141590118 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 *jet/accuracy_1_of_1‚0.3040274381637573 *jet/accuracy_1_of_2‚0.7374007701873779 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9351851344108582 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9929356575012207 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8524335026741028 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6168906092643738 8"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.713630199432373 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 6 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.548538088798523 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.031678080558776855 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7431610226631165 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.35300344228744507 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.26102718710899353 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.032084234058856964 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6272603869438171 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4074002802371979 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.5593531727790833 # _timestamp‚1.78430774839951e+09 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6547045111656189 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9860765337944031 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3499999940395355 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.028038041666150093 *jet/accuracy_2_of_3‚0.5302765965461731 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.12315338850021362 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6022607088088989 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.1790478527545929 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.11538461595773697 _runtime‚17932.617305623 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8536985516548157 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9981684684753418 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7385237812995911 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.2586546838283539 8"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.683556854724884 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 *jet/accuracy_1_of_3‚0.9055900573730469 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6475650072097778 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8804893493652344 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9924525022506714 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6124811172485352 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 6 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.942307710647583 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9629629850387573 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 particle/accuracy_2_of_5‚1 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 * Purity/2FRt*SRqqt2FBt/FRt_purity‚0.875 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.8571428656578064 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9974743723869324 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 jet/accuracy_5_of_5‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.09666667133569717 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 particle/accuracy_1_of_1‚1 +particle/sensitivity‚0.6214943528175354 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9966472387313843 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.15973635017871857 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6981595158576965 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.05000000074505806 particle/accuracy_4_of_5‚1 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 epoch‚26 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.18006941676139832 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2103816419839859 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6291773915290833 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9938808679580688 + Purity/0FRt2SRqqt1FBt/FBt_purity‚0.9375 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7870370149612427 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 8"Purity/1FRt*SRqqt1FBt/event_purity‚0.430826872587204 6 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.734089195728302 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.2084152102470398 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 7"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.60039222240448 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6652838587760925 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 particle/accuracy_2_of_4‚1 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 jet/accuracy_1_of_5‚1 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.1834835559129715 6 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.988095223903656 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9971989393234253 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9852234721183777 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.18393933773040771 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.018350748345255852 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.7514739036560059 particle/accuracy_1_of_4‚1 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7468253970146179 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.471197366714!…‘>47754 /"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.421875 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.42129626870155334 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9904571771621704 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7290438413619995 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.19684553146362305 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 *jet/accuracy_2_of_2‚0.1451265513896942 /particle/accuracy_4_of_4‚0.9410310387611389 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.12962962687015533 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.19030384719371796 6 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.998227596282959 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.20990978181362152 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6635848879814148 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9973021745681763 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9211474657058716 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.12998048961162567 'particle/f_score‚0.6329520344734192 jet/accuracy_4_of_5‚0 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6666008830070496 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.21472792327404022 +"Purity/2FRt*SRqqt2FBt/event_purity‚0.75 _step‚890 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9949456453323364 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9787510633468628 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.7290438413619995 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.12962962687015533 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.20060010254383087 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9988662004470825 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8888888955116272 particle/accuracy_3_of_4‚1 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6819165349006653 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6124411821365356 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.033257726579904556 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9285714030265808 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6596190333366394 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7206282615661621 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 6validation_average_jet_accuracy‚0.3461289703845978 )loss/val_total_loss‚8.408664334706817 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9943705797195435 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7193750143051147 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.5946249961853027 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.03376311436295509 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6308594346046448 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8189654350280762 *jet/accuracy_2_of_4‚0.7450773119926453 /"Purity/2FRt0SRqqt1FBt/event_purity‚0.203125 trainer/global_step‚10799 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.41435185074806213 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5008333325386047 jet/accuracy_3_of_5‚1 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9958122968673706 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5847552418708801 *jet/accuracy_1_of_4‚0.96294295787811285bÝå=…,ß{ *jet/accuracy_1_of_4‚0.9629429578781128 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.030235735699534416 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.11342839151620865 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.03461211174726486 particle/accuracy_1_of_5‚1 particle/accuracy_3_of_5‚1 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.6709372401237488 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.19433094561100006 particle/accuracy_5_of_5‚0 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9974743723869324 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.4505263864994049 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 6 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.995943546295166 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.19790281355381012 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9893755316734314 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6832459568977356 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.19790281355381012 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 jet/accuracy_2_of_5‚1 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5189473628997803 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9906229376792908 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 +jet/accuracy_4_of_4‚0.01628045178949833 particle/accuracy_2_of_2‚1 particle/accuracy_1_of_3‚1 +particle/specificity‚0.9064048528671265 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 + Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9375 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.17142921686172485 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.26221776008605957 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.05000000074505806 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6356741189956665 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2954733371734619 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.17049098014831543 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6654788255691528 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7870370149612427 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.1976618766784668 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9134849309921265 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 7"Purity/0FRt1SRqqt0FBt/event_purity‚0.60039222240448 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7252722978591919 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.2222222238779068 +jet/accuracy_3_of_3‚0.06356717646121979 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7389953136444092 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.16306404769420624 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6597391963005066 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 +jet/accuracy_3_of_4‚0.08418873697519302 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 6 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.998015820980072 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9629629850387573 particle/accuracy_3_of_3‚1 (particle/accuracy‚0.8452187776565552 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6294320225715637 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.35497936606407166 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6578702926635742 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 particle/accuracy_1_of_2‚1 particle/accuracy_2_of_3‚1 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.11538461595773697 . Purity/2FRt0SRqqt1FBt/FRt_purity‚ 0.5859375 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9937442541122437 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.04007149115204811 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4771360158920288 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7486168146133423 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.021208828315138817 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5610405802726746 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8189654350280762 8"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.647849977016449 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.43737053871154785 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.15019948780536652 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.2222222238779068 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.028038041666150093 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.13982953131198883 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.011354722082614899 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.19680257141590118 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 *jet/accuracy_1_of_1‚0.3040274381637573 *jet/accuracy_1_of_2‚0.7374007701873779 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9351851344108582 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9929356575012207 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8524335026741028 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6168906092643738 8"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.713630199432373 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 6 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.548538088798523 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.031678080558776855 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7431610226631165 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.35300344228744507 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.26102718710899353 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.032084234058856964 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6272603869438171 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4074002802371979 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.5593531727790833 # _timestamp‚1.78430774839951e+09 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6547045111656189 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9860765337944031 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3499999940395355 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.028038041666150093 *jet/accuracy_2_of_3‚0.5302765965461731 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.12315338850021362 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6022607088088989 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.1790478527545929 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.11538461595773697 _runtime‚17932.617305623 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8536985516548157 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9981684684753418 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7385237812995911 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.2586546838283539 8"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.683556854724884 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 *jet/accuracy_1_of_3‚0.9055900573730469 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6475650072097778 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8804893493652344 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9924525022506714 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6124811172485352 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 6 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.942307710647583 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9629629850387573 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 particle/accuracy_2_of_5‚1 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 * Purity/2FRt*SRqqt2FBt/FRt_purity‚0.875 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.8571428656578064 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9974743723869324 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 jet/accuracy_5_of_5‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.09666667133569717 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 particle/accuracy_1_of_1‚1 +particle/sensitivity‚0.6214943528175354 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9966472387313843 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.15973635017871857 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6981595158576965 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.05000000074505806 particle/accuracy_4_of_5‚1 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 epoch‚26 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.18006941676139832 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2103816419839859 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6291773915290833 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9938808679580688 + Purity/0FRt2SRqqt1FBt/FBt_purity‚0.9375 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7870370149612427 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 8"Purity/1FRt*SRqqt1FBt/event_purity‚0.430826872587204 6 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.734089195728302 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.2084152102470398 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 7"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.60039222240448 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6652838587760925 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 particle/accuracy_2_of_4‚1 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 jet/accuracy_1_of_5‚1 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.1834835559129715 6 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.988095223903656 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9971989393234253 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9852234721183777 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.18393933773040771 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.018350748345255852 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.7514739036560059 particle/accuracy_1_of_4‚1 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7468253970146179 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.47119736671447754 /"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.421875 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.42129626870155334 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9904571771621704 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7290438413619995 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.19684553146362305 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 *jet/accuracy_2_of_2‚0.1451265513896942 /particle/accuracy_4_of_4‚0.9410310387611389 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.12962962687015533 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.19030384719371796 6 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.998227596282959 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.20990978181362152 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6635848879814148 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9973021745681763 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9211474657058716 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.12998048961162567 'particle/f_score‚0.6329520344734192 jet/accuracy_4_of_5‚0 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6666008830070496 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.21472792327404022 +"Purity/2FRt*SRqqt2FBt/event_purity‚0.75 _step‚890 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9949456453323364 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9787510633468628 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.7290438413619995 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.12962962687015533 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.20060010254383087 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9988662004470825 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8888888955116272 particle/accuracy_3_of_4‚1 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6819165349006653 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6124411821365356 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.033257726579904556 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9285714030265808 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6596190333366394 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7206282615661621 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 6validation_average_jet_accuracy‚0.3461289703845978 )loss/val_total_loss‚8.408664334706817 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9943705797195435 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7193750143051147 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.5946249961853027 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.03376311436295509 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6308594346046448 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8189654350280762 *jet/accuracy_2_of_4‚0.7450773119926453 /"Purity/2FRt0SRqqt1FBt/event_purity‚0.203125 trainer/global_step‚10799 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.41435185074806213 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5008333325386047 jet/accuracy_3_of_5‚1 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9958122968673706 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5847552418708801ú v[蕆,‚(:Š «ÀéÒœŸþö gpu.0.gpu‚100.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚335.937' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚67.1874 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÒhâC‡,‚(:¸ «ÀéÒ–ÿêù) proc.memory.availableMB‚ 2813.60546875* proc.memory.percent‚17.107534408569336 cpu‚25.5887964511939$ disk./.usageGB‚562.9602890014648 disk.sda2.in‚0 disk.sda2.out‚0$ memory_percent‚82.82711505889893" proc.memory.rssMB‚ 2802.8984375 disk.dm-1.in‚ 106.21875 disk.sda1.in‚0 disk.rbd1.out‚ 46.943359375 network.recv‚ 4.467533e+06 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.rbd3.out‚ 0.22412109375 disk.sda.in‚ 106.21875 disk.sda.out‚ 1233.234375 disk.dm-1.out‚ 1233.234375& disk.rbd0.in‚1.759218604441423e+13) disk./.usagePercent‚37.78073735003284$ disk.nvme0c0n1.in‚42639.52734375( disk.nvme0c0n1.out‚141361.9208984375" disk.nvme0n1.in‚42639.52734375 disk.sda1.out‚0 proc.cpu.threads‚17 network.sent‚ 2.0777875e+07& disk.nvme0n1.out‚141361.9208984375 disk.rbd1.in‚ 0.52001953125 disk.sda3.out‚ 1233.234375 disk.dm-0.in‚42639.52734375 disk.sda3.in‚ 106.21875 disk.rbd3.in‚0 disk.rbd0.out‚0# disk.dm-0.out‚141361.9208984375°Þ»”ˆ,‚(:‰ ºÀéÒƸ ÷ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚264.758' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.9516 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0c˜ýÚJ‰,‚(:¿ ºÀéÒµŸù* proc.memory.percent‚17.107534408569336 disk.sda3.in‚ 106.21875 disk.rbd3.out‚ 0.22412109375' disk.nvme0n1.out‚141372.06689453125 network.recv‚ 4.471993e+06& disk.rbd0.in‚1.759218604441423e+13 disk.drbd2165.out‚0 disk.sda2.in‚0 cpu‚25.5887936067321 disk.sda.in‚ 106.21875 disk.rbd0.out‚0 disk.dm-0.in‚42639.94921875$ disk.nvme0c0n1.in‚42639.94921875" disk.nvme0n1.in‚42639.94921875 disk.sda.out‚ 1234.92578125 disk.dm-1.in‚ 106.21875 disk.dm-1.out‚ 1234.92578125 disk.sda1.in‚0$ disk.dm-0.out‚141372.06689453125 disk.rbd3.in‚0 disk.sda2.out‚0$ memory_percent‚82.82594680786133' proc.memory.availableMB‚ 2813.796875" proc.memory.rssMB‚ 2802.8984375 proc.cpu.threads‚17) disk./.usagePercent‚37.78109652860338 disk.sda1.out‚0 disk.rbd1.in‚ 0.52001953125 disk.rbd1.out‚ 47.005859375 network.sent‚ 2.0835682e+07$ disk./.usageGB‚562.9656410217285 disk.sda3.out‚ 1234.92578125 disk.drbd2165.in‚0) disk.nvme0c0n1.out‚141372.06689453125 Q»’Š,‚(:‡ ÉÀéÒ£«§÷ gpu.0.gpu‚24.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚40.0 gpu.0.powerWatts‚110.125' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.025 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Úå{yG‹,‚(:¼ ÉÀéÒÀªÂú& disk.rbd0.in‚1.759218604441423e+13 disk.sda2.out‚0" proc.memory.rssMB‚ 2802.8984375* disk./.usagePercent‚37.781336919678026 disk.sda.out‚ 1236.59765625 disk.drbd2165.in‚0$ disk.nvme0c0n1.in‚42640.16796875" disk.nvme0n1.in‚42640.16796875 disk.drbd2165.out‚0% disk.nvme0n1.out‚141391.970703125 network.sent‚ 2.0840899e+07 disk.rbd1.out‚ 47.009765625 disk.sda3.in‚ 106.21875$ memory_percent‚82.82592296600342 disk.dm-1.in‚ 106.21875 disk.sda3.out‚ 1236.59765625 disk.rbd3.out‚ 0.22412109375' disk.nvme0c0n1.out‚141391.970703125 proc.cpu.threads‚17 disk.dm-1.out‚ 1236.59765625) proc.memory.availableMB‚ 2813.80078125 disk.sda2.in‚0 network.recv‚ 4.4756e+06$ disk./.usageGB‚562.9692230224609* proc.memory.percent‚17.107534408569336 cpu‚25.588799176013843 disk.rbd3.in‚0 disk.sda.in‚ 106.21875 disk.rbd0.out‚0 disk.dm-0.in‚42640.16796875" disk.dm-0.out‚141391.970703125 disk.sda1.out‚0 disk.rbd1.in‚ 0.52392578125 disk.sda1.in‚0mãhæ“Œ,‚(:ˆ ØÀéÒÜøƒ÷ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚250.505' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚50.101 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0V¡Y¨D,‚(:¹ ØÀéҾ˃ú proc.cpu.threads‚17 disk.sda3.in‚ 106.21875 network.sent‚ 2.0846019e+07& disk.rbd0.in‚1.759218604441423e+13" disk.dm-0.out‚141398.201171875 disk.sda3.out‚ 1237.07421875 disk.rbd1.out‚ 47.087890625) proc.memory.availableMB‚ 2813.79296875) disk./.usagePercent‚37.78179466114925$ disk.nvme0c0n1.in‚42640.37109375" disk.nvme0n1.in‚42640.37109375 disk.sda2.in‚0 disk.rbd1.in‚ 0.52392578125 disk.sda.in‚ 106.21875 disk.sda.out‚ 1237.07421875" proc.memory.rssMB‚ 2802.8984375* proc.memory.percent‚17.107534408569336 disk.sda1.in‚0 disk.rbd3.in‚0' disk.nvme0c0n1.out‚141398.201171875% disk.nvme0n1.out‚141398.201171875$ disk./.usageGB‚562.9760437011719 disk.dm-1.out‚ 1237.07421875 cpu‚25.58878218539 network.recv‚ 4.478622e+06 disk.rbd0.out‚0 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.rbd3.out‚ 0.22412109375 disk.sda2.out‚0$ memory_percent‚82.82597064971924 disk.dm-0.in‚42640.37109375 disk.drbd2165.out‚0 disk.dm-1.in‚ 106.21875q––’Ž,‚(:‡ çÀéÒß•—÷ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚269.59' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.918 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0%ê”ÎH,‚(:½ çÀéÒÓÕƒú disk.drbd2165.in‚0! disk.nvme0n1.in‚ 42656.8359375 disk.sda2.in‚0) proc.memory.availableMB‚ 2813.96484375 disk.dm-1.out‚ 1237.32421875 disk.dm-0.in‚ 42656.8359375 disk.rbd3.in‚0# disk.nvme0c0n1.in‚ 42656.8359375& disk.nvme0n1.out‚141413.7470703125 network.sent‚ 2.0851196e+07 disk.rbd3.out‚ 0.22412109375$ memory_percent‚82.82492160797119" proc.memory.rssMB‚ 2802.8984375# disk.dm-0.out‚141413.7470703125$ disk./.usageGB‚562.9839744567871 disk.rbd0.out‚0 disk.rbd1.out‚ 47.087890625 disk.rbd1.in‚ 0.52392578125 network.recv‚ 4.482497e+06) disk./.usagePercent‚37.78232690081293 disk.sda3.in‚ 106.21875 disk.sda2.out‚0 cpu‚25.588666665687203 disk.sda3.out‚ 1237.32421875 disk.drbd2165.out‚0 disk.sda.in‚ 106.21875 disk.sda.out‚ 1237.32421875* proc.memory.percent‚17.107534408569336 proc.cpu.threads‚17 disk.sda1.out‚0( disk.nvme0c0n1.out‚141413.7470703125 disk.dm-1.in‚ 106.21875& disk.rbd0.in‚1.759218604441423e+13 disk.sda1.in‚0oà4,/ _wandbruntime‚18011  _runtime‚18011׺9É‘,à & lr-AdamW/pg1‚0.0003100042592064389 & lr-AdamW/pg2‚0.0003100042592064389 trainer/global_step‚10849 % _timestamp‚1.7843078269647908e+09 _runtime‚18011.183086628 _step‚891–ªtÎ’,È & lr-AdamW/pg1‚0.0003100042592064389 & lr-AdamW/pg2‚0.0003100042592064389 trainer/global_step‚10849 % _timestamp‚1.7843078269647908e+09 _runtime‚18011.183086628 _step‚891ûò}Ò”4“,/ _wandbruntime‚18011  _runtime‚18011ÖO_¬'”,‡P P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚93987011 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10208948001792 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10207602612224 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚842218305351680 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚93987011 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10190134872787 _runtime‚18011.187921668 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3129959870700544 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚3128593457666771 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚3118403348221952 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3119750896525312 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚92603633 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10191479473109 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3119750922698752 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚3129958499137536 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚30765062 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10207602612224 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚93977038 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 186580665 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3129959870700544 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚842218673442816 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13169365533696 VGDeviceStatsMonitor.on_train_batch_start/req÷ ‹üæuested_bytes.small_pool.peak‚ 1679318926 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚58601602 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚58601342 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 186590644 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3128594827695061 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3129958499137536 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚30764807 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _step‚892 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚260 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚27836540 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚255 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 trainer/global_step‚10849 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚3119750922698752 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚93977038 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚92603633 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚3119750896525312 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚855387636348928 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13169330997248 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 % _timestamp‚1.7843078269705036e+09 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚92603627 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚27836535 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10208948001792 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚855388038976512 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820ÀÅÂe(•,ŒP dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13169365533696 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚58601602 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚58601342 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 186590644 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3128594827695061 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3129958499137536 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚30764807 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _step‚892 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚260 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚27836540 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚255 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 trainer/global_step‚10849 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚3119750922698752 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚93977038 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚92603633 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚3119750896525312 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚855387636348928 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13169330997248 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 % _timestamp‚1.7843078269705036e+09 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚92603627 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚27836535 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10208948001792 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚855388038976512 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚93987011 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10208948001792 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10207602612224 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚842218305351680 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚93987011 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10190134872787 _runtime‚18011.187921668 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3129959870700544 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚3128593457666771 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚3118403348221952 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3119750896525312 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚92603633 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10191479473109 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3119750922698752 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚3129958499137536 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚30765062 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10207602612224 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚93977038 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 186580665 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3129959870700544 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚842218673442816 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448üöŸâ"4–,/ _wandbruntime‚18012  _runtime‚18012ª+fx'—,N ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10192402939467 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3130238796473344 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚244 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 _step‚893 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3120028897255424 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚27839083 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚58606659 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚3120028923428864 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 186597349 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚58606908 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3130237424903168 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 186607342 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚855466726924288 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3128873632939595 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10208527647744 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚92611839 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚92611839 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3118681204572160 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚93985516 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 186597349 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚30767825 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 QDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚93995503 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚92611833 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13170532544512 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚249 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚842295791759360 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10191058339033 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚3120028897255424 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚855466324303872 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 186607342 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _runtime‚18012.572611047 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3130238796473344 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚27839078 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚58606908 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3130237424903168 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 186607342 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚855466726924288 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3128873632939595 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10208527647744 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚92611839 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚92611839 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3118681204572160 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚93985516 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 186597349 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚30767825 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 QDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚93995503 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚92611833 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13170532544512 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚249 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚842295791759360 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10191058339033 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.laù¤u€ rge_pool.freed‚3120028897255424 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚855466324303872 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 186607342 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _runtime‚18012.572611047 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3130238796473344 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚27839078 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚58606659 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640ýÌ‹Þ4™,/ _wandbruntime‚18012  _runtime‚18012-&áš,‰ % _timestamp‚1.7843078284412096e+09 /loss/FRt1/assignment_loss‚1.221331169388511 2loss/FBt1/assignment_loss‚0.002063710069961644 1loss/SRqqt2/detection_loss‚0.5186955203709828 trainer/global_step‚10849 _step‚894 3loss/SRqqt2/assignment_loss‚0.29968354328220215 3loss/FBt2/assignment_loss‚0.0012371280963545672 0loss/FRt1/detection_loss‚0.47670921263807237 /loss/FRt2/detection_loss‚1.3393475442825913 1loss/SRqqt1/detection_loss‚0.6387675558565781 /loss/FBt1/detection_loss‚0.5077049267614105 _runtime‚18012.658173294 2loss/SRqqt1/assignment_loss‚0.2180694493858122 0loss/FBt2/detection_loss‚0.42882448688978914 epoch‚27 0loss/FRt2/assignment_loss‚0.3035609818513121 %loss/total_loss‚5.955995228873578OŽ~ô›,Ž epoch‚27 0loss/FRt2/assignment_loss‚0.3035609818513121 %loss/total_loss‚5.955995228873578 % _timestamp‚1.7843078284412096e+09 /loss/FRt1/assignment_loss‚1.221331169388511 2loss/FBt1/assignment_loss‚0.002063710069961644 1loss/SRqqt2/detection_loss‚0.5186955203709828 trainer/global_step‚10849 _step‚894 3loss/SRqqt2/assignment_loss‚0.29968354328220215 3loss/FBt2/assignment_loss‚0.0012371280963545672 0loss/FRt1/detection_loss‚0.47670921263807237 /loss/FRt2/detection_loss‚1.3393475442825913 1loss/SRqqt1/detection_loss‚0.6387675558565781 /loss/FBt1/detection_loss‚0.5077049267614105 _runtime‚18012.658173294 2loss/SRqqt1/assignment_loss‚0.2180694493858122 0loss/FBt2/detection_loss‚0.42882448688978914þ¡U±’”œ,‚(:‰ öÀéÒ«îàö gpu.0.gpu‚99.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚338.986' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚67.7972 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Wpò­A,‚(:¶ öÀéÒ€›¶ù* disk./.usagePercent‚37.750615247546584' disk.nvme0c0n1.out‚141424.509765625 disk.sda2.in‚0 disk.sda1.in‚0 disk.sda3.in‚ 106.21875 disk.rbd1.in‚ 0.52392578125$ disk./.usageGB‚562.5114479064941 disk.dm-1.in‚ 106.21875 disk.dm-1.out‚ 1238.27734375& disk.rbd0.in‚1.759218604441423e+13" disk.dm-0.out‚141424.509765625 disk.sda1.out‚0% disk.nvme0n1.out‚141424.509765625 disk.rbd1.out‚ 47.166015625 network.recv‚ 4.493373e+06 disk.rbd3.in‚0 disk.rbd3.out‚ 0.22412109375 disk.nvme0n1.in‚ 42707.640625 network.sent‚ 2.0860211e+07 disk.sda.in‚ 106.21875 disk.sda.out‚ 1238.27734375$ memory_percent‚82.82804489135742 proc.cpu.threads‚17 disk.sda3.out‚ 1238.27734375 disk.sda2.out‚0" proc.memory.rssMB‚ 2802.8984375 disk.dm-0.in‚ 42707.640625 disk.drbd2165.out‚0" disk.nvme0c0n1.in‚ 42707.640625' proc.memory.availableMB‚ 2813.453125* proc.memory.percent‚17.107534408569336 cpu‚25.588650167185037 disk.rbd0.out‚0 disk.drbd2165.in‚0¹&ŒíŸž,‚(:” …ÁéÒâÿö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚267.174' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚53.434799999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0h^F½-Ÿ,‚(:¢ …ÁéÒùÈîù network.sent‚ 2.0923469e+07$ disk./.usageGB‚562.5207252502441 disk.sda.in‚ 106.21875 disk.dm-1.in‚ 106.21875 disk.dm-1.out‚1238.5 disk.rbd0.out‚0$ disk.dm-0.out‚141437.58544921875 proc.cpu.threads‚17 disk.sda2.out‚0) disk./.usagePercent‚37.75123785786986 disk.drbd2165.out‚0 disk.drbd2165.in‚0 disk.nvme0c0n1.in‚ 42708.0625* proc.memory.percent‚17.107558250427246 disk.sda.out‚1238.5' disk.nvme0n1.out‚141437.58544921875 disk.sda2.in‚0 disk.rbd1.out‚ 47.166015625$ memory_percent‚82.82816410064697 disk.nvme0n1.in‚ 42708.0625& disk.rbd0.in‚1.759218604441423e+13 disk.dm-0.in‚ 42708.0625 disk.sda3.in‚ 106.21875 disk.rbd3.out‚ 0.22412109375 network.recv‚ 4.498156e+06 disk.rbd3.in‚0) disk.nvme0c0n1.out‚141437.58544921875# proc.memory.rssMB‚ 2802.90234375 cpu‚25.58863059371841 disk.sda1.in‚0 disk.rbd1.in‚ 0.52392578125 disk.sda1.out‚0 disk.sda3.out‚1238.5) proc.memory.availableMB‚ 2813.43359375jp¿ž ,‚(:“ ”ÁéÒÚžèö gpu.0.gpu‚83.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚330.913' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚66.18260000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Þ4÷B¡,‚(:· ”ÁéÒÒñ‘ù disk.sda1.in‚0 disk.rbd3.in‚0 disk.rbd3.out‚ 0.22412109375( disk.nvme0c0n1.out‚141451.4697265625 disk.sda.in‚ 106.21875 disk.dm-1.out‚ 1238.69921875 disk.rbd0.out‚0 disk.drbd2165.out‚0 cpu‚25.588642181982312 network.sent‚ 2.0928552e+07 disk.drbd2165.in‚0& proc.memory.availableMB‚ 2813.84375 disk.sda2.out‚0# disk.dm-0.out‚141451.4697265625 disk.sda1.out‚0 disk.sda.out‚ 1238.69921875 disk.dm-0.in‚ 42708.265625 disk.sda2.in‚0# memory_percent‚82.8256607055664 proc.cpu.threads‚17& disk.rbd0.in‚1.759218604441423e+13& disk.nvme0n1.out‚141451.4697265625# proc.memory.rssMB‚ 2802.90234375* proc.memory.percent‚17.107558250427246) disk./.usagePercent‚37.75152049018766$ disk./.usageGB‚562.5249366760254 disk.sda3.out‚ 1238.69921875 disk.rbd1.in‚ 0.52392578125 network.recv‚ 4.501354e+06 disk.sda3.in‚ 106.21875 disk.rbd1.out‚ 47.244140625 disk.dm-1.in‚ 106.21875" disk.nvme0c0n1.in‚ 42708.265625 disk.nvme0n1.in‚ 42708.265625rú˜Ÿ¢,‚(:” £ÁéÒÛà÷ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚263.954' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚52.790800000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0îMPE£,‚(:º £ÁéÒ¢üúù disk.rbd0.out‚0 disk.rbd3.out‚ 0.22412109375 network.sent‚ 2.0933657e+07 network.recv‚ 4.504946e+06$ disk./.usageGB‚562.5322875976562 disk.sda.out‚ 1238.94921875 disk.sda1.in‚0 disk.drbd2165.out‚0 disk.rbd3.in‚0 cpu‚25.588552414765942 disk.dm-1.in‚ 106.21875& disk.rbd0.in‚1.759218604441423e+13 disk.sda3.out‚ 1238.94921875 disk.drbd2165.in‚0* proc.memory.percent‚17.107558250427246 proc.cpu.threads‚17 disk.rbd1.out‚ 47.244140625$ memory_percent‚82.82585144042969 disk.sda.in‚ 106.21875 disk.dm-0.in‚ 42708.484375 disk.sda2.out‚0 disk.rbd1.in‚ 0.52392578125% proc.memory.availableMB‚ 2813.8125) disk./.usagePercent‚37.75201381670614 disk.dm-1.out‚ 1238.94921875 disk.sda1.out‚0" disk.nvme0c0n1.in‚ 42708.484375) disk.nvme0c0n1.out‚141464.61181640625' disk.nvme0n1.out‚141464.61181640625 disk.sda2.in‚0$ disk.dm-0.out‚141464.61181640625 disk.sda3.in‚ 106.21875 disk.nvme0n1.in‚ 42708.484375# proc.memory.rssMB‚ 2802.90234375fë=Çž¤,‚(:“ ²ÁéÒµÎÞø gpu.0.gpu‚99.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚328.257' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚65.65140000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Z+äÝB¥,‚(:· ²ÁéÒÕÔáù) disk./.usagePercent‚37.75248205448627 disk.rbd0.out‚0 disk.dm-0.in‚ 42958.1875 disk.sda3.out‚ 1239.13671875 disk.drbd2165.in‚0) proc.memory.availableMB‚ 2813.80859375 disk.sda.out‚ 1239.13671875 disk.drbd2165.out‚0 disk.rbd3.out‚ 0.22412109375 disk.sda2.in‚0 disk.sda2.out‚0 network.sent‚ 2.0938783e+07 disk.dm-1.out‚ 1239.13671875# memory_percent‚82.8258752822876 disk.dm-1.in‚ 106.21875 disk.sda1.in‚0 disk.rbd3.in‚0 disk.rbd1.in‚ 0.52392578125# proc.memory.rssMB‚ 2802.90234375 disk.sda.in‚ 106.21875 cpu‚25.588562503784107$ disk./.usageGB‚562.5392646789551& disk.rbd0.in‚1.759218604441423e+13$ disk.dm-0.out‚141478.50048828125 disk.nvme0c0n1.in‚ 42958.1875) disk.nvme0c0n1.out‚141478.50048828125 disk.nvme0n1.in‚ 42958.1875' disk.nvme0n1.out‚141478.50048828125 disk.rbd1.out‚ 47.306640625 disk.sda1.out‚0* proc.memory.percent‚17.107558250427246 proc.cpu.threads‚17 network.recv‚ 4.508384e+06 disk.sda3.in‚ 106.21875Ûúq"Ÿ¦,‚(:” ÁÁéÒÔåö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚284.911' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚56.982200000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0W á‘@§,‚(:µ ÁÁéÒ³ûëø disk.rbd0.out‚0 disk.dm-0.in‚ 42958.390625 disk.sda.in‚ 106.21875 disk.dm-1.out‚ 1239.359375& disk.rbd0.in‚1.759218604441423e+13 disk.sda1.in‚0 disk.drbd2165.out‚0 disk.rbd1.out‚47.33056640625 proc.cpu.threads‚17 disk.sda1.out‚0 disk.sda3.out‚ 1239.359375 disk.sda2.out‚0* proc.memory.percent‚17.107558250427246 disk.sda3.in‚ 106.21875 disk.rbd3.in‚0$ memory_percent‚82.82432556152344$ disk.dm-0.out‚141490.60107421875' disk.nvme0n1.out‚141490.60107421875% proc.memory.availableMB‚ 2814.0625 network.recv‚ 4.511781e+06$ disk./.usageGB‚562.5459060668945 disk.drbd2165.in‚0 disk.rbd3.out‚ 0.22412109375 disk.rbd1.in‚ 0.52392578125 cpu‚25.58850565120709 network.sent‚ 2.0943331e+07# proc.memory.rssMB‚ 2802.90234375 disk.sda.out‚ 1239.359375" disk.nvme0c0n1.in‚ 42958.390625) disk.nvme0c0n1.out‚141490.60107421875 disk.nvme0n1.in‚ 42958.390625 disk.sda2.in‚0) disk./.usagePercent‚37.75292776360338 disk.dm-1.in‚ 106.21875ÖñùQ4¨,/ _wandbruntime‚18090  _runtime‚18090~%ÔË©,Å _runtime‚18090.692028831 _step‚895 ' lr-AdamW/pg1‚0.00030724809965632804 ' lr-AdamW/pg2‚0.00030724809965632804 trainer/global_step‚10899 % _timestamp‚1.7843079064727287e+09cÚЪ,Ê % _timestamp‚1.7843079064727287e+09 _runtime‚18090.692028831 _step‚895 ' lr-AdamW/pg1‚0.00030724809965632804 ' lr-AdamW/pg2‚0.00030724809965632804 trainer/global_step‚10899ÿ"ãËÔ4«,/ _wandbruntime‚18090  _runtime‚18090ˆ‹ç (¬,†P G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 187426696 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚94412763 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚279 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36633600 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚93013927 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚3143904851573760 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13229100701696 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13229064068096 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚3133650959204352 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚93013927 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3133650959204352 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚30904597 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚94412763 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10255263932416 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚859322094528512 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 187416717 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10255263932416 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚30904318 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚859321689803776 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚846092625735680 trainer/global_step‚10899 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3143906223136768 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚3132297411702784 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚58868003 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚3133650933030912 _runtime‚18090.69604924 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3143904851573760 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚3132297437130752'‡V\!­,‹P VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚94412763 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚279 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36633600 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚93013927 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚3143904851573760 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13229100701696 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13229064068096 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚3133650959204352 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚93013927 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3133650959204352 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚30904597 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚94412763 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10255263932416 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚859322094528512 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 187416717 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10255263932416 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚30904318 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚859321689803776 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚846092625735680 trainer/global_step‚10899 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3143906223136768 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚3132297411702784 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚58868003 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚3133650933030912 _runtime‚18090.69604924 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3143904851573760 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚3132297437130752 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 187426696 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0€.²&Š4®,/ _wandbruntime‚18092  _runtime‚180922ÕJT'¯,‘N H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚30907088 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚94411268 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 187433401 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3133928959934464 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚94421255 trainer/global_step‚10899 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚93022139 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚58873316 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 187443394 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚93022133 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13230248613888 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚94411268 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 187443394 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚3133928959934464 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10238640052575 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚93022139 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10254843578368 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404717568 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13230285240320 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚3133928933761024 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10254843578368 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3144185148909568 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚58873592 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3142813958961503 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 % _timestamp‚1.7843079080525784e+09 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚276 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚94421255 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚846170480234496 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3144185148909568 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 _step‚897 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚846170112143360 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3132575318908928 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 _runtime‚18092.269997987 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 _step‚897 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚846170112143360 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3132575318908928 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 _runtime‚18092.269997987 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚58873316 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 187443394 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚93022133 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13230248613888 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚94411268 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 187443394 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚3133928959934464 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10238640052575 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚93022139 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10254843578368 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404717568 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13230285240320 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 474994711192à»c4±,/ _wandbruntime‚18092  _runtime‚18092 E’²,‰ % _timestamp‚1.7843079081367145e+09 %loss/total_loss‚6.060946941266764 2loss/SRqqt2/assignment_loss‚0.3039854465321534 3loss/FBt1/assignment_loss‚0.0002391984308747912 1loss/SRqqt2/detection_loss‚0.5750986162715578 /loss/FBt1/detection_loss‚0.5088854192460819 _runtime‚18092.353620837 epoch‚27 0loss/FRt2/assignment_loss‚0.3305233075519159 4loss/FBt2/assignment_loss‚0.00016975928411527345 1loss/SRqqt1/detection_loss‚0.6746632312761683 /loss/FBt2/detection_loss‚0.4153835174726508 trainer/global_step‚10899 _step‚898 0loss/FRt1/assignment_loss‚1.1661735536224807 3loss/SRqqt1/assignment_loss‚0.22781271490061386 0loss/FRt1/detection_loss‚0.45749018109261164 -loss/FRt2/detection_loss‚1.40052199558554q1U¨³,Ž epoch‚27 0loss/FRt2/assignment_loss‚0.3305233075519159 4loss/FBt2/assignment_loss‚0.00016975928411527345 1loss/SRqqt1/detection_loss‚0.6746632312761683 /loss/FBt2/detection_loss‚0.4153835174726508 trainer/global_step‚10899 _step‚898 0loss/FRt1/assignment_loss‚1.1661735536224807 3loss/SRqqt1/assignment_loss‚0.22781271490061386 0loss/FRt1/detection_loss‚0.45749018109261164 -loss/FRt2/detection_loss‚1.40052199558554 % _timestamp‚1.7843079081367145e+09 %loss/total_loss‚6.060946941266764 2loss/SRqqt2/assignment_loss‚0.3039854465321534 3loss/FBt1/assignment_loss‚0.0002391984308747912 1loss/SRqqt2/detection_loss‚0.5750986162715578 /loss/FBt1/detection_loss‚0.5088854192460819 _runtime‚18092.353620837‚ÞôqKŸ´,‚(:” ÐÁéÒñÊÑö gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚284.382' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚56.876400000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ä}ïEµ,‚(:º ÐÁéÒÔ³·ù# proc.memory.rssMB‚ 2802.90234375 disk.dm-1.in‚ 106.21875 disk.rbd3.in‚0 disk.dm-1.out‚ 1239.70703125 disk.rbd0.out‚0 disk.rbd3.out‚ 0.22412109375' disk.nvme0n1.out‚141500.27490234375* proc.memory.percent‚17.107558250427246 network.sent‚ 2.0949426e+07 disk.sda3.in‚ 106.21875) disk.nvme0c0n1.out‚141500.27490234375 disk.nvme0n1.in‚ 42958.8125 disk.rbd1.out‚47.33056640625 cpu‚25.588416888561227 proc.cpu.threads‚17$ disk./.usageGB‚562.5528869628906 disk.sda.in‚ 106.21875 disk.sda.out‚ 1239.70703125 disk.sda1.in‚0 disk.sda1.out‚0 disk.nvme0c0n1.in‚ 42958.8125 disk.sda2.in‚0 network.recv‚ 4.515364e+06 disk.dm-0.in‚ 42958.8125 disk.drbd2165.in‚0 disk.sda2.out‚0$ disk.dm-0.out‚141500.27490234375) proc.memory.availableMB‚ 2813.76171875) disk./.usagePercent‚37.75339625739104& disk.rbd0.in‚1.759218604441423e+13 disk.sda3.out‚ 1239.70703125 disk.drbd2165.out‚0$ memory_percent‚82.82616138458252 disk.rbd1.in‚ 0.52392578125é$×”¶,‚(:‰ ßÁéÒ¿Ý„÷ gpu.0.gpu‚94.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚337.528' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚67.5056 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0dÆ 2G·,‚(:¼ ßÁéÒ‹ž·ù disk.nvme0n1.in‚ 42959.03125 disk.sda3.in‚ 106.21875 disk.rbd3.in‚0 network.sent‚ 2.1016241e+07 disk.rbd1.out‚ 47.4326171875 disk.sda.in‚ 106.21875 disk.dm-0.in‚ 42959.03125) disk./.usagePercent‚37.75368733795747 disk.sda1.out‚0 disk.sda3.out‚ 1239.91796875 disk.rbd3.out‚ 0.22412109375) proc.memory.availableMB‚ 2813.80078125* proc.memory.percent‚17.107558250427246 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda.out‚ 1239.91796875 disk.dm-1.out‚ 1239.91796875 disk.drbd2165.in‚0! disk.nvme0c0n1.in‚ 42959.03125 disk.rbd1.in‚ 0.52392578125 disk.dm-1.in‚ 106.21875$ disk.dm-0.out‚141516.87158203125 disk.drbd2165.out‚0$ memory_percent‚82.82592296600342# proc.memory.rssMB‚ 2802.90234375 proc.cpu.threads‚17$ disk./.usageGB‚562.5572242736816' disk.nvme0n1.out‚141516.87158203125) disk.nvme0c0n1.out‚141516.87158203125 disk.rbd0.out‚0 network.recv‚ 4.526889e+06& disk.rbd0.in‚1.759218604441423e+13 disk.sda1.in‚0 cpu‚25.588440240812897ƒË{*”¸,‚(:‰ îÁéÒÁÊ‚ø gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚280.614' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚56.1228 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0|Yß›:¹,‚(:¯ îÁéÒ£¡ßù disk.sda.in‚ 106.21875 disk.sda.out‚ 1240.2109375& disk.rbd0.in‚1.759218604441423e+13 disk.sda1.in‚0) proc.memory.availableMB‚ 2813.75390625 disk.sda2.in‚0 disk.rbd1.out‚ 47.4326171875* proc.memory.percent‚17.107558250427246 cpu‚25.588366270398698 network.recv‚ 4.529967e+06 disk.sda1.out‚0" disk.nvme0c0n1.in‚ 43071.796875% disk.nvme0c0n1.out‚141526.0390625 disk.sda2.out‚0 disk.drbd2165.in‚0# proc.memory.rssMB‚ 2802.90234375 disk.dm-1.in‚ 106.21875 disk.dm-0.out‚141526.0390625 network.sent‚ 2.102132e+07 disk.nvme0n1.in‚ 43071.796875# disk.nvme0n1.out‚141526.0390625 disk.dm-0.in‚ 43071.796875 disk.sda3.out‚ 1240.2109375 disk.drbd2165.out‚0 disk.rbd3.out‚ 0.22412109375) disk./.usagePercent‚37.75411871065267$ disk./.usageGB‚562.5636520385742 disk.dm-1.out‚ 1240.2109375 disk.sda3.in‚ 106.21875 proc.cpu.threads‚17 disk.rbd1.in‚ 0.52392578125 disk.rbd0.out‚0 disk.rbd3.in‚0$ memory_percent‚82.82620906829834G\r&Ÿº,‚(:” ýÁéÒ©ïö gpu.0.gpu‚57.0 gpu.0.memory‚19, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚40.0 gpu.0.powerWatts‚109.588' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.917599999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0[fP»,‚(:Å ýÁéÒ¨ÂÌù disk.drbd2165.in‚0$ disk./.usageGB‚562.5701522827148 disk.sda.in‚ 106.21875 cpu‚25.588348794094934 disk.rbd3.in‚0" disk.nvme0n1.in‚43072.20703125 disk.sda2.out‚0 disk.rbd1.in‚ 0.52392578125 disk.drbd2165.out‚0 disk.rbd3.out‚ 0.22412109375) disk.nvme0c0n1.out‚141543.06787109375) proc.memory.availableMB‚ 2813.50390625 proc.cpu.threads‚17 network.recv‚ 4.533503e+06) disk./.usagePercent‚37.75455494749101 disk.sda3.in‚ 106.21875 disk.sda3.out‚ 1240.51171875 disk.sda2.in‚0* proc.memory.percent‚17.107558250427246& disk.rbd0.in‚1.759218604441423e+13 disk.rbd0.out‚0' disk.nvme0n1.out‚141543.06787109375# proc.memory.rssMB‚ 2802.90234375 disk.dm-1.in‚ 106.21875 disk.dm-1.out‚ 1240.51171875 disk.sda1.in‚0$ disk.nvme0c0n1.in‚43072.20703125 disk.rbd1.out‚ 47.4951171875$ memory_percent‚82.82773494720459 network.sent‚ 2.1025912e+07 disk.sda.out‚ 1240.51171875 disk.dm-0.in‚43072.20703125$ disk.dm-0.out‚141543.06787109375 disk.sda1.out‚0Üî=y’¼,‚(:‡ ŒÂéÒÎÆþö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚276.07' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.214 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0}ÒMA½,‚(:¶ ŒÂéÒŸ”éù disk.sda2.out‚0 disk.rbd1.in‚ 0.52392578125 disk.sda.in‚ 106.21875 disk.rbd0.out‚0# disk.dm-0.out‚141552.2216796875 disk.sda1.out‚0 disk.rbd3.in‚0" disk.nvme0n1.in‚43072.42578125 disk.rbd1.out‚ 47.4951171875 network.recv‚ 4.53696e+06) disk./.usagePercent‚37.75501755310538 disk.sda1.in‚0 disk.sda3.in‚ 106.21875 disk.sda3.out‚ 1240.78125# proc.memory.rssMB‚ 2802.90234375 disk.drbd2165.in‚0 disk.rbd3.out‚ 0.22412109375 disk.sda2.in‚0' proc.memory.availableMB‚ 2813.640625* proc.memory.percent‚17.107558250427246$ disk./.usageGB‚562.5770454406738 disk.dm-1.in‚ 106.21875 cpu‚25.588275263063196 disk.dm-0.in‚43072.42578125$ memory_percent‚82.82690048217773& disk.nvme0n1.out‚141552.2216796875 proc.cpu.threads‚17 network.sent‚ 2.1031565e+07& disk.rbd0.in‚1.759218604441423e+13 disk.drbd2165.out‚0$ disk.nvme0c0n1.in‚43072.42578125 disk.sda.out‚ 1240.78125 disk.dm-1.out‚ 1240.78125( disk.nvme0c0n1.out‚141552.2216796875i˜Z4¾,/ _wandbruntime‚18170  _runtime‚18170,2h§É¿,à & lr-AdamW/pg2‚0.0003044944242015361 trainer/global_step‚10949 % _timestamp‚1.7843079861697536e+09 _runtime‚18170.391064985 _step‚899 & lr-AdamW/pg1‚0.0003044944242015361÷9(ºÎÀ,È & lr-AdamW/pg1‚0.0003044944242015361 & lr-AdamW/pg2‚0.0003044944242015361 trainer/global_step‚10949 % _timestamp‚1.7843079861697536e+09 _runtime‚18170.391064985 _step‚899ƒoT=Ï4Á,/ _wandbruntime‚18170  _runtime‚18170Æ°ã¥Â,†P VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚3156474109710587 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10283953699325 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10300234473472 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚863254771483648 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3147550995709952 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚59134125 ÎKg#XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚31043588 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚863255178305536 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚94838515 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚303 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 $ _timestamp‚1.784307986175766e+09 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3156475479738877 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚3146191500611584 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3157852575572992 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚3147550969536512 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13287864094720 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚28090835 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚849966946119680 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10300234473472 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚3157851204009984 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚28090840 _runtime‚18170.393042058 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 406821888 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10282609099003 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚94828542 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3157851204009984 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3147550969536512 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10301579863040 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 188262748 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚298 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13287825363968 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504&$(Ã,‹P NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚28090835 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚849966946119680 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10300234473472 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚3157851204009984 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚28090840 _runtime‚18170.393042058 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 406821888 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10282609099003 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚94828542 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3157851204009984 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3147550969536512 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10301579863040 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 188262748 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚298 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13287825363968 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚3156474109710587 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10283953699325 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10300234473472 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚863254771483648 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3147550995709952 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚59134125 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚31043588 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚863255178305536 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚94838515 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚303 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 $ _timestamp‚1.784307986175766e+09 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3156475479738877 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚3146191500611584 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3157852575572992 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚3147550969536512 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13287864094720 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 J8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚31046055 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚863333431429120 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 188269453 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚93432439 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3147828970266624 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚38723584 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚94837020 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚94847007 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3158131501345792 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 188279446 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 188279446 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 QDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3158131501345792 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚3147828970266624 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10302504905728 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚28093378 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 406814720 trainer/global_step‚10949 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3156754284983411 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚59139433 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10284877165683 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995(²Q[ Æ,•N KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚31046055 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚863333431429120 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 188269453 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚93432439 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3147828970266624 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚38723584 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚94837020 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚94847007 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.pea)…ojk‚ 1679318926 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3158131501345792 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 188279446 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 188279446 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 QDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3158131501345792 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚3147828970266624 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10302504905728 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚28093378 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 406814720 trainer/global_step‚10949 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3156754284983411 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚59139433 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10284877165683 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 188269453 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚298 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10283532565249 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚293 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0…ÎÄõö4Ç,/ _wandbruntime‚18171  _runtime‚18171½¬ØEÈ,‹ 2loss/SRqqt1/assignment_loss‚0.2695051413750628 /loss/FRt2/detection_loss‚1.3416720130235742 /loss/FBt2/detection_loss‚0.4332191370664917 _runtime‚18171.863603975 4loss/FBt1/assignment_loss‚2.9566034687805394e-05 3loss/FBt2/assignment_loss‚5.015417654070995e-05 0loss/FRt1/detection_loss‚0.47677789376333024 /loss/FBt1/detection_loss‚0.4220991994016895 0loss/FRt1/assignment_loss‚1.2347568879946778 1loss/FRt2/assignment_loss‚0.33024252207954424 1loss/SRqqt1/detection_loss‚0.6280116637379264 epoch‚27 trainer/global_step‚10949 % _timestamp‚1.7843079876466584e+09 _step‚902 2loss/SRqqt2/assignment_loss‚0.2509585644105467 1loss/SRqqt2/detection_loss‚0.5327595232301245 %loss/total_loss‚5.920082266294196Èc¯AÉ, 1loss/SRqqt1/detection_loss‚0.6280116637379264 epoch‚27 trainer/global_step‚10949 % _timestamp‚1.7843079876466584e+09 _step‚902 2loss/SRqqt2/assignment_loss‚0.2509585644105467 1loss/SRqqt2/detection_loss‚0.5327595232301245 %loss/total_loss‚5.920082266294196 2loss/SRqqt1/assignment_loss‚0.2695051413750628 /loss/FRt2/detection_loss‚1.3416720130235742 /loss/FBt2/detection_loss‚0.4332191370664917 _runtime‚18171.863603975 4loss/FBt1/assignment_loss‚2.9566034687805394e-05 3loss/FBt2/assignment_loss‚5.015417654070995e-05 0loss/FRt1/detection_loss‚0.47677789376333024 /loss/FBt1/detection_loss‚0.4220991994016895 0loss/FRt1/assignment_loss‚1.2347568879946778 1loss/FRt2/assignment_loss‚0.33024252207954424†›.„“Ê,‚(:ˆ ›ÂéÒ¡ªÚù gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚38.0 gpu.0.powerWatts‚139.685' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚27.937 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0äGàNË,‚(:à ›ÂéÒÃúÍû) disk.nvme0c0n1.out‚141564.28759765625" disk.nvme0n1.in‚43072.84765625$ disk.dm-0.out‚141564.28759765625 disk.sda1.out‚0 network.recv‚ 4.539776e+06 disk.rbd3.out‚ 0.22412109375' disk.nvme0n1.out‚141564.28759765625 disk.rbd1.out‚ 47.5732421875 disk.sda1.in‚0 disk.sda3.in‚ 106.21875 disk.sda3.out‚ 1241.00390625) proc.memory.availableMB‚ 2813.71484375$ disk./.usageGB‚562.5832977294922$ disk.nvme0c0n1.in‚43072.84765625 disk.rbd0.out‚0# proc.memory.rssMB‚ 2802.90234375* proc.memory.percent‚17.107558250427246 cpu‚25.588211034794128 disk.dm-1.in‚ 106.21875 disk.drbd2165.out‚0 disk.sda2.out‚0 disk.sda.out‚ 1241.00390625$ memory_percent‚82.82644748687744 disk.rbd3.in‚0 disk.dm-1.out‚ 1241.00390625& disk.rbd0.in‚1.759218604441423e+13 proc.cpu.threads‚17 disk.sda2.in‚0 disk.sda.in‚ 106.21875 network.sent‚ 2.1035734e+07' disk./.usagePercent‚37.755437149454 disk.drbd2165.in‚0 disk.rbd1.in‚ 0.52392578125 disk.dm-0.in‚43072.84765625 “{ö”Ì,‚(:‰ ªÂéÒíÒ÷ö gpu.0.gpu‚99.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚337.146' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚67.4292 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0•‡®Ë@Í,‚(:µ ªÂéÒ³×½ù* disk./.usagePercent‚37.755675748475916$ disk./.usageGB‚562.5868530273438 disk.sda3.in‚ 106.21875% disk.nvme0n1.out‚141576.951171875 disk.rbd0.out‚0 disk.dm-0.in‚ 43268.4453125! disk.nvme0n1.in‚ 43268.4453125 disk.rbd1.out‚ 47.5732421875$ memory_percent‚82.82585144042969* proc.memory.percent‚17.107558250427246 network.recv‚ 4.543992e+06 disk.sda1.in‚0# disk.nvme0c0n1.in‚ 43268.4453125# proc.memory.rssMB‚ 2802.90234375 network.sent‚ 2.1098455e+07 disk.sda.in‚ 106.21875 disk.sda.out‚ 1241.1953125& disk.rbd0.in‚1.759218604441423e+13 disk.sda3.out‚ 1241.1953125 disk.drbd2165.out‚0 disk.dm-1.in‚ 106.21875 disk.sda1.out‚0" disk.dm-0.out‚141576.951171875 disk.rbd3.in‚0 disk.dm-1.out‚ 1241.1953125 disk.rbd3.out‚ 0.22412109375 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd1.in‚ 0.52392578125 cpu‚25.58818579057861 disk.drbd2165.in‚0' disk.nvme0c0n1.out‚141576.951171875% proc.memory.availableMB‚ 2813.8125 proc.cpu.threads‚17ï¬2ùŸÎ,‚(:” ¹ÂéÒÓôˆ÷ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚263.406' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚52.681200000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0­GuKÏ,‚(:À ¹ÂéÒÄÚù disk.dm-0.in‚ 43268.6640625 disk.sda1.out‚0( disk.nvme0c0n1.out‚141589.1416015625 disk.rbd1.in‚ 0.52392578125 disk.sda3.in‚ 106.21875* disk./.usagePercent‚37.756132465917005! disk.nvme0n1.in‚ 43268.6640625# proc.memory.rssMB‚ 2802.90234375 disk.rbd3.in‚0# disk.nvme0c0n1.in‚ 43268.6640625 disk.sda2.out‚0 disk.sda.out‚ 1241.63671875 disk.sda3.out‚ 1241.63671875 proc.cpu.threads‚17 network.recv‚ 4.547053e+06# disk.dm-0.out‚141589.1416015625$ memory_percent‚82.82735347747803 cpu‚25.588126286237145 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.rbd3.out‚ 0.22412109375& disk.rbd0.in‚1.759218604441423e+13* proc.memory.percent‚17.107558250427246 disk.sda.in‚ 106.21875 disk.dm-1.in‚ 106.21875) proc.memory.availableMB‚ 2813.56640625 network.sent‚ 2.1104075e+07& disk.nvme0n1.out‚141589.1416015625 disk.sda2.in‚0 disk.rbd0.out‚0 disk.sda1.in‚0$ disk./.usageGB‚562.5936584472656 disk.rbd1.out‚ 47.6982421875 disk.dm-1.out‚ 1241.63671875—àÚ“Ð,‚(:ˆ ÈÂéÒ¸‹£÷ gpu.0.gpu‚100.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚341.69' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚68.338 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‰L¨ŸIÑ,‚(:¾ ÈÂéÒ¾Õ§ú# proc.memory.rssMB‚ 2802.90234375 proc.cpu.threads‚17 disk.rbd1.in‚ 0.52392578125 network.sent‚ 2.1109302e+07 disk.rbd3.in‚0& disk.rbd0.in‚1.759218604441423e+13 cpu‚25.588093213518427 network.recv‚ 4.550133e+06 disk.dm-1.out‚ 1241.7578125 disk.drbd2165.in‚0* proc.memory.percent‚17.107558250427246) disk./.usagePercent‚37.75657126283069 disk.sda2.in‚0 disk.rbd0.out‚0 disk.sda3.out‚ 1241.7578125! disk.nvme0n1.in‚ 43269.0859375 disk.sda.out‚ 1241.7578125 disk.sda1.out‚0 disk.sda3.in‚ 106.21875 disk.drbd2165.out‚0( proc.memory.availableMB‚ 2813.5546875$ disk./.usageGB‚562.6001968383789) disk.nvme0c0n1.out‚141601.50830078125$ disk.dm-0.out‚141601.50830078125 disk.sda1.in‚0 disk.dm-0.in‚ 43269.0859375 disk.rbd1.out‚ 47.6982421875 disk.rbd3.out‚ 0.22412109375# disk.nvme0c0n1.in‚ 43269.0859375' disk.nvme0n1.out‚141601.50830078125 disk.sda2.out‚0 disk.sda.in‚ 106.21875 disk.dm-1.in‚ 106.21875$ memory_percent‚82.82742500305176~V”PžÒ,‚(:“ ×ÂéÒÒòö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚291.384' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚58.27680000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0G÷0JÓ,‚(:¿ ×ÂéÒÍå§ù network.recv‚ 4.553454e+06# disk.nvme0c0n1.in‚ 43269.2890625 disk.sda1.out‚0$ memory_percent‚82.82754421234131 network.sent‚ 2.1114387e+07 disk.rbd3.out‚ 0.22412109375 disk.sda2.in‚0 disk.sda.out‚ 1241.9765625 disk.sda3.out‚ 1241.9765625* proc.memory.percent‚17.107558250427246$ disk./.usageGB‚562.6078224182129 disk.sda2.out‚0 disk.rbd1.in‚ 0.52392578125 disk.dm-1.in‚ 106.21875 disk.dm-1.out‚ 1241.9765625$ disk.dm-0.out‚141612.25634765625 disk.drbd2165.in‚0) proc.memory.availableMB‚ 2813.53515625 disk.sda.in‚ 106.21875# proc.memory.rssMB‚ 2802.90234375 cpu‚25.588023761118254! disk.nvme0n1.in‚ 43269.2890625 disk.rbd0.out‚0 disk.dm-0.in‚ 43269.2890625 disk.sda1.in‚0 disk.drbd2165.out‚0 disk.rbd3.in‚0 proc.cpu.threads‚17) disk./.usagePercent‚37.75708302189164) disk.nvme0c0n1.out‚141612.25634765625 disk.rbd1.out‚ 47.7763671875' disk.nvme0n1.out‚141612.25634765625& disk.rbd0.in‚1.759218604441423e+13 disk.sda3.in‚ 106.21875¯H+æ4Ô,/ _wandbruntime‚18250  _runtime‚18250 ‰&ËÕ,Å trainer/global_step‚10999 % _timestamp‚1.7843080659017742e+09 _runtime‚18250.118947096 _step‚903 ' lr-AdamW/pg1‚0.00030174340970549223 ' lr-AdamW/pg2‚0.00030174340970549223²Äc,ÐÖ,Ê ' lr-AdamW/pg1‚0.00030174340970549223 ' lr-AdamW/pg2‚0.00030174340970549223 trainer/global_step‚10999 % _timestamp‚1.7843080659017742e+09 _runtime‚18250.118947096 _step‚903‡[Zc4×,/ _wandbruntime‚18250  _runtime‚18250œ ¬ (Ø,…P QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚287 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚28217990 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚3170414435732495 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10328846212111 _step‚904 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3161451006042112 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3171798928009216 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚3171797556446208 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚93834533 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚38730752 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3171797556446208 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚3161451006042112 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _runtime‚ 18250.1198076 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚3161451032215552 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚95254294 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 189088821 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3171797556446208 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚3161451006042112 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _runtime‚ 18250.1198076 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚3161451032215552 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚95254294 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 189088821 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚28217990 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚3170414435732495 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10328846212111 _step‚904 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3161451006042112 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3171798928009216 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚3171797556446208 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚93834533 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚38730752 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640ˆÓ[¡84Ú,/ _wandbruntime‚18251  _runtime‚18251(@£­'Û,N VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10347475439616 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3161729032945664 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚95262772 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚867266766945280 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 189105505 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7843080672879257e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 189115498 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚95272759 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚93842733 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚31183970 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10329769678357 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 189115498 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 406814720 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚3161729006772224 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _runtime‚18251.50532586 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10348820836352 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚59404503 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚28220528 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚93842739 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 189105505 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3170694611005319 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3172077853782016 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3172077853782016 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚3170693240976917 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚95272759 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10347475439616 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚28220533 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚31183685 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚59404213 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10348820836352 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚290 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚867266360130560 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚93842739 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚3161729032945664 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚93842733 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13347607219200 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚95262772 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7843080672879257e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 189115498 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚95272759 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚93842733 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚31183970 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10329769678357 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 189115498 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 406814720 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚3161729006772224 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _runtime‚18251.50532586 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10348820836352 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚59404503 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚28220528 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚93842739 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 189105505 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3170694611005319 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3172077853782016 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3172077853782016 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚3170693240976917 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚95272759 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10347475439616 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚28220533 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚31183685 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚59404213 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10348820836352 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚290 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚867266360130560 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚93842739 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚3161729032945664 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚93842733 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13347607219200 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚95262772 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 189105505‰é‹Š#4Ý,/ _wandbruntime‚18251  _runtime‚18251"8_ÎÞ,Œ 0loss/FRt1/assignment_loss‚1.1558478895656488 /loss/FBt1/detection_loss‚0.9855654894872087 _runtime‚18251.591668424 _step‚906 1loss/FRt2/assignment_loss‚0.34641630827885256 2loss/SRqqt2/assignment_loss‚0.2936028763529888 4loss/FBt2/assignment_loss‚1.3967408003761255e-05 /loss/FRt2/detection_loss‚1.4590299701088538 trainer/global_step‚10999 3loss/SRqqt1/assignment_loss‚0.25508044309733047 1loss/SRqqt2/detection_loss‚0.6164411371757932 %loss/total_loss‚6.759998429999806 epoch‚27 4loss/FBt1/assignment_loss‚1.1394246275999414e-05 /loss/FRt1/detection_loss‚0.4955269805398246 1loss/SRqqt1/detection_loss‚0.8046006532285588 /loss/FBt2/detection_loss‚0.3478613205104663 % _timestamp‚1.7843080673747573e+09/ÝiGß,‘ %loss/total_loss‚6.759998429999806 epoch‚27 4loss/FBt1/assignment_loss‚1.1394246275999414e-05 /loss/FRt1/detection_loss‚0.4955269805398246 1loss/SRqqt1/detection_loss‚0.8046006532285588 /loss/FBt2/detection_loss‚0.3478613205104663 % _timestamp‚1.7843080673747573e+09 0loss/FRt1/assignment_loss‚1.1558478895656488 /loss/FBt1/detection_loss‚0.9855654894872087 _runtime‚18251.591668424 _step‚906 1loss/FRt2/assignment_loss‚0.34641630827885256 2loss/SRqqt2/assignment_loss‚0.2936028763529888 4loss/FBt2/assignment_loss‚1.3967408003761255e-05 /loss/FRt2/detection_loss‚1.4590299701088538 trainer/global_step‚10999 3loss/SRqqt1/assignment_loss‚0.25508044309733047 1loss/SRqqt2/detection_loss‚0.6164411371757932ŠkÏŒžà,‚(:“ æÂéÒúçˆ÷ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚329.225' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚65.84500000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0þ|#Há,‚(:½ æÂéÒ‹Íÿù( proc.memory.availableMB‚ 2813.5234375$ memory_percent‚82.82761573791504) disk./.usagePercent‚37.75762447782655$ disk./.usageGB‚562.6158905029297$ disk.nvme0c0n1.in‚43383.12890625 disk.sda2.out‚0 disk.sda.out‚ 1242.390625# proc.memory.rssMB‚ 2802.90234375 cpu‚25.587963185482458 network.sent‚ 2.1119523e+07 disk.drbd2165.out‚0) disk.nvme0c0n1.out‚141620.05517578125 disk.rbd1.in‚ 0.52392578125 disk.sda.in‚ 106.21875 disk.dm-1.in‚ 106.21875* proc.memory.percent‚17.107558250427246 proc.cpu.threads‚17& disk.rbd0.in‚1.759218604441423e+13 disk.rbd0.out‚0 disk.sda3.in‚ 106.21875 disk.dm-1.out‚ 1242.390625 network.recv‚ 4.55678e+06$ disk.dm-0.out‚141620.05517578125 disk.drbd2165.in‚0' disk.nvme0n1.out‚141620.05517578125 disk.rbd1.out‚ 47.7763671875 disk.dm-0.in‚43383.12890625 disk.sda1.in‚0 disk.rbd3.out‚ 0.22412109375 disk.sda1.out‚0 disk.sda3.out‚ 1242.390625 disk.rbd3.in‚0" disk.nvme0n1.in‚43383.12890625 disk.sda2.in‚0q”vŸâ,‚(:” õÂéÒÞ÷ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚283.559' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚56.711800000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0'KH`Aã,‚(:¶ õÂéÒòèšú disk.sda.in‚ 106.21875 disk.dm-1.out‚ 1242.671875 disk.sda3.out‚ 1242.671875$ disk.nvme0c0n1.in‚43383.34765625& disk.nvme0c0n1.out‚141635.40234375 disk.sda2.out‚0 network.sent‚ 2.1182759e+07& disk.rbd0.in‚1.759218604441423e+13 disk.dm-0.in‚43383.34765625 disk.drbd2165.out‚0$ disk.nvme0n1.out‚141635.40234375# proc.memory.rssMB‚ 2802.90234375$ disk./.usageGB‚562.6072578430176 disk.dm-1.in‚ 106.21875 disk.sda3.in‚ 106.21875" disk.nvme0n1.in‚43383.34765625 cpu‚25.587917809301462 network.recv‚ 4.560847e+06 disk.sda.out‚ 1242.671875 disk.drbd2165.in‚0) proc.memory.availableMB‚ 2813.27734375* proc.memory.percent‚17.107558250427246 disk.rbd3.out‚ 0.22412109375 disk.sda2.in‚0 disk.rbd1.out‚ 47.8388671875 disk.rbd0.out‚0 disk.sda1.in‚0 disk.rbd3.in‚0 disk.rbd1.in‚ 0.52392578125) disk./.usagePercent‚37.75704513277657! disk.dm-0.out‚141635.40234375 disk.sda1.out‚0$ memory_percent‚82.82911777496338 proc.cpu.threads‚17ˆM ±”ä,‚(:‰ „ÃéÒÍ‹£ø gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚288.439' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚57.6878 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0»ÃöíMå,‚(: „ÃéÒþã‘ù disk.rbd1.out‚ 47.8388671875 disk.sda3.in‚ 106.21875 disk.sda.in‚ 106.21875 disk.sda.out‚ 1242.80078125# proc.memory.rssMB‚ 2802.90234375" disk.nvme0n1.in‚43383.55078125& disk.rbd0.in‚1.759218604441423e+13 disk.sda1.in‚0 disk.sda3.out‚ 1242.80078125 disk.sda2.out‚0 disk.dm-0.in‚43383.55078125$ memory_percent‚82.82761573791504 disk.drbd2165.out‚0 disk.rbd3.out‚ 0.22412109375( proc.memory.availableMB‚ 2813.5234375* proc.memory.percent‚17.107558250427246 network.recv‚ 4.563973e+06' disk.nvme0n1.out‚141643.32958984375 disk.rbd1.in‚ 0.52392578125 disk.dm-1.out‚ 1242.80078125 disk.sda1.out‚0 network.sent‚ 2.1187894e+07$ disk./.usageGB‚562.6138458251953$ disk.nvme0c0n1.in‚43383.55078125 disk.rbd0.out‚0$ disk.dm-0.out‚141643.32958984375 cpu‚25.58788985473327 proc.cpu.threads‚17) disk.nvme0c0n1.out‚141643.32958984375 disk.sda2.in‚0 disk.dm-1.in‚ 106.21875( disk./.usagePercent‚37.7574872577882 disk.drbd2165.in‚0 disk.rbd3.in‚0òˆLŸæ,‚(:” “ÃéÒÈùîö gpu.0.gpu‚99.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚181.517' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚36.303399999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0(Ññ"Gç,‚(:¼ “ÃéÒ¢µù) proc.memory.availableMB‚ 2813.51953125 disk.rbd3.out‚ 0.22412109375" disk.nvme0n1.in‚43383.97265625& disk.nvme0n1.out‚141659.4990234375 disk.sda.in‚ 106.21875* proc.memory.percent‚17.107558250427246 disk.drbd2165.out‚0 disk.sda2.out‚0 disk.rbd1.in‚ 0.52392578125 cpu‚25.588049027429992& disk.rbd0.in‚1.759218604441423e+13 disk.sda1.in‚0 disk.sda1.out‚0$ disk./.usageGB‚562.6318702697754# proc.memory.rssMB‚ 2802.90234375) disk./.usagePercent‚37.75869689338748 disk.dm-1.in‚ 106.21875$ memory_percent‚82.82763957977295 network.sent‚ 2.1192438e+07 network.recv‚ 4.567406e+06 disk.rbd0.out‚0 disk.sda3.out‚ 1243.015625 disk.drbd2165.in‚0 disk.rbd3.in‚0 disk.sda2.in‚0 disk.dm-1.out‚ 1243.015625 disk.sda3.in‚ 106.21875( disk.nvme0c0n1.out‚141659.4990234375 proc.cpu.threads‚17 disk.dm-0.in‚43383.97265625# disk.dm-0.out‚141659.4990234375$ disk.nvme0c0n1.in‚43383.97265625 disk.rbd1.out‚ 47.9169921875 disk.sda.out‚ 1243.015625È÷š¦Ÿè,‚(:” ¢ÃéÒÓµéö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚295.399' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚59.079800000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0wXú3é,‚(:¶ ¢ÃéÒ·‚¦ù disk.rbd0.out‚0* proc.memory.percent‚17.107558250427246 disk.dm-1.in‚ 106.21875& disk.rbd0.in‚1.759218604441423e+13 disk.dm-0.in‚43464.15234375 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.rbd1.in‚ 0.52392578125 disk.rbd3.in‚ögÈÂ0( disk.nvme0c0n1.out‚141668.0537109375 disk.sda2.out‚0 disk.rbd1.out‚ 47.9169921875 cpu‚25.588127313027805 proc.cpu.threads‚17 disk.sda.in‚ 106.21875# proc.memory.rssMB‚ 2802.90234375 network.recv‚ 4.57077e+06) disk./.usagePercent‚37.75900999060186 disk.sda.out‚ 1243.21875 disk.sda3.in‚ 106.21875" disk.nvme0n1.in‚43464.15234375& disk.nvme0n1.out‚141668.0537109375$ disk./.usageGB‚562.6365356445312 disk.sda3.out‚ 1243.21875 disk.rbd3.out‚ 0.22412109375$ disk.nvme0c0n1.in‚43464.15234375 disk.sda2.in‚0( proc.memory.availableMB‚ 2813.7578125$ memory_percent‚82.82618522644043 disk.dm-1.out‚ 1243.21875# disk.dm-0.out‚141668.0537109375 disk.drbd2165.out‚0 network.sent‚ 2.119821e+07þð½4ê,/ _wandbruntime‚18329  _runtime‚18329˜ JôÉë,à % _timestamp‚1.7843081451827693e+09 _runtime‚18329.399960953 _step‚907 & lr-AdamW/pg1‚0.0002989952328607167 & lr-AdamW/pg2‚0.0002989952328607167 trainer/global_step‚11049ÚG)¶Îì,È & lr-AdamW/pg1‚0.0002989952328607167 & lr-AdamW/pg2‚0.0002989952328607167 trainer/global_step‚11049 % _timestamp‚1.7843081451827693e+09 _runtime‚18329.399960953 _step‚907‹Ä –†4í,/ _wandbruntime‚18329  _runtime‚18329­400 (î,‡P JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚94244827 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚28345140 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚871120989806592 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚284 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚3173979678429184 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚59663609 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3184356131782693 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3185745280445440 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚871120582984704 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10392866334720 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10376427925541 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 % _timestamp‚1.7843081451839018e+09 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚95690019 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚95680046 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚38730752 _runtime‚18329.401234457 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 189924873 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10394211724288 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3185745280445440 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3175351068721152 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚3184354761754403 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 trainer/global_step‚11049 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚3175351042547712 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚289 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13404996097024 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3175351042547712 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚95680046 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚3185743908882432 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚94244833 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3185743908882432 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚3175351068721152 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚31318758 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚31318474 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚857715586887680 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚94244833 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13405034827776 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 189934852 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10375083325219 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚94244827 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚28345135 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 406821888 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10394211724288 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51° Ç(ï,ŒP VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚3185743908882432 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚94244833 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3185743908882432 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚3175351068721152 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚31318758 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚31318474 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚857715586887680 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚94244833 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13405034827776 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 189934852 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10375083325219 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚94244827 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚28345135 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 406821888 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10394211724288 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚94244827 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚28345140 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚871120989806592 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚284 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚3173979678429184 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚59663609 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3184356131782693 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3185745280445440 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚871120582984704 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10392866334720 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10376427925541 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 % _timestamp‚1.7843081451839018e+09 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚95690019 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚95680046 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚38730752 _runtime‚18329.401234457 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 189924873 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10394211724288 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3185745280445440 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3175351068721152 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚3184354761754403 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 trainer/global_step‚11049 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚3175351042547712 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚289 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13404996097024 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3175351042547712 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚95680046 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568Œɲ]•ð,‚(:Š ±Ãé҇ǭù gpu.0.gpu‚100.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚345.994' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚69.1988 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ò\Ê 'ñ,‚(:œ ±ÃéÒ»¸Øú$ memory_percent‚82.82475471496582" disk.nvme0n1.in‚43464.57421875 disk.sda2.out‚0 disk.dm-1.in‚ 106.21875 disk.sda3.in‚ 106.21875 disk.drbd2165.in‚0 disk.sda2.in‚0 disk.rbd1.in‚ 0.52392578125 disk.dm-0.in‚43464.57421875 disk.sda.out‚ 1243.5703125 disk.rbd0.out‚0( proc.memory.availableMB‚ 2813.9921875) disk./.usagePercent‚37.75962952883472 disk.sda1.out‚0 disk.nvme0n1.out‚141680 disk.dm-1.out‚ 1243.5703125 network.sent‚ 2.1203302e+07 disk.sda3.out‚ 1243.5703125$ disk.nvme0c0n1.in‚43464.57421875& disk.rbd0.in‚1.759218604441423e+13# proc.memory.rssMB‚ 2802.90234375 cpu‚25.58821473958426 proc.cpu.threads‚17 disk.dm-0.out‚141680 disk.drbd2165.out‚0 disk.nvme0c0n1.out‚141680 disk.sda.in‚ 106.21875 network.recv‚ 4.574103e+06$ disk./.usageGB‚562.6457672119141 disk.rbd3.out‚ 0.22412109375* proc.memory.percent‚17.107558250427246 disk.sda1.in‚0 disk.rbd3.in‚0 disk.rbd1.out‚ 47.9794921875Ì¢4ò,/ _wandbruntime‚18330  _runtime‚18330#¢vŸ—$ó,‘N D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 189941557 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3175629069451264 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 406814720 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚3184633566998825 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚59668884 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10376006791465 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3175629043277824 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13406200951808 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3184634937027227 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 trainer/global_step‚11049 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚95698511 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3186022834648064 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 189941557 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 189951550 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚94253033 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚871199235523584 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚857793441386496 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 189951550 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10377351391899 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10393791370240 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚871199642338304 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3174257585635328 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMoni&â$ƒ€tor.on_train_batch_end/active.large_pool.freed‚94253033 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3174257560207360 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚59668884 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10376006791465 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3175629043277824 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13406200951808 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3184634937027227 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 trainer/global_step‚11049 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚95698511 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3186022834648064 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 189941557 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 189951550 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚94253033 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚871199235523584 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚857793441386496 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 189951550 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10377351391899 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10393791370240 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚871199642338304 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3174257585635328 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚94253033 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3174257560207360 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 EDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚96115771 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚861590275362816 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3189251105226752 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚96115771 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚875053454583808 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚28472285 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 190760925 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10439182265344 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 406821888 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚3189251105226752 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 190770904 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3199690261318656 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚3198295087776311 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10422665038649 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 J ‡-‹P G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚96115771 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚861590275362816 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3189251105226752 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚96115771 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚875053454583808 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚28472285 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 190760925 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.sm<Ãh all_pool.freed‚10439182265344 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 406821888 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚3189251105226752 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 190770904 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3199690261318656 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚3198295087776311 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10422665038649 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3199970558654464 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 trainer/global_step‚11099 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 190777609 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚94663339 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10422243904573 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10440107300864 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10440107300864 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚875132114199552 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚96124263 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚31458512 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 190777609 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚861667761770496 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3188151649116160 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚94663333 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13464759243776 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3189529079783424 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13464720520192 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 190787602 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3199970558654464 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚861667393679360 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3188151674544128 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚913 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚59933062 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚96124263 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 $ _timestamp‚1.784308225898868e+09 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚38723584 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _runtime‚18410.116266762 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10441452697600 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10423588505007 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚94663339 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚28474828 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚3198573893020733 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3199969187084288 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚283 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚28474833 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚96114276 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 190787602.æ 'Š-•N VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚28474828 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚3198573893020733 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3199969187084288 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚283 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚28474833 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚96114276 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 190787602 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3198575263049135 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚3189529105956864 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚96114276 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3199970558654464 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 trainer/global_step‚11099 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 190777609 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚94663339 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10422243904573 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10440107300864 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10440107300864 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚875132114199552 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚96124263 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚31458512 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 190777609 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚861667761770496 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3188151649116160 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚94663333 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13464759243776 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3189529079783424 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13464720520192 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 190787602 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3199970558654464 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚861667393679360 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3188151674544128 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚913 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚59933062 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚96124263 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 $ _timestamp‚1.784308225898868e+09 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚38723584 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _runtime‚18410.116266762 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10441452697600 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10423588505007 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚94663339 QDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚31592906 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3203151141732352 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚38730752 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3212236783826509 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10485498195968 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 _step‚916 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚28599440 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚96531550 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚31593195 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚95065433 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚865464595746816 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13522228780544 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚3201767856246784 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚95065433 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 191596977 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚865464227655680 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13522190049792 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3203151115558912 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚3212235413798219 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚878986824527360 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚878986417705472 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 trainer/global_step‚11149 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚18488.03218933 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚3203151141732352 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10468902151757 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚96531550 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3213636613754880 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚60192341 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3213637985317888 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚96541523 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10486843585536 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚289 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚3213636613754880 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚3201767881674752ì/úÓ(-‹P O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚96531550 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3213636613754880 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚60192341 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3213637985317888 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚96541523 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10486843585536 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚289 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚3213636613754880 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚3201767881674752 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10467557551435 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚96541523 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚28599435 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚294 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 406821888 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚31592906 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3203151141732352 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚38730752 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3212236783826509 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10485498195968 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 _step‚916 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚28599440 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚96531550 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚31593195 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚95065433 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚865464595746816 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13522228780544 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚3201767856246784 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚95065433 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 191596977 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚865464227655680 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13522190049792 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3203151115558912 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚3212235413798219 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚878986824527360 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚878986417705472 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 trainer/global_step‚11149 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚18488.03218933 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚3203151141732352 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10468902151757”§Sð4ž-/ _wandbruntime‚18489  _runtime‚184898³S÷ÔŸ-‘N \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚3203429142462464 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 191623654 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10486423231488 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 trainer/global_step‚11149 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚288 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 191613661 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚95073633 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚95073633 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10486423231488 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13523396428288 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13523357704704 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚3212514219042641 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3202045763452928 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚96540028 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚3203429116289024 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10487768628224 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚31595923 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10468481017681 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚95073639 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10487768628224 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 _runtime‚18489.417186087 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚28601983 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _step‚917 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3213916911090688 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚879065478582784 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚96550015 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚28601978 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 % _timestamp‚1.7843083051997678e+09 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3203429142462464 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚865542082154496 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3202045738024960 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3213916911090688 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚31595640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3203429116289024 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 191623654 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚283 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚60197618 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10469825618115 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚95073639 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 406814720 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚879065071768064 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3212515589071043 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3213916911090688 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚31595640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3203429116289024 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 191623654 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚283 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚60197618 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10469825618115 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚95073639 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 406814720 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚879065071768064 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3212515589071043 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 191623654 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10486423231488 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 trainer/global_step‚11149 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚288 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 191613661 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚95073633 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚95073633 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10486423231488 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13523396428288 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13523357704704 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚3212514219042641 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3202045763452928 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚96540028 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚3203429116289024 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10487768628224 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚31595923 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10468481017681 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚95073639 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10487768628224 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 _runtime‚18489.417186087 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚28601983 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _step‚917 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3213916911090688 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚879065478582784•ëö&24¡-/ _wandbruntime‚18489  _runtime‚18489 øë³¢-‹ 1loss/FRt2/assignment_loss‚0.31996828792107046 2loss/SRqqt2/assignment_loss‚0.2845273413247936 1loss/SRqqt2/detection_loss‚0.5195784933448094 % _timestamp‚1.7843083052862601e+09 _step‚918 4loss/FBt1/assignment_loss‚0.00014081059800233488 4loss/FBt2/assignment_loss‚1.1904929509649478e-05 trainer/global_step‚11149 _runtime‚18489.503206108 2loss/SRqqt1/assignment_loss‚0.2815005514246259 /loss/FRt1/detection_loss‚0.4906922687856058 .loss/FRt2/detection_loss‚1.364151725908672 /loss/FBt1/detection_loss‚0.4251388658523419 &loss/total_loss‚5.9303621605510735 0loss/FRt1/assignment_loss‚1.1572280546252647 1loss/SRqqt1/detection_loss‚0.6652959711979193 /loss/FBt2/detection_loss‚0.4221278846384581 epoch‚27Km§ü£- _step‚918 4loss/FBt1/assignment_loss‚0.00014081059800233488 4loss/FBt2/assignment_loss‚1.1904929509649478e-05 trainer/global_step‚11149 _runtime‚18489.503206108 2loss/SRqqt1/assignment_loss‚0.2815005514246259 /loss/FRt1/detection_loss‚0.4906922687856058 .loss/FRt2/detection_loss‚1.364151725908672 /loss/FBt1/detection_loss‚0.4251388658523419 &loss/total_loss‚5.9303621605510735 0loss/FRt1/assignment_loss‚1.1572280546252647 1loss/SRqqt1/detection_loss‚0.6652959711979193 /loss/FBt2/detection_loss‚0.4221278846384581 epoch‚27 1loss/FRt2/assignment_loss‚0.31996828792107046 2loss/SRqqt2/assignment_loss‚0.2845273413247936 1loss/SRqqt2/detection_loss‚0.5195784933448094 % _timestamp‚1.7843083052862601e+09–}Ùk”¤-‚(:‰ ÖÄéÒÀˆê÷ gpu.0.gpu‚99.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚247.557' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚49.5114 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0b$EJ¥-‚(:¿ ÖÄéÒ½”ù$ disk.nvme0c0n1.in‚43624.65234375 disk.sda.in‚ 106.21875 proc.cpu.threads‚17 network.sent‚ 2.1412952e+07 disk.dm-1.out‚ 1245.9609375& disk.rbd0.in‚1.759218604441423e+13 disk.dm-0.in‚43624.65234375) disk.nvme0c0n1.out‚141806.77783203125 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda.out‚ 1245.9609375* disk./.usagePercent‚37.764327011080496 disk.sda1.out‚0 disk.rbd1.in‚ 0.52392578125# proc.memory.rssMB‚ 2802.90234375* proc.memory.percent‚17.107558250427246 disk.sda3.in‚ 106.21875 disk.sda3.out‚ 1245.9609375' disk.nvme0n1.out‚141806.77783203125 disk.rbd1.out‚ 48.3466796875" disk.nvme0n1.in‚43624.65234375 disk.drbd2165.out‚0 disk.dm-1.in‚ 106.21875 cpu‚25.58936365403374 network.recv‚ 4.613874e+06# disk./.usageGB‚562.715763092041 disk.rbd0.out‚0$ disk.dm-0.out‚141806.77783203125 disk.sda1.in‚0 disk.rbd3.out‚ 0.22412109375$ memory_percent‚82.83014297485352' proc.memory.availableMB‚ 2813.109375 disk.drbd2165.in‚0 disk.rbd3.in‚0mu”Ö”¦-‚(:‰ åÄéÒÐô÷ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚293.271' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚58.6542 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0S—ÑÞ8§-‚(:­ åÄéÒ˜ï­ø$ memory_percent‚82.83295631408691( proc.memory.availableMB‚ 2812.6484375 disk.dm-1.out‚ 1246.3828125 disk.rbd3.out‚ 0.22412109375 cpu‚25.589455499735426 disk.dm-1.in‚ 106.21875! disk.nvme0n1.out‚ 141814.78125 disk.sda2.in‚0 disk.rbd1.out‚ 48.4404296875* disk./.usagePercent‚37.764792432777746$ disk./.usageGB‚562.7226982116699 disk.dm-0.in‚ 43706.5234375* proc.memory.percent‚17.107558250427246 disk.sda.in‚ 106.21875 disk.sda3.out‚ 1246.3828125 disk.sda2.out‚0# proc.memory.rssMB‚ 2802.90234375 network.sent‚ 2.1475646e+07 network.recv‚ 4.618511e+06 disk.dm-0.out‚ 141814.78125 disk.drbd2165.in‚0 disk.drbd2165.out‚0 proc.cpu.threads‚17 disk.sda1.out‚0 disk.rbd3.in‚0# disk.nvme0c0n1.in‚ 43706.5234375 disk.rbd1.in‚ 0.52392578125 disk.sda.out‚ 1246.3828125& disk.rbd0.in‚1.759218604441423e+13 disk.rbd0.out‚0 disk.sda1.in‚0 disk.sda3.in‚ 106.21875# disk.nvme0c0n1.out‚ 141814.78125! disk.nvme0n1.in‚ 43706.5234375—ßC`”¨-‚(:‰ ôÄéÒ‹¿¯ù gpu.0.gpu‚99.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚46.0 gpu.0.powerWatts‚333.089' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚66.6178 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0µ°©@©-‚(:µ ôÄéÒÃâŠû disk.sda.in‚ 106.21875# disk.nvme0c0n1.in‚ 43706.9453125 disk.rbd0.out‚0 disk.rbd3.in‚0$ memory_percent‚82.83309936523438 disk.sda1.in‚0 disk.sda3.in‚ 106.21875 disk.sda3.out‚ 1246.5546875$ proc.memory.availableMB‚2812.625# proc.memory.rssMB‚ 2802.90234375* proc.memory.percent‚17.107558250427246 disk.sda.out‚ 1246.5546875) disk./.usagePercent‚37.76523276573664( disk.nvme0c0n1.out‚141828.9580078125 disk.sda1.out‚0 disk.drbd2165.in‚0$ disk./.usageGB‚562.7292594909668 disk.dm-1.in‚ 106.21875# disk.dm-0.out‚141828.9580078125 disk.rbd3.out‚ 0.22412109375 proc.cpu.threads‚17 network.sent‚ 2.1481305e+07 disk.sda2.in‚0 disk.rbd1.in‚ 0.52392578125 disk.dm-0.in‚ 43706.9453125 network.recv‚ 4.62168e+06 disk.sda2.out‚0 disk.rbd1.out‚ 48.4404296875! disk.nvme0n1.in‚ 43706.9453125& disk.nvme0n1.out‚141828.9580078125& disk.rbd0.in‚1.759218604441423e+13 disk.dm-1.out‚ 1246.5546875 disk.drbd2165.out‚0 cpu‚25.58959097570886WþMª-‚(: ƒÅéÒ»ÆÑú$ disk./.usageGB‚562.7336540222168& disk.rbd0.in‚1.759218604441423e+13) proc.memory.availableMB‚ 2812.85546875# proc.memory.rssMB‚ 2802.90234375 disk.dm-1.out‚ 1246.74609375 disk.rbd0.out‚0 disk.sda1.in‚0 disk.drbd2165.in‚0* proc.memory.percent‚17.107558250427246 network.sent‚ 2.1484923e+07* disk./.usagePercent‚37.765527686416085 disk.sda1.out‚0 disk.sda3.out‚ 1246.74609375 disk.rbd3.out‚ 0.22412109375' disk.nvme0n1.out‚141837.12451171875 disk.sda.out‚ 1246.74609375 disk.sda3.in‚ 106.21875 disk.sda2.out‚0 disk.rbd1.out‚ 48.5185546875$ memory_percent‚82.83169269561768 network.recv‚ 4.624357e+06 disk.rbd3.in‚0 disk.sda2.in‚0 disk.rbd1.in‚ 0.52392578125 cpu‚25.58969492702363 disk.drbd2165.out‚0! disk.nvme0n1.in‚ 43707.1484375 proc.cpu.threads‚17 disk.sda.in‚ 106.21875 disk.dm-1.in‚ 106.21875 disk.dm-0.in‚ 43707.1484375# disk.nvme0c0n1.in‚ 43707.1484375) disk.nvme0c0n1.out‚141837.12451171875$ disk.dm-0.out‚141837.12451171875&;R@«-‚(:’ ƒÅéÒ¼œŒŠ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚281.23' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚56.24600000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0'ªl•¬-‚(:Š ’ÅéÒƒçýö gpu.0.gpu‚100.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚46.0 gpu.0.powerWatts‚335.642' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚67.1284 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0D+7ÃG­-‚(:¼ ’ÅéÒ¹‡ðù disk.rbd0.out‚0 disk.dm-0.in‚ 43707.3671875# proc.memory.rssMB‚ 2802.90234375# disk./.usageGB‚562.739688873291 disk.sda3.in‚ 106.21875$ memory_percent‚82.83169269561768( disk.nvme0c0n1.out‚141847.0478515625 disk.sda.out‚ 1246.9609375 disk.dm-1.in‚ 106.21875* proc.memory.percent‚17.107558250427246 network.recv‚ 4.627456e+06* disk./.usagePercent‚37.765932690335255 disk.drbd2165.in‚0& disk.rbd0.in‚1.759218604441423e+13 proc.cpu.threads‚17# disk.dm-0.out‚141847.0478515625 disk.sda1.out‚0 disk.sda3.out‚ 1246.9609375) proc.memory.availableMB‚ 2812.85546875 network.sent‚ 2.1489997e+07 disk.sda1.in‚0 disk.rbd3.in‚0 disk.rbd1.in‚ 0.52392578125 disk.sda.in‚ 106.21875 cpu‚25.589813112780586 disk.drbd2165.out‚0 disk.sda2.in‚0 disk.rbd1.out‚ 48.5185546875& disk.nvme0n1.out‚141847.0478515625 disk.rbd3.out‚ 0.22412109375# disk.nvme0c0n1.in‚ 43707.3671875! disk.nvme0n1.in‚ 43707.3671875 disk.sda2.out‚0 disk.dm-1.out‚ 1246.9609375VU¸54®-/ _wandbruntime‚18567  _runtime‚18567]0s}ɯ-à trainer/global_step‚11199 $ _timestamp‚1.784308382944864e+09 _runtime‚18567.16210239 _step‚919 ' lr-AdamW/pg1‚0.00029076949235733916 ' lr-AdamW/pg2‚0.00029076949235733916¸7É Î°-È ' lr-AdamW/pg2‚0.00029076949235733916 trainer/global_step‚11199 $ _timestamp‚1.784308382944864e+09 _runtime‚18567.16210239 _step‚919 ' lr-AdamW/pg1‚0.00029076949235733916—J%Ü4±-/ _wandbruntime‚18567  _runtime‚18567Ò¡t8w&²-‡P _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10533159516160 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚96957302 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚269 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3226177109848417 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10533159516160 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404724736 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚869338916130816 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36633600 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3227584337754112 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚264 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚3217051152064512 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13580983958016 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚3226175739820127 % _timestamp‚1.7843083829460979e+09 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3227584337754112 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 192443008 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 192433029 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 _runtime‚18567.163676128 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚3227582966191104 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚3217051178237952 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚95475727 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13581020591616 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚3215661970583552 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚60457452 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚31731131 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚31730867 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚95475733 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3227582966191104 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3217051178237952 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10515139264865 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚869338548039680 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚3215661945155584 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10531814126592 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚28726590 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚882919936722432 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚882919531997696 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚96957302 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3217051152064512 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚95475733 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚95475733 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚60457452 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚31731131 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚31730867 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚95475733 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3227582966191104 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3217051178237952 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10515139264865 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚869338548039680 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚3215661945155584 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10531814126592 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚28726590 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚882919936722432 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚882919531997696 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627˜¨¤¨T4´-/ _wandbruntime‚18568  _runtime‚18568Œè{>'µ-‘N D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 192449713 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3227863263526912 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚31733843 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 192459706 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚28729133 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚869416402538496 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3215939852361728 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 192459706 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 192449713 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13582161916416 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚18568.549141212 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚263 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3215939826933760 _step‚921 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚869416034447360 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10534084558848 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3217329178968064 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13582198542848 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚60462713 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404717568 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648!GQ'¶-–N H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚263 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3215939826933760 _step‚921 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚869416034447360 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10534084558848 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3217329178968064 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13582198542848 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚60462713 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404717568 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 192449713 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3227863263526912 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚31733843 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 192459706 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚28729133 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚869416402538496 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3215939852361728 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 192459706 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 192449713 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13582161916416 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚18568.549141212™\¶šÑ4·-/ _wandbruntime‚18568  _runtime‚18568-z€¸-‹ /loss/FRt1/detection_loss‚0.5032476000928876 /loss/FRt2/detection_loss‚1.4321792691620068 /loss/FBt1/detection_loss‚0.3727486293968925 trainer/global_step‚11199 _runtime‚18568.634084948 1loss/FRt2/assignment_loss‚0.37737545758485797 3loss/FBt1/assignment_loss‚1.980297850635378e-06 1loss/SRqqt1/detection_loss‚0.5737029340900064 epoch‚27 _step‚922 3loss/SRqqt2/assignment_loss‚0.31709267199587854 /loss/FBt2/detection_loss‚0.4330709553003544 % _timestamp‚1.7843083844169486e+09 1loss/SRqqt2/detection_loss‚0.5768780985594855 %loss/total_loss‚6.021090325865785 /loss/FRt1/assignment_loss‚1.183502290741544 3loss/SRqqt1/assignment_loss‚0.25128675020846675 4loss/FBt2/assignment_loss‚3.6884355546717985e-06/e$ع- %loss/total_loss‚6.021090325865785 /loss/FRt1/assignment_loss‚1.183502290741544 3loss/SRqqt1/assignment_loss‚0.25128675020846675 4loss/FBt2/assignment_loss‚3.6884355546717985e-06 /loss/FRt1/detection_loss‚0.5032476000928876 /loss/FRt2/detection_loss‚1.4321792691620068 /loss/FBt1/detection_loss‚0.3727486293968925 trainer/global_step‚11199 _runtime‚18568.634084948 1loss/FRt2/assignment_loss‚0.37737545758485797 3loss/FBt1/assignment_loss‚1.980297850635378e-06 1loss/SRqqt1/detection_loss‚0.5737029340900064 epoch‚27 _step‚922 3loss/SRqqt2/assignment_loss‚0.31709267199587854 /loss/FBt2/detection_loss‚0.4330709553003544 % _timestamp‚1.7843083844169486e+09 1loss/SRqqt2/detection_loss‚0.5768780985594855š1¿ž{Wº-‚(:Š ¡ÅéÒŒ½õö gpu.0.gpu‚100.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚40.0 gpu.0.powerWatts‚110.527' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.1054 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.> (µ>correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0^m‡û6»-‚(:« ¡ÅéÒ¥¥‚ú network.sent‚ 2.1495768e+07 disk.sda.out‚ 1247.59375 disk.dm-1.out‚ 1247.59375 disk.sda3.in‚ 106.21875# disk.nvme0c0n1.in‚ 43800.9921875$ memory_percent‚83.14604759216309( proc.memory.availableMB‚ 2761.3515625( proc.memory.percent‚17.4210786819458$ disk./.usageGB‚562.7465438842773 disk.rbd1.in‚ 0.52392578125 disk.rbd1.out‚ 48.5810546875# proc.memory.rssMB‚ 2854.26953125 cpu‚25.590036039558385 disk.sda1.out‚0 disk.drbd2165.out‚0 disk.rbd3.out‚ 0.22412109375# disk.nvme0n1.out‚141859.3203125 disk.sda2.in‚0& disk.rbd0.in‚1.759218604441423e+13 disk.sda1.in‚0 disk.sda3.out‚ 1247.59375 disk.rbd3.in‚0% disk.nvme0c0n1.out‚141859.3203125* disk./.usagePercent‚37.766392735874284 disk.sda.in‚ 106.21875 disk.rbd0.out‚0 disk.dm-0.out‚141859.3203125 network.recv‚ 4.631216e+06 disk.dm-1.in‚ 106.21875! disk.nvme0n1.in‚ 43800.9921875 disk.sda2.out‚0 proc.cpu.threads‚17 disk.dm-0.in‚ 43800.9921875 disk.drbd2165.in‚0‚Ÿ¼-‚(:” °ÅéÒçëõö gpu.0.gpu‚100.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚315.76' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚63.151999999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ܹ¤KG½-‚(:¼ °ÅéÒ£åÏù network.sent‚ 2.1558458e+07$ disk./.usageGB‚562.7512817382812 disk.sda.in‚ 106.21875 disk.sda.out‚ 1247.87109375 disk.dm-1.out‚ 1247.87109375& disk.rbd0.in‚1.759218604441423e+13 disk.sda3.in‚ 106.21875 disk.sda2.in‚0 disk.dm-1.in‚ 106.21875 disk.rbd0.out‚0# disk.nvme0c0n1.in‚ 43801.1953125' disk.nvme0c0n1.out‚141868.201171875! disk.nvme0n1.in‚ 43801.1953125 disk.rbd1.in‚ 0.52392578125) proc.memory.availableMB‚ 2812.55078125) disk./.usagePercent‚37.76671069723181* proc.memory.percent‚17.107558250427246 disk.sda1.in‚0% disk.nvme0n1.out‚141868.201171875 proc.cpu.threads‚17 disk.dm-0.in‚ 43801.1953125" disk.dm-0.out‚141868.201171875 disk.sda3.out‚ 1247.87109375 disk.rbd3.out‚ 0.22412109375 disk.sda1.out‚0 disk.rbd3.in‚0 cpu‚25.592020105665462 network.recv‚ 4.635677e+06 disk.drbd2165.in‚0# proc.memory.rssMB‚ 2802.90234375 disk.drbd2165.out‚0 disk.sda2.out‚0 disk.rbd1.out‚ 48.5810546875$ memory_percent‚82.83355236053467*ã““4¾-/ _wandbruntime‚18596  _runtime‚18596–H—Ï=¿-É{ 8"Purity/0FRt2SRqqt*FBt/event_purity‚0.403922438621521 6 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.998476505279541 8"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.581828773021698 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5668392777442932 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6531379818916321 6 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.995943546295166 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.8967061638832092 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4673430025577545 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.15493787825107574 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9351851344108582 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.6936697363853455 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 % _timestamp‚1.7843084118789952e+09 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9814814329147339 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.5632035136222839 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6314482092857361 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.44907405972480774 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.04289275407791138 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7008242607116699 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.4716948866844177 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7106764912605286 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.5714285969734192 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9877339601516724 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6524994969367981 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.6936697363853455 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.2395833283662796 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.40740740299224854 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9950438141822815 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7193121910095215 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5965425968170166 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.024941980838775635 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.18694815039634705 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9814814329147339 , Purity/2FRt0SRqqt1FBt/FRt_purity‚0.59375 particle/accuracy_4_of_5‚1 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.2528342604637146 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9992673993110657 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.18031537532806396 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.6356198191642761 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6497074961662292 particle/accuracy_2_of_4‚1 +particle/sensitivity‚0.6158433556556702 jet/accuracy_2_of_5‚1 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9946916699409485 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.11033008247613907 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4208793640136719 8"Purity/0FRt*SRqqt1FBt/event_purity‚0.855756402015686 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9988662004470825 jet/accuracy_3_of_5‚1 trainer/global_step‚11199 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.5935657620429993 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.6951097249984741 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 *jet/accuracy_2_of_4‚0.7332520484924316 ,jet/accuracy_4_of_4‚0.013899499550461769 6 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.998015820980072 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6449155211448669 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.16074016690254211 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.1969214230775833 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6450320482254028 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8381924629211426 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9940670728683472 particle/accuracy_1_of_1‚1 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9830084443092346 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6284459233283997 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9994588494300842 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.036176104098558426 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.1878093183040619 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3317480683326721 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7685185074806213 ) Purity/2FRt*SRqqt2FBt/FRt_purity‚0.75 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9958674311637878 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5826573371887207 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 *jet/accuracy_1_of_4‚0.9685646295547485 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9856237769126892 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.02645418792963028 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.28780585527420044 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.03214805945754051 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5513209700584412 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5668392777442932 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.11654725670814514 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7219027280807495 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 particle/accuracy_3_of_4‚1 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.7102419137954712 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.1415655016899109 (particle/accuracy‚0.8431223034858704 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.11538461595773697 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.11538461595773697 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.16002783179283142 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9923442602157593 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.19196780025959015 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7685185074806213 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2823280692100525 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 particle/accuracy_2_of_5‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.18603859841823578 6 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.988095223903656 'particle/f_score‚0.6277107000350952 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9941692352294922 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8888888955116272 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 *jet/accuracy_1_of_2‚0.7059761881828308 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4662889838218689 7 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.0358964242041111 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0072327470406889915 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8010379076004028 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.19983811676502228 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 +jet/accuracy_1_of_1‚0.28997182846069336 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8931018114089966 /particle/accuracy_4_of_4‚0.9399527311325073 jet/accuracy_5_of_5‚0 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6190650463104248 , Purity/0FRt2SRqqt*FBt/FBt_purity‚0.96875 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.1732892245054245 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.17678898572921753 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8010379076004028 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6475316882133484 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 6validation_average_jet_accuracy‚0.3316146433353424 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9929356575012207 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7216599583625793 +particle/specificity‚0.9052814245223999 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 *jet/accuracy_2_of_3‚0.5149620771408081 particle/accuracy_2_of_3‚1 jet/accuracy_4_of_5‚0 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 +jet/accuracy_3_of_3‚0.06489287316799164 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.17308542132377625 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6180349588394165 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9140046238899231 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9992673993110657 , Purity/0FRt2SRqqt1FBt/FBt_purity‚0.96875 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.1733854115009308 )loss/val_total_loss‚8.524471274345931 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.24368572235107422 particle/accuracy_2_of_2‚1 _step‚923 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9981684684753418 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.1878093183040619 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.10666666179895401 *jet/accuracy_1_of_3‚0.8971330523490906 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5229381322860718 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.013951537199318409 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7342361807823181 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9983081817626953 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.03258246183395386 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6085898876190186 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6095284223556519 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.09965094923973083 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5208333730697632 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6175739169120789 6 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.543822705745697 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6726809144020081 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6179105043411255 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.04289275407791138 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4140625 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 *"Purity/2FRt*SRqqt2FBt/event_purity‚0.5 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7857142686843872 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5697435140609741 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.03624043986201286 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9971989393234253 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.01344817690551281 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 particle/accuracy_1_of_5‚1 _runtime‚18596.096942802 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.15229997038841248 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.1718490719795227 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 particle/accuracy_3_of_5‚1 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9712474346160889 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6407579779624939 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.1275511533021927 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 particle/accuracy_1_of_3‚1 6 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.942307710647583 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6927760243415833 epoch‚27 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.18252962827682495 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.1994808316230774 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7308251261711121 +jet/accuracy_3_of_4‚0.09238454699516296 particle/accuracy_1_of_2‚1 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6285345554351807 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3499999940395355 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7215493321418762 particle/accuracy_1_of_4‚1 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.19966697692871094 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5529528260231018 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6446910500526428 +jet/accuracy_2_of_2‚0.14112427830696106 particle/accuracy_3_of_3‚1 jet/accuracy_1_of_5‚1 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9831113219261169 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9906229376792908 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_5_of_5‚0 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.4160252511501312 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3354426324367523 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7256209254264832 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1(%cïr7À-Î{ particle/accuracy_4_of_5‚1 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.2528342604637146 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9992673993110657 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.18031537532806396 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.6356198191642761 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6497074961662292 particle/accuracy_2_of_4‚1 +particle/sensitivity‚0.6158433556556702 jet/accuracy_2_of_5‚1 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9946916699409485 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.11033008247613907 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4208793640136719 8"Purity/0FRt*SRqqt1FBt/event_purity‚0.855756402015686 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9988662004470825 jet/accuracy_3_of_5‚1 trainer/global_step‚11199 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.5935657620429993 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.6951097249984741 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 *jet/accuracy_2_of_4‚0.7332520484924316 ,jet/accuracy_4_of_4‚0.013899499550461769 6 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.998015820980072 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6449155211448669 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.16074016690254211 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.1969214230775833 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6450320482254028 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8381924629211426 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9940670728683472 particle/accuracy_1_of_1‚1 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9830084443092346 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6284459233283997 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9994588494300842 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.036176104098558426 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.1878093183040619 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3317480683326721 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7685185074806213 ) Purity/2FRt*SRqqt2FBt/FRt_purity‚0.75 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9958674311637878 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5826573371887207 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 *jet/accuracy_1_of_4‚0.9685646295547485 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9856237769126892 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.02645418792963028 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.28780585527420044 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.03214805945754051 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5513209700584412 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5668392777442932 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.11654725670814514 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7219027280807495 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 particle/accuracy_3_of_4‚1 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.7102419137954712 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.1415655016899109 (particle/accuracy‚0.8431223034858704 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.11538461595773697 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.11538461595773697 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.16002783179283142 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9923442602157593 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.19196780025959015 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7685185074806213 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2823280692100525 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 particle/accuracy_2_of_5‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.18603859841823578 6 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.988095223903656 'particle/f_score‚0.6277107000350952 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9941692352294922 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8888888955116272 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 *jet/accuracy_1_of_2‚0.7059761881828308 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4662889838218689 7 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.0358964242041111 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0072327470406889915 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8010379076004028 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.19983811676502228 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 +jet/accuracy_1_of_1‚0.28997182846069336 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8931018114089966 /particle/accuracy_4_of_4‚0.9399527311325073 jet/accuracy_5_of_5‚0 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6190650463104248 , Purity/0FRt2SRqqt*FBt/FBt_purity‚0.96875 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.1732892245054245 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.17678898572921753 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8010379076004028 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6475316882133484 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 6validation_average_jet_accuracy‚0.3316146433353424 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9929356575012207 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7216599583625793 +particle/specificity‚0.9052814245223999 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 *jet/accuracy_2_of_3‚0.5149620771408081 particle/accuracy_2_of_3‚1 jet/accuracy_4_of_5‚0 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 +jet/accuracy_3_of_3‚0.06489287316799164 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.17308542132377625 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6180349588394165 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9140046238899231 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9992673993110657 , Purity/0FRt2SRqqt1FBt/FBt_purity‚0.96875 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.1733854115009308 )loss/val_total_loss‚8.524471274345931 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.24368572235107422 particle/accuracy_2_of_2‚1 _step‚923 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9981684684753418 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.1878093183040619 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.10666666179895401 *jet/accuracy_1_of_3‚0.8971330523490906 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5229381322860718 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.013951537199318409 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7342361807823181 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9983081817626953 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.03258246183395386 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6085898876190186 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.6095284223556519 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.09965094923973083 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5208333730697632 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6175739169120789 6 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.543822705745697 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6726809144020081 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6179105043411255 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.04289275407791138 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4140625 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 *"Purity/2FRt*SRqqt2FBt/event_purity‚0.5 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7857142686843872 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5697435140609741 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.03624043986201286 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9971989393234253 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.01344817690551281 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 particle/accuracy_1_of_5‚1 _runtime‚18596.096942802 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.15229997038841248 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.1718490719795227 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 particle/accuracy_3_of_5‚1 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9712474346160889 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6407579779624939 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.1275511533021927 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 particle/accuracy_1_of_3‚1 6 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.942307710647583 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6927760243415833 epoch‚27 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.18252962827682495 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.1994808316230774 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7308251261711121 +jet/accuracy_3_of_4‚0.09238454699516296 particle/accuracy_1_of_2‚1 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6285345554351807 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3499999940395355 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7215493321418762 particle/accuracy_1_of_4‚1 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.19966697692871094 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5529528260231018 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6446910500526428 +jet/accuracy_2_of_2‚0.14112427830696106 particle/accuracy_3_of_3‚1 jet/accuracy_1_of_5‚1 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9831113219261169 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9906229376792908 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_5_of_5‚0 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.4160252511501312 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3354426324367523 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7256209254264832 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 8"Purity/0FRt2SRqqt*FBt/event_purity‚0.403922438621521 6 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.998476505279541 8"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.581828773021698 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5668392777442932 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6531379818916321 6 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.995943546295166 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.8967061638832092 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4673430025577545 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.15493787825107574 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9351851344108582 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.6936697363853455 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 % _timestamp‚1.7843084118789952e+09 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9814814329147339 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.5632035136222839 9"Purity/1FRt1SRqqt*FBt/SRqqt_[i„¹bpurity‚0.6314482092857361 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.44907405972480774 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.04289275407791138 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7008242607116699 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.4716948866844177 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7106764912605286 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.5714285969734192 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9877339601516724 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6524994969367981 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.6936697363853455 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.2395833283662796 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.40740740299224854 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9950438141822815 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7193121910095215 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5965425968170166 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.024941980838775635 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.18694815039634705 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9814814329147339 , Purity/2FRt0SRqqt1FBt/FRt_purity‚0.59375›;¼LÜ•Á-‚(:Š ¿ÅéÒÙ׈ø gpu.0.gpu‚100.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚233.548' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚46.7096 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0þzMqLÂ-‚(:Á ¿ÅéÒ†ñ®ù$ disk.dm-0.out‚141876.20166015625 disk.sda1.out‚0 proc.cpu.threads‚17 disk.rbd3.in‚0' disk.nvme0n1.out‚141876.20166015625 network.sent‚ 2.1564066e+07 disk.drbd2165.in‚0 disk.drbd2165.out‚0# disk.nvme0c0n1.in‚ 43801.6171875 disk.sda2.out‚0 disk.rbd1.out‚ 48.6591796875 disk.dm-1.out‚ 1248.14453125 cpu‚25.593745607897148 disk.rbd1.in‚ 0.52392578125 disk.sda.out‚ 1248.14453125 disk.sda3.in‚ 106.21875# proc.memory.rssMB‚ 2803.55859375* proc.memory.percent‚17.111563682556152 disk.sda2.in‚0 disk.sda.in‚ 106.21875 disk.sda3.out‚ 1248.14453125 network.recv‚ 4.639145e+06! disk.nvme0n1.in‚ 43801.6171875& disk.rbd0.in‚1.759218604441423e+13$ memory_percent‚82.83686637878418 disk.rbd3.out‚ 0.22412109375) disk.nvme0c0n1.out‚141876.20166015625 disk.dm-0.in‚ 43801.6171875 disk.sda1.in‚0( proc.memory.availableMB‚ 2812.0078125) disk./.usagePercent‚37.76738476506946$ disk./.usageGB‚562.7613258361816 disk.dm-1.in‚ 106.21875 disk.rbd0.out‚0å©o·”Ã-‚(:‰ ÎÅéÒ£º‡÷ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚277.374' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.4748 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0õ²øÌKÄ-‚(:À ÎÅéÒûÂðù disk.rbd1.out‚ 48.6591796875 disk.sda.in‚ 106.21875 disk.sda1.out‚0 disk.sda3.out‚ 1248.3515625* proc.memory.percent‚17.111563682556152 disk.rbd3.out‚ 0.22412109375* disk./.usagePercent‚37.767678661718776$ disk./.usageGB‚562.7657051086426) disk.nvme0c0n1.out‚141892.27392578125& disk.rbd0.in‚1.759218604441423e+13 disk.rbd0.out‚0 disk.drbd2165.in‚0 proc.cpu.threads‚17 network.recv‚ 4.643275e+06' disk.nvme0n1.out‚141892.27392578125 disk.sda.out‚ 1248.3515625 disk.dm-0.in‚ 43801.8359375 disk.drbd2165.out‚0# proc.memory.rssMB‚ 2803.55859375 network.sent‚ 2.1621814e+07 disk.sda2.in‚0 disk.dm-1.out‚ 1248.3515625$ memory_percent‚82.83669948577881# disk.nvme0c0n1.in‚ 43801.8359375 disk.sda2.out‚0$ disk.dm-0.out‚141892.27392578125 disk.rbd1.in‚ 0.52392578125 disk.dm-1.in‚ 106.21875 disk.sda1.in‚0 disk.sda3.in‚ 106.21875 disk.rbd3.in‚0) proc.memory.availableMB‚ 2812.03515625 cpu‚25.593775949533786! disk.nvme0n1.in‚ 43801.8359375îðu”Å-‚(:‰ ÝÅéÒ¢Ëòö gpu.0.gpu‚99.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚46.0 gpu.0.powerWatts‚330.673' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚66.1346 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0…!ÓJÆ-‚(:¿ ÝÅéÒ¤©Åù disk.dm-1.in‚ 106.21875 disk.sda3.in‚ 106.21875 disk.rbd3.out‚ 0.22412109375! disk.nvme0n1.in‚ 43967.5859375' disk.nvme0n1.out‚141900.08837890625 disk.sda3.out‚ 1248.66796875 network.recv‚ 4.646961e+06) disk.nvme0c0n1.out‚141900.08837890625 disk.sda1.out‚0 disk.sda2.in‚0& disk.rbd0.in‚1.759218604441423e+13$ disk.dm-0.out‚141900.08837890625$ memory_percent‚82.83605575561523 proc.cpu.threads‚17 disk.drbd2165.in‚0 disk.dm-1.out‚ 1248.66796875' proc.memory.availableMB‚ 2812.140625 cpu‚25.59391597554888 network.sent‚ 2.1626896e+07) disk./.usagePercent‚37.76815278767218 disk.sda2.out‚0 disk.rbd1.in‚ 0.52392578125 disk.sda.in‚ 106.21875 disk.sda.out‚ 1248.66796875 disk.rbd0.out‚0 disk.sda1.in‚0 disk.rbd1.out‚ 48.7216796875 disk.dm-0.in‚ 43967.5859375# proc.memory.rssMB‚ 2803.55859375 disk.drbd2165.out‚0* proc.memory.percent‚17.111563682556152$ disk./.usageGB‚562.7727699279785 disk.rbd3.in‚0# disk.nvme0c0n1.in‚ 43967.5859375úë.cŸÇ-‚(:” ìÅéÒ˜áýö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚272.324' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚54.464800000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0°_º„JÈ-‚(:¿ ìÅéÒ½­Šù' disk.nvme0n1.out‚141914.33642578125 disk.sda.out‚ 1248.90234375 disk.dm-1.in‚ 106.21875 disk.rbd0.out‚0 disk.dm-0.in‚ 43968.0078125$ memory_percent‚82.83772468566895* proc.memory.percent‚17.111563682556152 cpu‚25.59398911488508 disk.drbd2165.in‚0# disk.nvme0c0n1.in‚ 43968.0078125 disk.rbd1.in‚ 0.52392578125& disk.rbd0.in‚1.759218604441423e+13 proc.cpu.threads‚17! disk.nvme0n1.in‚ 43968.0078125 disk.sda2.out‚0 disk.rbd1.out‚ 48.7216796875$ disk.dm-0.out‚141914.33642578125 disk.sda3.in‚ 106.21875# proc.memory.rssMB‚ 2803.55859375 disk.sda3.out‚ 1248.90234375 network.sent‚ 2.1632077e+07* disk./.usagePercent‚37.768598496789295# disk./.usageGB‚562.779411315918 disk.drbd2165.out‚0 disk.rbd3.in‚0 disk.dm-1.out‚ 1248.90234375) disk.nvme0c0n1.out‚141914.33642578125 disk.sda2.in‚0 disk.rbd3.out‚ 0.22412109375 disk.sda.in‚ 106.21875 disk.sda1.in‚0( proc.memory.availableMB‚ 2811.8671875 network.recv‚ 4.65023e+06 disk.sda1.out‚0xü,•É-‚(:Š ûÅéÒ¿êô÷ gpu.0.gpu‚100.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚46.0 gpu.0.powerWatts‚342.642' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚68.5284 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¹±IÊ-‚(:¾ ûÅéÒª·ëø disk.sda2.in‚0 disk.rbd1.out‚ 48.7998046875 network.sent‚ 2.1637315e+07 disk.dm-1.in‚ 106.21875# disk.dm-0.out‚141922.9091796875 disk.sda1.out‚0) proc.memory.availableMB‚ 2811.83984375 cpu‚25.59414182232384 disk.sda.out‚ 1249.11328125& disk.rbd0.in‚1.759218604441423e+13 disk.rbd3.in‚0 disk.rbd3.out‚ 0.22412109375$ memory_percent‚82.83789157867432* proc.memory.percent‚17.111563682556152 disk.sda3.in‚ 106.21875 network.recv‚ 4.653783e+06) disk./.usagePercent‚37.76878896639477 disk.dm-1.out‚ 1249.11328125 disk.rbd0.out‚0 disk.sda1.in‚0 disk.sda3.out‚ 1249.11328125 disk.drbd2165.out‚0# proc.memory.rssMB‚ 2803.55859375 disk.dm-0.in‚ 43968.2265625 disk.drbd2165.in‚0 proc.cpu.threads‚17$ disk./.usageGB‚562.7822494506836( disk.nvme0c0n1.out‚141922.9091796875! disk.nvme0n1.in‚ 43968.2265625 disk.sda.in‚ 106.21875# disk.nvme0c0n1.in‚ 43968.2265625& disk.nvme0n1.out‚141922.9091796875 disk.sda2.out‚0 disk.rbd1.in‚ 0.52392578125Vr£r4Ë-/ _wandbruntime‚18673  _runtime‚18673³Ì›ÉÌ-à _step‚924 & lr-AdamW/pg1‚0.0002880344292277215 & lr-AdamW/pg2‚0.0002880344292277215 trainer/global_step‚11249 % _timestamp‚1.7843084896607976e+09 _runtime‚18673.877973497•›×ÎÍ-È & lr-AdamW/pg2‚0.0002880344292277215 trainer/global_step‚11249 % _timestamp‚1.7843084896607976e+09 _runtime‚18673.877973497 _step‚924 & lr-AdamW/pg1‚0.0002880344292277215œ C„&4Î-/ _wandbruntime‚18673  _runtime‚18673=÷V (Ï-‡P G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3245364514236928 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10565860909088 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚11249 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 193460069 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 193470048 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚97442036 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚3233381942996992 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10567205509410 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚246 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3234779170004992 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚96018033 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3245363142673920 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚3234779196178432 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10583972668928 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10585318058496 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚28862653 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚886909638002688 % _timestamp‚1.7843084896618414e+09 _step‚925 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13648289241088 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3243949173934370 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚60763935 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚3233381968424960 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚96018039 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3245364514236928 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚96018039 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚31901031 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚97452009 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚31901277 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3245364514236928 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10565860909088 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚11249 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 193460069 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 193470048 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚97442036 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚3233381942996992 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10567205509410 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚246 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3234779170004992 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚96018033 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3245363142673920 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚3234779196178432 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10583972668928 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10585318058496 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚28862653 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚886909638002688 % _timestamp‚1.7843084896618414e+09 _step‚925 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13648289241088 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3243949173934370 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚60763935 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚3233381968424960 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚96018039 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3245364514236928 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚96018039 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚31901031 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚97452009 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚31901277 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 JDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3233659824775168 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚60768991 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10584897704448 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3235057196908544 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚252 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚3235057170735104 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 193486746 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚873338467078144 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3233659850203136 _runtime‚18675.265076811 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚97450514 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚97450514 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚3235057196908544 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚31903795 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 193476753 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13649488970752 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 193486746 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚97460501 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚3244226609150502 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 trainer/global_step‚11249 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10586243101184 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚926 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚28865196 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 193476753 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13649454441472 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚886987921519616 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3245643440009728 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚96026245 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚873338835169280 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚28865201 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚247 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0ÏÔT:'Õ-•N UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚97460501 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚3244226609150502 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 trainer/global_step‚11249 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10586243101184 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚926 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚28865196 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 193476753 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13649454441472 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚886987921519616 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3245643440009728 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚96026245 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚873338835169280 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚28865201 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚247 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10586243101184 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚97460501 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3233659824775168 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚60768991 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10584897704448 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3235057196908544 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚252 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚3235057170735104 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 193486746 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚873338467078144 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3233659850203136 _runtime‚18675.265076811 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚97450514 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚97450514 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚3235057196908544 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚31903795 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 193476753 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13649488970752 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 193486746žÞalË4Ö-/ _wandbruntime‚18675  _runtime‚18675D®²Ð×- 4loss/FBt2/assignment_loss‚2.7920015319730683e-05 1loss/SRqqt2/detection_loss‚0.5406197586028146 0loss/FBt1/detection_loss‚0.23804554441770495 /loss/FBt2/detection_loss‚0.3633509010903842 0loss/FRt1/assignment_loss‚1.1368569736275376 1loss/FRt2/assignment_loss‚0.24052860724835912 1loss/SRqqt1/detection_loss‚0.5828232037633975 _runtime‚18675.350476143 _step‚927 4loss/FBt1/assignment_loss‚0.00013677615717297873 /loss/FRt1/detection_loss‚0.4704391202544375 trainer/global_step‚11249 /loss/FRt2/detection_loss‚1.4370858659791839 % _timestamp‚1.7843084911335514e+09 &loss/total_loss‚5.4318106157909405 epoch‚28 3loss/SRqqt1/assignment_loss‚0.19181714119244506 3loss/SRqqt2/assignment_loss‚0.23007880344218365W7þXØ-” &loss/total_loss‚5.4318106157909405 epoch‚28 3loss/SRqqt1/assignment_loss‚0.19181714119244506 3loss/SRqqt2/assignment_loss‚0.23007880344218365 4loss/FBt2/assignment_loss‚2.7920015319730683e-05 1loss/SRqqt2/detection_loss‚0.5406197586028146 0loss/FBt1/detection_loss‚0.23804554441770495 /loss/FBt2/detection_loss‚0.3633509010903842 0loss/FRt1/assignment_loss‚1.1368569736275376 1loss/FRt2/assignment_loss‚0.24052860724835912 1loss/SRqqt1/detection_loss‚0.5828232037633975 _runtime‚18675.350476143 _step‚927 4loss/FBt1/assignment_loss‚0.00013677615717297873 /loss/FRt1/detection_loss‚0.4704391202544375 trainer/global_step‚11249 /loss/FRt2/detection_loss‚1.4370858659791839 % _timestamp‚1.7843084911335514e+09ŸHyŸè‘Ù-‚(:† ™ÆéÒÝÓÑù gpu.0.gpu‚86.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚338.35' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚67.67 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‡VŸ>Ú-‚(:³ ™ÆéÒëÙàú disk.dm-0.in‚44135.02734375$ memory_percent‚82.83958435058594& disk.rbd0.in‚1.759218604441423e+13% disk.nvme0c0n1.out‚141946.8046875" disk.nvme0n1.in‚44135.02734375* proc.memory.percent‚17.111563682556152 network.sent‚ 2.169485e+07 network.recv‚ 4.661727e+06 disk.dm-1.in‚ 106.21875 disk.rbd1.in‚ 0.52392578125 disk.sda2.out‚0 disk.sda.in‚ 106.21875 disk.dm-1.out‚ 1249.44921875 disk.dm-0.out‚141946.8046875 disk.sda3.in‚ 106.21875# proc.memory.rssMB‚ 2803.55859375) disk./.usagePercent‚37.76970470534475 disk.rbd0.out‚0 disk.sda1.in‚0# disk.nvme0n1.out‚141946.8046875 disk.rbd1.out‚ 48.9248046875 proc.cpu.threads‚17$ disk./.usageGB‚562.7958946228027$ disk.nvme0c0n1.in‚44135.02734375 disk.sda.out‚ 1249.44921875 disk.sda1.out‚0 disk.rbd3.in‚0 disk.sda2.in‚0 disk.sda3.out‚ 1249.44921875 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.rbd3.out‚ 0.22412109375% proc.memory.availableMB‚ 2811.5625 cpu‚25.59434613695118@ÎóžÛ-‚(:“ ¨ÆéÒ¯¿÷ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚272.542' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚54.50839999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ï_HÜ-‚(:½ ¨ÆéÒóå…û disk.sda.out‚ 1249.984375 disk.dm-1.out‚ 1249.984375 disk.sda1.in‚0 disk.sda2.in‚0 disk.sda2.out‚0$ disk./.usageGB‚562.8029098510742 disk.sda.in‚ 106.21875 disk.sda1.out‚0" disk.nvme0n1.in‚44135.24609375 disk.rbd1.out‚ 48.9248046875* proc.memory.percent‚17.111563682556152* disk./.usagePercent‚37.770175503200214 disk.rbd0.out‚0 disk.sda3.out‚ 1249.984375 disk.rbd3.out‚ 0.22412109375 disk.dm-0.in‚44135.24609375 disk.sda3.in‚ 106.21875 disk.drbd2165.out‚0$ disk.nvme0c0n1.in‚44135.24609375' disk.nvme0n1.out‚141958.35400390625& disk.rbd0.in‚1.759218604441423e+13# proc.memory.rssMB‚ 2803.55859375 network.sent‚ 2.1746931e+07 disk.dm-1.in‚ 106.21875$ disk.dm-0.out‚141958.35400390625) disk.nvme0c0n1.out‚141958.35400390625' proc.memory.availableMB‚ 2811.578125 cpu‚25.594456675169784 disk.drbd2165.in‚0 disk.rbd3.in‚0 disk.rbd1.in‚ 0.52392578125# memory_percent‚82.8394889831543 proc.cpu.threads‚17 network.recv‚ 4.665794e+06Ú׋“Ý-‚(:ˆ ·ÆéÒ‚Ëçö gpu.0.gpu‚10.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚327.587' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚65.5174 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÉÉ.=Þ-‚(:² ·ÆéÒÔÛ—ù* proc.memory.percent‚17.111587524414062$ disk.nvme0c0n1.in‚44135.66796875 disk.sda1.in‚0 disk.sda1.out‚0% disk.nvme0n1.out‚141972.861328125$ memory_percent‚82.83946514129639) proc.memory.availableMB‚ 2811.58203125 network.sent‚ 2.1752e+07" disk.nvme0n1.in‚44135.66796875 proc.cpu.threads‚17 network.recv‚ 4.669292e+06 disk.sda3.out‚ 1250.234375$ disk./.usageGB‚562.8098297119141 disk.rbd1.out‚ 49.0029296875 disk.sda.in‚ 106.21875 disk.sda.out‚ 1250.234375' disk.nvme0c0n1.out‚141972.861328125 disk.drbd2165.out‚0 disk.rbd3.out‚ 0.22412109375 disk.sda2.in‚0 disk.dm-1.in‚ 106.21875 disk.drbd2165.in‚0 disk.rbd3.in‚0 disk.rbd1.in‚ 0.52392578125 disk.dm-0.in‚44135.66796875 proc.memory.rssMB‚ 2803.5625 cpu‚25.594319648475693) disk./.usagePercent‚37.77063990086733 disk.sda3.in‚ 106.21875 disk.sda2.out‚0 disk.dm-1.out‚ 1250.234375& disk.rbd0.in‚1.759218604441423e+13 disk.rbd0.out‚0" disk.dm-0.out‚141972.861328125U”µ”ß-‚(:‰ ÆÆéÒ»Ûœ÷ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚271.478' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.2956 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ãv¶—=à-‚(:² ÆÆéÒ—óù" disk.dm-0.out‚141985.490234375 disk.drbd2165.in‚0 disk.rbd3.in‚0 network.recv‚ 4.672445e+06 disk.rbd0.out‚0 disk.rbd1.in‚ 0.52392578125 disk.rbd1.out‚ 49.0029296875 cpu‚25.594122269893177) disk./.usagePercent‚37.77093226147143 disk.sda1.in‚0" disk.nvme0n1.in‚44135.87109375$ disk./.usageGB‚562.8141860961914& disk.rbd0.in‚1.759218604441423e+13 disk.drbd2165.out‚0 proc.memory.rssMB‚ 2803.5625* proc.memory.percent‚17.111587524414062 disk.sda.in‚ 106.21875 disk.dm-0.in‚44135.87109375# memory_percent‚82.8394889831543 network.sent‚ 2.1757623e+07 disk.sda1.out‚0 disk.rbd3.out‚ 0.22412109375' disk.nvme0c0n1.out‚141985.490234375 disk.sda2.in‚0' proc.memory.availableMB‚ 2811.578125 disk.sda3.out‚ 1250.546875% disk.nvme0n1.out‚141985.490234375 disk.sda2.out‚0 proc.cpu.threads‚17 disk.sda.out‚ 1250.546875 disk.dm-1.in‚ 106.21875 disk.dm-1.out‚ 1250.546875 disk.sda3.in‚ 106.21875$ disk.nvme0c0n1.in‚44135.87109375‚(~Ížá-‚(:“ ÕÆéÒ‡„ôö gpu.0.gpu‚60.0 gpu.0.memory‚19, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚333.524' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚66.70479999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0îíë½Iâ-‚(:¾ ÕÆéÒ¡Øù disk.drbd2165.out‚0 disk.rbd3.in‚0 disk.rbd1.out‚ 49.0654296875 network.sent‚ 2.1762736e+07 disk.rbd0.out‚0& disk.nvme0n1.out‚141995.2685546875 disk.sda2.in‚0& disk.rbd0.in‚1.759218604441423e+13# disk.dm-0.out‚141995.2685546875 disk.sda1.out‚0 disk.sda3.out‚ 1250.80078125 proc.memory.rssMB‚ 2803.5625 network.recv‚ 4.675859e+06 disk.rbd1.in‚ 0.52392578125 disk.sda.out‚ 1250.80078125 disk.dm-1.in‚ 106.21875$ memory_percent‚82.83946514129639 proc.cpu.threads‚17 disk.dm-0.in‚44255.08203125 disk.sda1.in‚0 disk.sda3.in‚ 106.21875( disk.nvme0c0n1.out‚141995.2685546875 disk.sda.in‚ 106.21875* disk./.usagePercent‚37.771414323658405$ disk./.usageGB‚562.8213691711426 disk.sda2.out‚0 disk.rbd3.out‚ 0.22412109375$ disk.nvme0c0n1.in‚44255.08203125" disk.nvme0n1.in‚44255.08203125 disk.dm-1.out‚ 1250.80078125* proc.memory.percent‚17.111587524414062 disk.drbd2165.in‚0) proc.memory.availableMB‚ 2811.58203125 cpu‚25.59398105332086”jÐ4ã-/ _wandbruntime‚18752  _runtime‚18752ÿ:F¸Ëä-Å _step‚928 ' lr-AdamW/pg1‚0.00028530308425157093 ' lr-AdamW/pg2‚0.00028530308425157093 trainer/global_step‚11299 % _timestamp‚1.7843085686265771e+09 _runtime‚18752.843979236×4RÐå-Ê _runtime‚18752.843979236 _step‚928 ' lr-AdamW/pg1‚0.00028530308425157093 ' lr-AdamW/pg2‚0.00028530308425157093 trainer/global_step‚11299 % _timestamp‚1.7843085686265771e+09 É[$„4æ-/ _wandbruntime‚18752  _runtime‚18752o¤Êç-†P VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 % _timestamp‚1.7843085686278958e+09 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3259310866673152 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚28989803 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13707864458752 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36633600 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚3247276057333760 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 trainer/global_step‚11299 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚97877761 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3248679206510592 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚96428339 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚32039760 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚877135669145600 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚3257888129927988 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10630288599552 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚3248679206510592 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 194296121 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 194306100 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.curren¼ÓÕªût‚281 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 _runtime‚18752.84523146 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚97867788 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 _step‚929 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚877135301054464 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3257889499956278 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10612098022196 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚96428339 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚890843165513216 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚286 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚61029282 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10630288599552 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚97877761 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚32039479 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚3247276031905792 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3259309495110144 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚3248679232684032 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10631633989120 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚28989808 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚97867788 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚281 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 _runtime‚18752.84523146 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚97867788 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 _step‚929 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚877135301054464 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3257889499956278 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10612098022196 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚96428339 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚890843165513216 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚286 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚61029282 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10630288599552 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚97877761 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚32039479 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚3247276031905792 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3259309495110144 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚3248679232684032 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10631633989120 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚28989808 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚97867788 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 194296121 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 194306100¡ݨg4é-/ _wandbruntime‚18754  _runtime‚18754÷©Q…'ê-‘N E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10632559031808 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 trainer/global_step‚11299 % _timestamp‚1.7843085700119221e+09 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3259589792445952 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚890921828801024 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚96436545 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚61034578 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10614366088876 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚32042503 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 _runtime‚18754.229294074 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 194322798 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 _step‚930 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚97886253 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3258168305200812 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚61034854 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚28992346 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10632559031808 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3248957207240704 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚96436539 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚877212787462144 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10631213635072 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 194322798 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚3248957233414144 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10631213635072 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3247553913683968 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10613021488442 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚276 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚28992351 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚877213155553280 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3259589792445952 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚3248957207240704 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚38723584 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 194312805 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚97886253 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚61034578 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10614366088876 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚32042503 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 _runtime‚18754.229294074 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 194322798 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 _step‚930 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚97886253 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3258168305200812 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚61034854 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚28992346 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10632559031808 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3248957207240704 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚96436539 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚877212787462144 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10631213635072 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 194322798 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚3248957233414144 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10631213635072 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 UEDeviceStatsMonitor.on_train_batch_end/allocaúo–ted_bytes.large_pool.peak‚ 46141448192 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3247553913683968 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10613021488442 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚276 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚28992351 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚877213155553280 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3259589792445952 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚3248957207240704 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚38723584 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 194312805 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚97886253 FDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3273255847546368 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚3273255847546368 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10677949919744 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13767183172096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10676604530176 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚3271828455949896 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚61292531 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚3262579269189632 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10677949919744 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10658335135304 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚96838633 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3273257219109376 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚98293540 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3262579269189632 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚261 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚32175312 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚96838639 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 195132173 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚98303513 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚29116953 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚894777172701696 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚96838633 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚61292265 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚3261170146242560 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3262579243016192 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 195142152 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584„A7"þ-ŒP dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13767183172096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10676604530176 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚3271828455949896 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚61292531 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚3262579269189632 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10677949919744 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10658335135304 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚96838633 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3273257219109376 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚98293540 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3262579269189632 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚261 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚32175312 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚96838639 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 195132173 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚98303513 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚29116953 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚894777172701696 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚96838633 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚61292265 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚3261170146242560 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3262579243016192 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 195142152 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 trainer/global_step‚11349 _runtime‚18831.425749857 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3273255847546368 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚3273255847546368 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10677949919744 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192¥ÏK[å4ÿ-/ _wandbruntime‚18832  _runtime‚18832áaÃx'€.N T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚3272107261194318 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10659258601550 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3273534773312000 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3262857269919744 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404717568 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 trainer/global_step‚11349 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚29119496 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚98312005 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚260 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3261448002592768 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚61297557 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚894855848566272 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36626432 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚3262857269919744 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10677529565696 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 195158850 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚98312005 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 195158850 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚32178316 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3262857243746304 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3272108631222720 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3273536144882176 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚255 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3261448028020736 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚96846845 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚96846839 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚98302018 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚881087475937280 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 195148857 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚98302018 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚61297557 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚894855848566272 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36626432 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚3262857269919744 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10677529565696 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 195158850 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚98312005 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 195158850 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚32178316 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3262857243746304 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3272108631222720 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3273536144882176 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚255 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3261448028020736 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚96846845 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚96846839 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚98302018 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚881087475937280 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 195148857 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚98302018 EDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3287202199982592 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 473m;1¥e"51595008 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚61555771 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚61556043 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚97248939 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚32311668 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10722920460800 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚98729265 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 trainer/global_step‚11399 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚97248933 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10724265850368 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚898710493908480 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13826552086016 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10705916848734 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚267 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚29244103 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3287203571545600 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3276479279521792 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36633600 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚98719292 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3276479305695232 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚3275064235151360 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3285770152000094 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 195968225 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 195978204 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚884884309913600 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13826588719616 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10722920460800 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚272 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚898710898633216 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10704572248412 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚29244108 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚97248933 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚98719292 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352×=~(”.ŒP GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 195978204 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚884884309913600 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13826588719616 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10722920460800 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚272 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚898710898633216 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10704572248412 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚29244108 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚97248933 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚98719292 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 % _timestamp‚1.7843087259687593e+09 _step‚937 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚3276479279521792 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚3275064209723392 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10724265850368 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚3285768781971804 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3287202199982592 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚61555771 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚61556043 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚97248939 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚32311668 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10722920460800 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚98729265 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 trainer/global_step‚11399 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚97248933 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10724265850368 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚898710493908480 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13826552086016 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10705916848734 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚267 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚29244103 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3287203571545600 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3276479279521792 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36633600 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚98719292 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3276479305695232 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚3275064235151360 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3285770152000094 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 195968225 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0©…ùÏ4•./ _wandbruntime‚18911  _runtime‚18911î½'–.N EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 195994902 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚61561026 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚884961428230144 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚18911.57251326 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚98727770 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3275342091501568 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10723845496320 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚32314380 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404717568 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 195984909 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 195994902 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚32314633 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13827774988800 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3276757280251904 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚97257145 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13827738362368 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚898789166592512 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10705495714658 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚3276757306425344 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10723845496320 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚97257139 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3287482497318400 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚3276757280251904 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3287481125748224 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚3286047587216226 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚938 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚97257139 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10706840315092 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚253 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚898789571310080 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚11399 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10725190893056 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10725190893056 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚258 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚97257145 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3276757306425344 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0B8¤J—.•N ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36626432 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚98727770 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚61561284 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3275342116929536 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚98737757 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚61561026 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚884961428230144 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚18911.57251326 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚98727770 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3275342091501568 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10723845496320 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚32314380 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404717568 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 195984909 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 195994902 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚32314633 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13827774988800 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3276757280251904 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚97257145 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13827738362368 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 XDD¼S*QÿeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚898789166592512 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10705495714658 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚3276757306425344 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10723845496320 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚97257139 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3287482497318400 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚3276757280251904 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3287481125748224 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚3286047587216226 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚938 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚97257139 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10706840315092 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚253 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚898789571310080 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚11399 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10725190893056 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10725190893056 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚258 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚97257145 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3276757306425344 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 195994902 EDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3301148552418816 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚99145044 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10770581780992 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚3299709107993712 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚32446753 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚99155017 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 _runtime‚18988.93295187 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚97659239 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚32447031 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10770581780992 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10769236391424 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚29371258 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13885850713088 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚3288958324060160 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3301149923981824 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10752153961842 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚3290379342200832 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3301149923981824 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 $ _timestamp‚1.784308804712173e+09 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚902644517644288 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404724736 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚97659239 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚32447031 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10770581780992 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10769236391424 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚29371258 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13885850713088 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚3288958324060160 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3301149923981824 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10752153961842 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚3290379342200832 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3301149923981824 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 $ _timestamp‚1.784308804712173e+09 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚902644517644288 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404724736 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3301148552418816 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚99145044 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10770581780992 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚3299709107993712 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚32446753 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚99155017 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 _runtime‚18988.93295187­¼‘‘Ÿ«.‚(:” ÅÈéҦؘø gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚262.342' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚52.468399999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0àP¤ÍF¬.‚(:» ÅÈéÒ¦žëù' disk.nvme0n1.out‚142179.88427734375 disk.rbd1.in‚ 0.52392578125& disk.rbd0.in‚1.759218604441423e+13$ disk.dm-0.out‚142179.88427734375 disk.sda1.out‚0 disk.drbd2165.out‚0 disk.sda.out‚ 1256.8828125 disk.dm-0.in‚44748.01953125$ memory_percent‚82.84111022949219 proc.cpu.threads‚17 network.sent‚ 2.2018232e+07 disk.sda2.in‚0 disk.dm-1.in‚ 106.21875 disk.rbd0.out‚0 disk.sda1.in‚0 disk.rbd3.in‚0" disk.nvme0n1.in‚44748.01953125 disk.sda2.out‚0 disk.sda.in‚ 106.21875% proc.memory.availableMB‚ 2811.3125* proc.memory.percent‚17.111587524414062 network.recv‚ 4.732599e+06 disk.rbd1.out‚ 49.6279296875 disk.drbd2165.in‚0* disk./.usagePercent‚37.778306558495224$ disk./.usageGB‚562.9240684509277) disk.nvme0c0n1.out‚142179.88427734375 disk.dm-1.out‚ 1256.8828125 disk.sda3.out‚ 1256.8828125 disk.rbd3.out‚ 0.22412109375 cpu‚25.591308527874794$ disk.nvme0c0n1.in‚44748.01953125 disk.sda3.in‚ 106.21875 proc.memory.rssMB‚ 2803.5625UgÖB4­./ _wandbruntime‚18990  _runtime‚18990ݾ¸±'®.‘N H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚99163509 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 196830954 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚268 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3299989283266536 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10753077428200 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10751732827766 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚29373801 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚32449352 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 _runtime‚18990.315319135 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚99163509 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3290657316757504 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚61823416 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚97667439 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 196820961 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 196820961 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚61823148 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚99153522 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚32449615 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚263 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 trainer/global_step‚11449 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3301428849754624 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7843088060979183e+09 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 196830954 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚888836116705280 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36626432 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3289236205838336 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚97667445 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚97667445 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚3290657342930944 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10771506823680 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10770161426944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13887029525504 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚3299987913238134 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3301427478184448 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3301428849754624 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13887066151936 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚3290657316757504 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚29373796 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚902723182857216 _step‚942 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚97667439 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚99153522 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚902722778139648ÇÜž!'¯.–N N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚97667439 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚99153522 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚902722778139648 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚99163509 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 196830954 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚268 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3299989283266536 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10753077428200 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10751732827766 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚29373801 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚32449352 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 _runtime‚18990.315319135 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚99163509 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3290657316757504 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚61823416 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚97667439 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 196820961 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 196820961 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚61823148 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚99153522 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚32449615 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚263 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 trainer/global_step‚11449 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3301428849754624 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7843088060979183e+09 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 196830954 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚888836116705280 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36626432 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3289236205838336 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚97667445 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚97667445 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚3290657342930944 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10771506823680 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10770161426944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13887029525504 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚3299987913238134 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3301427478184448 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3301428849754624 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13887066151936 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚3290657316757504 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚29373796 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚902723182857216 _step‚942®.ÔGä4°./ _wandbruntime‚18990  _runtime‚18990g¨„±.Š 0loss/FRt1/assignment_loss‚1.2345934740994988 /loss/FRt2/detection_loss‚1.2538049378426654 1loss/SRqqt2/detection_loss‚0.5057558496426448 %loss/total_loss‚5.677754733165303 _runtime‚18990.401093558 0loss/FRt2/assignment_loss‚0.3480723524965891 3loss/SRqqt1/assignment_loss‚0.22136314742895233 0loss/FRt1/detection_loss‚0.44765348207045297 3loss/FBt2/assignment_loss‚4.822508125792262e-05 /loss/FBt1/detection_loss‚0.4200811375138493 /loss/FBt2/detection_loss‚0.2886995498659849 epoch‚28 trainer/global_step‚11449 % _timestamp‚1.7843088061841524e+09 _step‚943 2loss/SRqqt2/assignment_loss‚0.2848415574223827 3loss/FBt1/assignment_loss‚0.0006414581554825378 1loss/SRqqt1/detection_loss‚0.6721995615455421b×’². _runtime‚18990.401093558 0loss/FRt2/assignment_loss‚0.3480723524965891 3loss/SRqqt1/assignment_loss‚0.22136314742895233 0loss/FRt1/detection_loss‚0.44765348207045297 3loss/FBt2/assignment_loss‚4.822508125792262e-05 /loss/FBt1/detection_loss‚0.4200811375138493 /loss/FBt2/detection_loss‚0.2886995498659849 epoch‚28 trainer/global_step‚11449 % _timestamp‚1.7843088061841524e+09 _step‚943 2loss/SRqqt2/assignment_loss‚0.2848415574223827 3loss/FBt1/assignment_loss‚0.0006414581554825378 1loss/SRqqt1/detection_loss‚0.6721995615455421 0loss/FRt1/assignment_loss‚1.2345934740994988 /loss/FRt2/detection_loss‚1.2538049378426654 1loss/SRqqt2/detection_loss‚0.5057558496426448 %loss/total_loss‚5.677754733165303¯ÑL?ž³.‚(:“ ÔÈéÒ¡¢©ø gpu.0.gpu‚94.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚332.561' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚66.51219999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0M+ÄÉ@´.‚(:µ ÔÈéÒÑœ§ù network.sent‚ 2.2070238e+07" disk.nvme0n1.in‚45023.38671875 disk.dm-0.in‚45023.38671875 disk.sda3.out‚ 1257.140625 proc.memory.rssMB‚ 2803.5625 disk.drbd2165.in‚0 disk.drbd2165.out‚0 network.recv‚ 4.737143e+06( disk.nvme0c0n1.out‚142190.6806640625 disk.rbd1.out‚ 49.6279296875 disk.sda.in‚ 106.21875 disk.sda.out‚ 1257.140625 disk.dm-1.out‚ 1257.140625$ disk./.usageGB‚562.9307708740234 disk.sda2.in‚0 disk.rbd1.in‚ 0.52392578125 disk.rbd0.out‚0$ memory_percent‚82.84082412719727* proc.memory.percent‚17.111587524414062& disk.rbd0.in‚1.759218604441423e+13# disk.dm-0.out‚142190.6806640625 disk.sda1.out‚0 disk.sda1.in‚0 disk.sda3.in‚ 106.21875 disk.rbd3.in‚0& disk.nvme0n1.out‚142190.6806640625 disk.dm-1.in‚ 106.21875 cpu‚25.59115922723565 proc.cpu.threads‚17) disk./.usagePercent‚37.77875636373288$ disk.nvme0c0n1.in‚45023.38671875 disk.sda2.out‚0 disk.rbd3.out‚ 0.22412109375' proc.memory.availableMB‚ 2811.359375·ü°ƒŸµ.‚(:” ãÈéÒæÞûö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚287.087' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚57.417399999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0`p¡@F¶.‚(:» ãÈéÒפÙù disk.drbd2165.out‚0 disk.rbd3.out‚ 0.22412109375 disk.rbd1.out‚ 49.7060546875$ memory_percent‚82.84130096435547 network.sent‚ 2.2122789e+07 disk.dm-1.out‚ 1257.52734375& disk.rbd0.in‚1.759218604441423e+13 disk.dm-0.in‚45023.80859375 disk.sda1.out‚0 disk.sda2.in‚0& proc.memory.availableMB‚ 2811.28125 cpu‚25.590994434266932 disk.rbd1.in‚ 0.52392578125$ disk./.usageGB‚562.9376983642578 disk.sda3.in‚ 106.21875 disk.sda3.out‚ 1257.52734375& disk.nvme0n1.out‚142200.6513671875* proc.memory.percent‚17.111587524414062) disk./.usagePercent‚37.77922127341506 disk.sda2.out‚0 disk.sda.out‚ 1257.52734375 disk.rbd0.out‚0 disk.sda1.in‚0 disk.rbd3.in‚0$ disk.nvme0c0n1.in‚45023.80859375 proc.memory.rssMB‚ 2803.5625 disk.drbd2165.in‚0 network.recv‚ 4.741494e+06( disk.nvme0c0n1.out‚142200.6513671875" disk.nvme0n1.in‚45023.80859375 proc.cpu.threads‚17 disk.sda.in‚ 106.21875 disk.dm-1.in‚ 106.21875# disk.dm-0.out‚142200.65136718758#Ö@”·.‚(:‰ òÈéÒ‚Œèö gpu.0.gpu‚91.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚335.402' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚67.0804 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÙÛj¹I¸.‚(:¾ òÈéÒÁÛ¿ù disk.rbd0.out‚0# disk.dm-0.out‚142217.4912109375 disk.sda3.out‚ 1257.62890625 disk.drbd2165.in‚0 disk.sda2.out‚0) proc.memory.availableMB‚ 2811.72265625 proc.cpu.threads‚17 disk.rbd1.out‚ 49.7060546875 disk.sda.in‚ 106.21875 disk.sda2.in‚0 network.recv‚ 4.745012e+06 disk.dm-0.in‚45024.02734375" disk.nvme0n1.in‚45024.02734375& disk.nvme0n1.out‚142217.4912109375* proc.memory.percent‚17.111587524414062 disk.rbd3.in‚0( disk.nvme0c0n1.out‚142217.4912109375 disk.rbd1.in‚ 0.52392578125 proc.memory.rssMB‚ 2803.5625 network.sent‚ 2.2128416e+07 disk.sda1.in‚0 disk.sda3.in‚ 106.21875$ memory_percent‚82.83860683441162) disk./.usagePercent‚37.77951337801163 disk.sda.out‚ 1257.62890625 disk.dm-1.in‚ 106.21875 disk.dm-1.out‚ 1257.62890625& disk.rbd0.in‚1.759218604441423e+13 disk.drbd2165.out‚0$ disk.nvme0c0n1.in‚45024.02734375$ disk./.usageGB‚562.9420509338379 disk.sda1.out‚0 disk.rbd3.out‚ 0.22412109375 cpu‚25.590831527903656;%k”¹.‚(:‰ ÉéÒÚ“¦÷ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚270.943' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.1886 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÌÂ%¨[º.‚(:» ÉéÒÚ‰áù network.recv‚ 4.748767e+06) disk./.usagePercent‚37.77999262411573& disk.rbd0.in‚1.759218604441423e+13 disk.dm-0.in‚45024.23828125 disk.drbd2165.out‚0 disk.rbd3.in‚0 disk.rbd3.out‚ 0.22412109375 disk.rbd1.in‚ 0.52392578125 disk.sda.in‚ 106.21875 disk.dm-1.out‚ 1257.921875$ disk.dm-0.out‚142224.78857421875 disk.drbd2165.in‚0$ disk.nvme0c0n1.in‚45024.23828125$ memory_percent‚82.83851146697998 disk.sda.out‚ 1257.921875) disk.nvme0c0n1.out‚142224.78857421875" disk.nvme0n1.in‚45024.23828125 cpu‚25.590°ˆöë654528454518 proc.cpu.threads‚17 disk.sda3.in‚ 106.21875) proc.memory.availableMB‚ 2811.73828125 proc.memory.rssMB‚ 2803.5625* proc.memory.percent‚17.111587524414062 network.sent‚ 2.2133598e+07 disk.sda2.out‚0 disk.rbd1.out‚ 49.7685546875' disk.nvme0n1.out‚142224.78857421875 disk.sda1.in‚0$ disk./.usageGB‚562.9491920471191 disk.dm-1.in‚ 106.21875 disk.sda3.out‚ 1257.921875 disk.rbd0.out‚0 disk.sda1.out‚0 disk.sda2.in‚04÷•».‚(:Š ÉéÒ–ðœ÷ gpu.0.gpu‚100.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚297.626' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚59.5252 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0tÅX²E¼.‚(:º ÉéÒ²Œù" disk.dm-0.out‚142240.689453125 disk.sda1.out‚0 disk.sda2.out‚0 disk.rbd1.in‚ 0.52392578125) proc.memory.availableMB‚ 2815.62109375 proc.cpu.threads‚17 network.recv‚ 4.751902e+06 disk.drbd2165.in‚0" disk.nvme0n1.in‚45141.12109375 disk.dm-1.in‚ 106.21875$ memory_percent‚82.81481266021729 network.sent‚ 2.213872e+07 disk.sda2.in‚0 disk.sda.in‚ 106.21875 disk.dm-1.out‚ 1258.06640625* proc.memory.percent‚17.111587524414062 disk.dm-0.in‚45141.12109375 disk.sda1.in‚0 disk.sda3.in‚ 106.21875 disk.rbd3.in‚0 cpu‚25.59051963575811$ disk./.usageGB‚562.9559745788574& disk.rbd0.in‚1.759218604441423e+13 disk.sda3.out‚ 1258.06640625 disk.drbd2165.out‚0$ disk.nvme0c0n1.in‚45141.12109375% disk.nvme0n1.out‚142240.689453125 disk.rbd1.out‚ 49.7685546875 disk.sda.out‚ 1258.06640625* disk./.usagePercent‚37.780447805511606 disk.rbd0.out‚0 proc.memory.rssMB‚ 2803.5625 disk.rbd3.out‚ 0.22412109375' disk.nvme0c0n1.out‚142240.689453125åá!V4½./ _wandbruntime‚19067  _runtime‚19067È´.Cʾ.Ä ' lr-AdamW/pg2‚0.00027441839064666244 trainer/global_step‚11499 $ _timestamp‚1.784308883302952e+09 _runtime‚19067.520137873 _step‚944 ' lr-AdamW/pg1‚0.00027441839064666244Ÿ«+Ï¿.É ' lr-AdamW/pg2‚0.00027441839064666244 trainer/global_step‚11499 $ _timestamp‚1.784308883302952e+09 _runtime‚19067.520137873 _step‚944 ' lr-AdamW/pg1‚0.00027441839064666244°„VˆŒ4À./ _wandbruntime‚19067  _runtime‚19067eùàÿ (Á.†P _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10816897711616 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚98069533 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3304279352532992 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚29498408 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚906578097879040 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚3313649434015620 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚62080779 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚98069539 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 197650308 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10815552322048 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36633600 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7843088833040617e+09 _runtime‚19067.52138112 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚892632950681600 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3304279378706432 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚266 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 197640329 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3315094904855040 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚99580769 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3315096276418048 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚98069533 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3315096276418048 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 _step‚945 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10816897711616 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚99570796 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚3304279378706432 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10815552322048 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚3302852387540992 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3313650804043910 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚3302852412968960 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚3315094904855040 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404724736 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 trainer/global_step‚11499 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚3304279352532992 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10797046474628 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13945147197440“˜ è(Â.‹P VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3304279378706432 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚266 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 197640329 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3315094904855040 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚99580769 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3315096276418048 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚98069533 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3315096276418048 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 _step‚945 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10816897711616 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚99570796 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚3304279378706432 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10815552322048 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚3302852387540992 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3313650804043910 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚3302852412968960 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚3315094904855040 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404724736 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 trainer/global_step‚11499 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚3304279352532992 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10797046474628 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13945147197440 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10816897711616 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚98069533 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3304279352532992 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚29498408 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚906578097879040 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚3313649434015620 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚62080779 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚98069539 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 197650308 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10815552322048 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36633600 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7843088833040617e+09 _runtime‚19067.52138112 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚892632950681600 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0±ÏÖÕ4Ã./ _wandbruntime‚19068  _runtime‚19068bž3œ%Ä.‘N TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚32585268 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚892710068998144 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13946327577600 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13946290951168 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚254 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚99579274 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚62085960 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10817822754304 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚99589261 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10816477357568 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3304557379436544 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚98077739 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10799314541308 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10797969940874 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 trainer/global_step‚11499 _step‚946 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚892710437089280 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3315375202190848 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3303130294747136 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚99589261 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10816477357568 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚3313928239260042 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚906656764666880 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 % _timestamp‚1.7843088846883736e+09 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3315373830620672 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚98077745 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3315375202190848 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚62086219 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10817822754304 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3304557353263104 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚29500951 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚32585014 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚98077745 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚259 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 197657013 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 197667006 WHDeviceStatsMonitor.on_train_batch_end/reque´hv{sted_bytes.small_pool.current‚ 1344600434 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚3304557379436544 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3313929609288444ZçOÕ'Å.–N XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 trainer/global_step‚11499 _step‚946 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚892710437089280 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3315375202190848 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3303130294747136 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚99589261 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10816477357568 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚3313928239260042 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚906656764666880 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 % _timestamp‚1.7843088846883736e+09 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3315373830620672 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚98077745 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3315375202190848 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚62086219 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10817822754304 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3304557353263104 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚29500951 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚32585014 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚98077745 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚259 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 197657013 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 197667006 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚3304557379436544 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3313929609288444 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚32585268 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚892710068998144 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13946327577600 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13946290951168 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚254 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚99579274 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚62085960 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10817822754304 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚99589261 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10816477357568 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3304557379436544 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚98077739 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10799314541308 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10797969940874²’ç74Æ./ _wandbruntime‚19068  _runtime‚19068ÆXl' Ç.‡ 2loss/SRqqt1/assignment_loss‚0.2158588757366914 .loss/FRt2/detection_loss‚1.382771959132572 1loss/SRqqt1/detection_loss‚0.6764723311027941 2loss/FBt2/assignment_loss‚0.006514003433150035 epoch‚28 _runtime‚19068.992345535 3loss/FBt1/assignment_loss‚6.840352826819262e-05 /loss/FRt1/detection_loss‚0.4443207414155262 0loss/FBt2/detection_loss‚0.38642723312131794 % _timestamp‚1.7843088847754533e+09 2loss/SRqqt2/assignment_loss‚0.2938960679823735 1loss/SRqqt2/detection_loss‚0.5044670388087676 0loss/FBt1/detection_loss‚0.44704979692422153 $loss/total_loss‚5.85259207426114 trainer/global_step‚11499 _step‚947 0loss/FRt1/assignment_loss‚1.1745331954757003 0loss/FRt2/assignment_loss‚0.3202124275997573¸WŽÈ.Œ 3loss/FBt1/assignment_loss‚6.840352826819262e-05 /loss/FRt1/detection_loss‚0.4443207414155262 0loss/FBt2/detection_loss‚0.38642723312131794 % _timestamp‚1.7843088847754533e+09 2loss/SRqqt2/assignment_loss‚0.2938960679823735 1loss/SRqqt2/detection_loss‚0.5044670388087676 0loss/FBt1/detection_loss‚0.44704979692422153 $loss/total_loss‚5.85259207426114 trainer/global_step‚11499 _step‚947 0loss/FRt1/assignment_loss‚1.1745331954757003 0loss/FRt2/assignment_loss‚0.3202124275997573 2loss/SRqqt1/assignment_loss‚0.2158588757366914 .loss/FRt2/detection_loss‚1.382771959132572 1loss/SRqqt1/detection_loss‚0.6764723311027941 2loss/FBt2/assignment_loss‚0.006514003433150035 epoch‚28 _runtime‚19068.992345535³EÖol”É.‚(:‰ ŸÉéÒ×ï©ø gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚285.532' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚57.1064 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0žº÷ì@Ê.‚(:µ ŸÉéÒÍɳù% disk.nvme0n1.out‚142248.853515625 disk.rbd1.in‚ 0.52392578125 disk.dm-0.in‚45141.33984375 network.recv‚ 4.75539e+06 disk.dm-1.out‚ 1258.38671875$ disk./.usageGB‚562.9601745605469 disk.rbd3.out‚ 0.22412109375 disk.sda2.in‚0 network.sent‚ 2.21438e+07 disk.sda.in‚ 106.21875' disk.nvme0c0n1.out‚142248.853515625" disk.nvme0n1.in‚45141.33984375& disk.rbd0.in‚1.759218604441423e+13 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda3.in‚ 106.21875( proc.memory.availableMB‚ 2821.4765625 proc.cpu.threads‚17 disk.rbd1.out‚ 49.8466796875" disk.dm-0.out‚142248.853515625( disk./.usagePercent‚37.7807296698068 disk.sda.out‚ 1258.38671875 disk.rbd0.out‚0 disk.drbd2165.out‚0 disk.drbd2165.in‚0 disk.rbd3.in‚0$ disk.nvme0c0n1.in‚45141.33984375 disk.dm-1.in‚ 106.21875 disk.sda3.out‚ 1258.38671875$ memory_percent‚82.77907371520996 proc.memory.rssMB‚2804.875* proc.memory.percent‚17.119598388671875 disk.sda2.out‚0 cpu‚25.590329953467826AWQžË.‚(:“ ®ÉéÒ¥¸ðø gpu.0.gpu‚84.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚336.465' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚67.29299999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0÷@Ì.‚(:µ ®ÉéÒìËú disk.sda.in‚ 106.21875 disk.dm-1.in‚ 106.21875$ disk./.usageGB‚562.9695205688477 disk.sda1.in‚0 disk.sda3.in‚ 106.21875* proc.memory.percent‚17.119598388671875 disk.sda3.out‚ 1258.57421875# memory_percent‚82.7768325805664 cpu‚25.590191716147025 proc.cpu.threads‚17 network.sent‚ 2.220646e+07& disk.rbd0.in‚1.759218604441423e+13 disk.rbd0.out‚0& proc.memory.availableMB‚ 2821.84375 disk.sda1.out‚0 disk.drbd2165.out‚0 disk.rbd3.out‚ 0.22412109375" disk.nvme0n1.in‚45141.76171875 disk.dm-0.in‚45141.76171875$ disk.nvme0c0n1.in‚45141.76171875' disk.nvme0c0n1.out‚142258.740234375 disk.sda2.in‚0 disk.rbd1.out‚ 49.8466796875 proc.memory.rssMB‚2804.875 network.recv‚ 4.759501e+06" disk.dm-0.out‚142258.740234375 disk.rbd3.in‚0% disk.nvme0n1.out‚142258.740234375 disk.sda2.out‚0 disk.sda.out‚ 1258.57421875 disk.dm-1.out‚ 1258.57421875) disk./.usagePercent‚37.78135688826569 disk.drbd2165.in‚0 disk.rbd1.in‚ 0.52392578125fηʞÍ.‚(:“ ½ÉéҦњ÷ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚273.684' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚54.73680000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0õ©-:Î.‚(:¯ ½ÉéÒý²ñù" disk.nvme0n1.in‚45141.96484375 disk.sda.in‚ 106.21875 disk.rbd3.in‚0 disk.rbd3.out‚ 0.22412109375& proc.memory.availableMB‚ 2821.53125 cpu‚25.590017099228987 network.recv‚ 4.763193e+06 disk.rbd1.out‚ 49.9091796875 disk.drbd2165.out‚0 proc.memory.rssMB‚2804.875 disk.dm-0.in‚45141.96484375 disk.sda1.out‚0 disk.sda2.in‚0 proc.cpu.threads‚17 disk.sda.out‚ 1258.9375& disk.rbd0.in‚1.759218604441423e+13 disk.sda3.in‚ 106.21875 disk.drbd2165.in‚0& disk.nvme0n1.out‚142270.7490234375 disk.rbd1.in‚ 0.52392578125* proc.memory.percent‚17.119598388671875* disk./.usagePercent‚37.781648992862266$ disk./.usageGB‚562.9738731384277 disk.dm-1.in‚ 106.21875# disk.dm-0.out‚142270.7490234375 disk.sda1.in‚0( disk.nvme0c0n1.out‚142270.7490234375 disk.sda2.out‚0$ memory_percent‚82.77873992919922 disk.dm-1.out‚ 1258.9375$ disk.nvme0c0n1.in‚45141.96484375 network.sent‚ 2.2211572e+07 disk.rbd0.out‚0 disk.sda3.out‚ 1258.9375°õCÏ“Ï.‚(:ˆ ÌÉéÒĶŠ÷ gpu.0.gpu‚99.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚242.135' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚48.427 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0`żHÐ.‚(:½ ÌÉéÒÇý¦ú disk.drbd2165.in‚0 cpu‚25.589867085190914 disk.rbd3.in‚0) disk.nvme0c0n1.out‚142279.22314453125" disk.nvme0n1.in‚45219.30078125 disk.sda2.out‚0 disk.sda.in‚ 106.21875 disk.rbd0.out‚0 proc.memory.rssMB‚2804.875* proc.memory.percent‚17.119598388671875 network.recv‚ 4.766539e+06 disk.sda1.out‚0 disk.sda3.in‚ 106.21875 disk.rbd1.out‚ 49.9091796875$ disk.dm-0.out‚142279.22314453125 proc.cpu.threads‚17 disk.sda1.in‚0 disk.sda3.out‚ 1259.0859375 disk.dm-1.out‚ 1259.0859375$ disk.nvme0c0n1.in‚45219.30078125 network.sent‚ 2.2217197e+07 disk.drbd2165.out‚0 disk.sda.out‚ 1259.0859375) disk./.usagePercent‚37.78209879809992 disk.sda2.in‚0 disk.rbd1.in‚ 0.52392578125 disk.dm-0.in‚45219.30078125$ disk./.usageGB‚562.9805755615234 disk.rbd3.out‚ 0.22412109375 disk.dm-1.in‚ 106.21875& disk.rbd0.in‚1.759218604441423e+13$ memory_percent‚82.77876377105713) proc.memory.availableMB‚ 2821.52734375' disk.nvme0n1.out‚142279.22314453125g¢Ï/ŸÑ.‚(:” ÛÉéÒ¸óô÷ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚288.151' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚57.630199999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0\ÁµKBÒ.‚(:· ÛÉéÒç£ýù! disk./.usageGB‚562.9873046875 disk.sda3.in‚ 106.21875 disk.rbd3.in‚0 disk.rbd3.out‚ 0.22412109375 disk.sda2.in‚0 disk.dm-1.out‚ 1259.42578125 proc.cpu.threads‚17 network.recv‚ 4.77032e+06$ disk.nvme0c0n1.in‚45219.72265625 disk.sda.in‚ 106.21875& disk.rbd0.in‚1.759218604441423e+13 disk.rbd1.out‚ 49.9873046875 disk.dm-0.in‚45219.72265625& proc.memory.availableMB‚ 2821.78125 cpu‚25.589719340037913* disk./.usagePercent‚37.782550395390324" disk.nvme0n1.in‚45219.72265625 disk.rbd1.in‚ 0.52392578125 disk.sda.out‚ 1259.42578125$ memory_percent‚82.77721405029297 proc.memory.rssMB‚2804.875 network.sent‚ 2.2222365e+07 disk.drbd2165.out‚0 disk.sda2.out‚0 disk.rbd0.out‚0# disk.dm-0.out‚142291.9873046875 disk.sda1.out‚0 disk.sda3.out‚ 1259.42578125( disk.nvme0c0n1.out‚142291.9873046875 disk.dm-1.in‚ 106.21875* proc.memory.percent‚17.119598388671875 disk.drbd2165.in‚0& disk.nvme0n1.out‚142291.9873046875 disk.sda1.in‚0bÝDr4Ó./ _wandbruntime‚19146  _runtime‚19146LW‡ËÔ.Å trainer/global_step‚11549 % _timestamp‚1.7843089620485625e+09 _runtime‚19146.265734957 _step‚948 ' lr-AdamW/pg1‚0.00027170826270538496 ' lr-AdamW/pg2‚0.00027170826270538496ÿ¡DÐÕ.Ê ' lr-AdamW/pg1‚0.00027170826270538496 ' lr-AdamW/pg2‚0.00027170826270538496 trainer/global_step‚11549 % _timestamp‚1.7843089620485625e+09 _runtime‚19146.265734957 _step‚948´?ìc&4Ö./ _wandbruntime‚19146  _runtime‚19146F->q(×.ˆP ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚99996548 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10861868252672 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚32718000 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404724736 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3327591130065818 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚3316746476449792 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36633600 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3329042628854272 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚3316746501877760 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚98479839 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚910511219543040 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 _runtime‚19146.267091077 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10861868252672 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14004316568576 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚29625558 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10844628188058 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3318179389038592 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 198486360 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚98479833 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚896507271065600 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 _step‚949 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 $ _timestamp‚1.784308962049772e+09 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚98479839 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 100006521 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚32718267 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10843283587736 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3329042628854272 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10863213642240 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚62343553 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3329041257291264 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚3327589760037528 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚267 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚62343825 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚98479833 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 100006521 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚3318179389038592 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚3318179415212032 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚29625553 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 198476381 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10863213642240 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚272 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚99996548 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10861868252672 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚32718000 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404724736 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3327591130065818 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚3316746476449792 D;DeviceStatsMonitor.on_train_batch_start/segment.all.currøéÈçent‚1995 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36633600 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3329042628854272 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚3316746501877760 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚98479839 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚910511219543040 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 _runtime‚19146.267091077 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10861868252672 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14004316568576 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚29625558 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10844628188058 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3318179389038592 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 198486360 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚98479833 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚896507271065600 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 _step‚949 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 $ _timestamp‚1.784308962049772e+09 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚98479839 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 100006521 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚32718267 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10843283587736 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3329042628854272 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10863213642240 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚62343553 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3329041257291264 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚3327589760037528 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚267 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚62343825 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚98479833 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 100006521 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚3318179389038592 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚3318179415212032 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚29625553 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 198476381 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10863213642240 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚272 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚99996548 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 198503058 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚98488045 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚32720739 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14005536433152 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 198493065 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚3318457415942144 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404717568 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚29628096 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3317024383655936 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3329321554627072 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 trainer/global_step‚11549 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3318457415942144 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10864138684928 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚19147.652328503 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚29628101 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚3318457389768704 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 100015013 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10862793288192 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 100015013 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚98488039 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚62348835 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚62349096 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚32720995 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚98488039 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚896584757473280 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 $ _timestamp‚1.784308963434988e+09 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚256 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚910590293906432 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3327869935310352 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3317024358227968 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10844207053982 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚98488045 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995<<ŽA'Û.™N KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3317024358227968 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10844207053982 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚98488045 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 198503058 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚98488045 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚32720739 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14005536433152 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 198493065 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚3318457415942144 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404717568 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚29628096 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3317024383655936 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3329321554627072 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 trainer/global_step‚11549 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3318457415942144 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10864138684928 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚19147.652328503 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚29628101 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚3318457389768704 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 100015013 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10862793288192 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 100015013 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚98488039 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚62348835 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚62349096 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚32720995 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚98488039 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚896584757473280 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 $ _timestamp‚1.784308963434988e+09 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚256 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚910590293906432 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3327869935310352¶=ðf4Ü./ _wandbruntime‚19147  _runtime‚191476 è²Ý.‹ 0loss/FRt2/assignment_loss‚0.2528473948146783 3loss/FBt1/assignment_loss‚6.263452083903332e-05 /loss/FRt2/detection_loss‚1.3458773426742374 _runtime‚19147.739239662 _step‚951 2loss/SRqqt2/assignment_loss‚0.2573829382540584 %loss/total_loss‚5.719980166192017 0loss/FRt1/assignment_loss‚1.1664383145234056 3loss/SRqqt1/assignment_loss‚0.22491922031489772 4loss/FBt2/assignment_loss‚0.00047779796940793166 /loss/FRt1/detection_loss‚0.4516840452313769 1loss/SRqqt2/detection_loss‚0.5211286290918364 /loss/FBt1/detection_loss‚0.4251267512794584 % _timestamp‚1.7843089635223398e+09 1loss/SRqqt1/detection_loss‚0.7519748687603639 0loss/FBt2/detection_loss‚0.32206022875745677 epoch‚28 trainer/global_step‚11549)r‘Þ. epoch‚28 trainer/global_step‚11549 0loss/FRt2/assignment_loss‚0.2528473948146783 3loss/FBt1/assignment_loss‚6.263452083903332e-05 /loss/FRt2/detection_loss‚1.3458773426742374 _runtime‚19147.739239662 _step‚951 2loss/SRqqt2/assignment_loss‚0.2573829382540584 %loss/total_loss‚5.719980166192017 0loss/FRt1/assignment_loss‚1.1664383145234056 3loss/SRqqt1/assignment_loss‚0.22491922031489772 4loss/FBt2/assignment_loss‚0.00047779796940793166 /loss/FRt1/detection_loss‚0.4516840452313769 1loss/SRqqt2/detection_loss‚0.5211286290918364 /loss/FBt1/detection_loss‚0.4251267512794584 % _timestamp‚1.7843089635223398e+09 1loss/SRqqt1/detection_loss‚0.7519748687603639 0loss/FBt2/detection_loss‚0.32206022875745677·ÁØgžß.‚(:“ êÉéÒЩïö gpu.0.gpu‚90.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚294.335' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚58.86699999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0æÈêGà.‚(:¼ êÉéÒÔ¤²ú& disk.rbd0.in‚1.759218604441423e+13 disk.sda2.out‚0 disk.rbd1.out‚ 49.9873046875 proc.cpu.threads‚17 disk.sda2.in‚0 disk.sda.out‚ 1259.62890625 disk.drbd2165.in‚0 disk.rbd0.out‚0# disk.dm-0.out‚142302.2177734375 disk.sda1.out‚0$ disk.nvme0c0n1.in‚45219.92578125' proc.memory.availableMB‚ 2821.296875 disk.sda3.out‚ 1259.62890625( disk.nvme0c0n1.out‚142302.2177734375 proc.memory.rssMB‚2804.875* proc.memory.percent‚17.119598388671875 disk.drbd2165.out‚0 disk.rbd3.out‚ 0.22412109375" disk.nvme0n1.in‚45219.92578125 disk.rbd1.in‚ 0.52392578125 disk.dm-1.in‚ 106.21875$ memory_percent‚82.78017044067383 network.sent‚ 2.2226958e+07& disk.nvme0n1.out‚142302.2177734375 network.recv‚ 4.773688e+06$ disk./.usageGB‚562.9923553466797 disk.sda1.in‚0 cpu‚25.589530706537165 disk.dm-0.in‚45219.92578125 disk.rbd3.in‚0 disk.sda.in‚ 106.21875 disk.dm-1.out‚ 1259.62890625* disk./.usagePercent‚37.782889349365654 disk.sda3.in‚ 106.21875“q8ÉŸá.‚(:” ùÉéÒÔ¡úö gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚151.553' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚30.310599999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0'`Ø[>â.‚(:³ ùÉéÒï÷êù! disk.dm-0.out‚142309.63671875& proc.memory.availableMB‚ 2821.53125 disk.sda2.out‚0 disk.sda1.in‚0 proc.cpu.threads‚17( disk./.usagePercent‚37.7833450427766 disk.rbd3.in‚0 disk.rbd1.in‚ 0.52392578125 network.sent‚ 2.2291017e+07 disk.drbd2165.out‚0 disk.rbd3.out‚ 0.22412109375 disk.sda2.in‚0 proc.memory.rssMB‚2804.875* proc.memory.percent‚17.119598388671875$ disk./.usageGB‚562.9991455078125" disk.nvme0n1.in‚45220.14453125 disk.dm-1.in‚ 106.21875 disk.dm-1.out‚ 1259.81640625 cpu‚25.589397157444317 network.recv‚ 4.778126e+06 disk.drbd2165.in‚0& disk.nvme0c0n1.out‚142309.63671875$ disk.nvme0n1.out‚142309.63671875 disk.sda.in‚ 106.21875 disk.sda.out‚ 1259.81640625 disk.sda1.out‚0$ memory_percent‚82.77873992919922 disk.sda3.in‚ 106.21875 disk.rbd0.out‚0 disk.dm-0.in‚45220.14453125 disk.sda3.out‚ 1259.81640625$ disk.nvme0c0n1.in‚45220.14453125 disk.rbd1.out‚ 50.1123046875& disk.rbd0.in‚1.759218604441423e+13ÓŸpq;ã.‚(:ˆ ˆÊéÒ™¨ü÷ gpu.0.gpu‚99.0 gpu.0.memon †Xry‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚289.215' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚57.843 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0íôq 7ä.‚(:¬ ˆÊéÒ÷ã¥ù proc.memory.rssMB‚2804.875) disk.nvme0c0n1.out‚142324.54931640625& disk.rbd0.in‚1.759218604441423e+13$ memory_percent‚82.77873992919922 disk.rbd3.in‚0 disk.sda2.in‚0 disk.rbd1.out‚ 50.1123046875 disk.dm-1.out‚ 1260.0859375 disk.sda1.in‚0& proc.memory.availableMB‚ 2821.53125* disk./.usagePercent‚37.783780767599886$ disk./.usageGB‚563.0056381225586 disk.rbd0.out‚0 network.sent‚ 2.2295188e+07 disk.rbd3.out‚ 0.22412109375 disk.sda3.in‚ 106.21875* proc.memory.percent‚17.119598388671875' disk.nvme0n1.out‚142324.54931640625 disk.rbd1.in‚ 0.52392578125 disk.sda.in‚ 106.21875 disk.dm-1.in‚ 106.21875 disk.drbd2165.in‚0 disk.drbd2165.out‚0 proc.cpu.threads‚17 disk.nvme0n1.in‚ 45326.375 disk.dm-0.in‚ 45326.375 disk.sda1.out‚0 cpu‚25.589224722605003 disk.sda2.out‚0 disk.sda.out‚ 1260.0859375$ disk.dm-0.out‚142324.54931640625 network.recv‚ 4.780421e+06 disk.nvme0c0n1.in‚ 45326.375 disk.sda3.out‚ 1260.0859375ñD”å.‚(:‰ —ÊéÒ³õéö gpu.0.gpu‚82.0 gpu.0.memory‚25, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚332.413' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚66.4826 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ÿu©Dæ.‚(:¹ —ÊéÒÌĪù) disk./.usagePercent‚37.78408336850536' disk.nvme0n1.out‚142332.69677734375 disk.rbd3.in‚0 disk.rbd3.out‚ 0.22412109375 disk.rbd0.out‚0$ disk./.usageGB‚563.0101470947266 disk.rbd1.in‚ 0.52392578125 disk.rbd1.out‚ 50.1904296875 disk.sda1.in‚0 disk.sda3.in‚ 106.21875 network.recv‚ 4.78394e+06 disk.drbd2165.out‚0 disk.sda2.in‚0$ memory_percent‚82.77904987335205 cpu‚25.589063955467505$ disk.dm-0.out‚142332.69677734375 disk.sda.out‚ 1260.54296875 disk.dm-1.in‚ 106.21875 disk.sda1.out‚0 disk.drbd2165.in‚0) disk.nvme0c0n1.out‚142332.69677734375) proc.memory.availableMB‚ 2821.48046875 proc.cpu.threads‚17" disk.nvme0c0n1.in‚ 45326.578125 disk.dm-0.in‚ 45326.578125 disk.sda3.out‚ 1260.54296875 disk.dm-1.out‚ 1260.54296875 disk.nvme0n1.in‚ 45326.578125 disk.sda2.out‚0 proc.memory.rssMB‚2804.875 network.sent‚ 2.2299771e+07 disk.sda.in‚ 106.21875& disk.rbd0.in‚1.759218604441423e+13* proc.memory.percent‚17.119598388671875xJ$sŸç.‚(:” ¦ÊéÒÆ«þö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚266.863' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚53.372600000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0'ÆÛ Bè.‚(:· ¦ÊéÒÚ®ôù proc.cpu.threads‚17 disk.nvme0n1.in‚ 45326.796875 disk.sda2.out‚0 disk.rbd1.in‚ 0.52392578125 disk.dm-1.in‚ 106.21875 disk.dm-1.out‚ 1260.82421875 disk.rbd0.out‚0 disk.sda3.out‚ 1260.82421875$ memory_percent‚82.77919292449951) proc.memory.availableMB‚ 2821.45703125 cpu‚25.588979967928356 network.sent‚ 2.2305366e+07$ disk./.usageGB‚563.0169982910156 proc.memory.rssMB‚2804.875 disk.sda2.in‚0 disk.sda1.out‚0 disk.sda3.in‚ 106.21875& disk.nvme0n1.out‚142342.9873046875 disk.sda.in‚ 106.21875 disk.sda.out‚ 1260.82421875& disk.rbd0.in‚1.759218604441423e+13 disk.dm-0.in‚ 45326.796875# disk.dm-0.out‚142342.9873046875 disk.drbd2165.in‚0" disk.nvme0c0n1.in‚ 45326.796875* proc.memory.percent‚17.119598388671875( disk.nvme0c0n1.out‚142342.9873046875 network.recv‚ 4.787164e+06 disk.sda1.in‚0 disk.drbd2165.out‚0) disk./.usagePercent‚37.78454315803685 disk.rbd1.out‚ 50.1904296875 disk.rbd3.in‚0 disk.rbd3.out‚ 0.22412109375î V4é./ _wandbruntime‚19225  _runtime‚19225ÝžºÊê.Ä trainer/global_step‚11599 % _timestamp‚1.7843090409457679e+09 _runtime‚19225.16295714 _step‚952 ' lr-AdamW/pg1‚0.00026900290151674213 ' lr-AdamW/pg2‚0.00026900290151674213šJc‰Ïë.É % _timestamp‚1.7843090409457679e+09 _runtime‚19225.16295714 _step‚952 ' lr-AdamW/pg1‚0.00026900290151674213 ' lr-AdamW/pg2‚0.00026900290151674213 trainer/global_step‚11599¸³%.4ì./ _wandbruntime‚19225  _runtime‚19225“êA(í.ŠP MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3342987609727488 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚32854704 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚900381223358464 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚3342987609727488 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚3330640590786560 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 100422300 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 100432273 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _runtime‚19225.164204311 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚62607678 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚29752703 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3332079425544192 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 100422300 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36633600 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚3330640565358592 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10908184183296 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚62607407 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚98890139 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 199322412 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚914445135717376 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3341531456087726 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 100432273 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚98890133 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚3332079425544192 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 _step‚953 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚3341530086059436 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚914444730992640 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10889520700844 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3342988981290496 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚29752708 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚914445135717376 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3341531456087726 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 100432273 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚98890133 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚3332079425544192 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 _step‚953 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚3341530086059436 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚914444730992640 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10889520700844 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3342988981290496 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚29752708 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3342987609727488 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚32854704 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚900381223358464 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚3342987609727488 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚3330640590786560 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 100422300 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 100432273 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _runtime‚19225.164204311 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚62607678 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚29752703 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3332079425544192 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 100422300 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36633600 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚3330640565358592 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10908184183296 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872¹ä e4ï./ _wandbruntime‚19226  _runtime‚19226• üð.”N E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10890444167090 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 _step‚954 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚98898345 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚62612957 _runtime‚19226.54965538 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚98898345 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3332357452447744 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3330918472564736 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3343267907063296 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚900458709766144 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 100440765 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 199339110 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚32857706 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚914523396444160 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚29755251 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14064686678016 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10909109218816 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚62612696 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10891788767524 WGDeviceStatsMonitor.on_train_batch_end/reswB\¹ erved_bytes.large_pool.current‚ 47351595008 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3343266535493120 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚29755246 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚256 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14064723304448 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 199329117 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 100430778 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚98898339 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10910454615552 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚32857450 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3341810261332260 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 trainer/global_step‚11599 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚98898339 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504í<ôÖ'ñ.™N H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 199339110 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 199329117 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚98898345 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚62612957 _runtime‚19226.54965538 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚98898345 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3332357452447744 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3330918472564736 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3343267907063296 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚900458709766144 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 100440765 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 199339110 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚32857706 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚914523396444160 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚29755251 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14064686678016 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10909109218816 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚62612696 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10891788767524 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3343266535493120 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚29755246 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚256 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14064723304448 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 199329117 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 100430778 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚98898339 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10910454615552 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚32857450 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3341810261332260 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 trainer/global_step‚11599 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚98898339 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10890444167090 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 _step‚954 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 B&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.1694391965866089 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.4628221392631531 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 particle/accuracy_3_of_3‚1 6 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.942307710647583 jet/accuracy_1_of_5‚1 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8629205822944641 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 *jet/accuracy_2_of_4‚0.7478668689727783 particle/accuracy_2_of_3‚1 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7370359301567078 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.021154994145035744 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7976946234703064 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.18896345794200897 particle/accuracy_3_of_4‚1 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6517536640167236 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.5598958730697632 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.028119683265686035 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.03432755172252655 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.017098452895879745 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9970017671585083 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.18574312329292297 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.15343768894672394 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.11522343009710312 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.1402275264263153 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9960039854049683 , Purity/0FRt2SRqqt1FBt/FBt_purity‚0.96875 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.48245611786842346 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.5922490358352661 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9814814329147339 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 *jet/accuracy_3_of_3‚0.0718173235654831 +particle/specificity‚0.9037465453147888 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5798068046569824 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.02278808504343033 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8101851940155029 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8304316401481628 8"Purity/0FRt*SRqqt*FBt/event_purity‚0.620089054107666 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.5825408101081848 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8101851940155029 ) Purity/2FRt*SRqqt2FBt/FRt_purity‚0.75 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6959940791130066 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.18701235949993134 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.17369401454925537 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6187440752983093 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7375471591949463 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8071429133415222 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9904571771621704 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5762588977813721 8"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.660709023475647 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7175413966178894 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.42129629850387573 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 *jet/accuracy_1_of_2‚0.7177823185920715 /particle/accuracy_4_of_4‚0.9359927177429199 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6106952428817749 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9970346093177795 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.4933333396911621 *jet/accuracy_1_of_3‚0.8914276361465454 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.5714285969734192 9"Purity/*FRt2SRqqt*FBt/event_purity‚0.1981169879436493 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.43836358189582825 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.0974741205573082 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.15749499201774597 _runtime‚19253.028056656 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.17339469492435455 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6073070168495178 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9954171180725098 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9922067523002625 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 8"Purity/1FRt*SRqqt0FBt/event_purity‚0.130881667137146 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7384824752807617 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8638511896133423 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9981850981712341 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9787510633468628 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9893755316734314 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9937976002693176 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 particle/accuracy_2_of_2‚1 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.11538461595773697 particle/accuracy_3_of_5‚1 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.23942218720912933 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5633923411369324 /"Purity/2FRt0SRqqt1FBt/event_purity‚0.234375 particle/accuracy_2_of_5‚1 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.10572153329849243 particle/accuracy_4_of_5‚1 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9232968688011169 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.14205147325992584 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7455910444259644 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3499999940395355 particle/accuracy_1_of_2‚1 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9961785674095154 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.27886828780174255 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 6 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.997636616230011 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7134841680526733 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 jet/accuracy_2_of_5‚1 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.24048054218292236 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4296875 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9988662004470825 jet/accuracy_3_of_5‚1 trainer/global_step‚11599 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6681514978408813 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9944478273391724 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.03461533039808273 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 % _timestamp‚1.7843090688101702e+09 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9211474657058716 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6532559990882874 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 'particle/f_score‚0.6201565861701965 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.47985348105430603 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.16655097901821136 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7284769415855408 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 )jet/accuracy_1_of_1‚0.287670373916626 *jet/accuracy_2_of_3‚0.5208443999290466 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.12380415201187134 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 5 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.55209881067276 8"Purity/0FRt*SRqqt0FBt/event_purity‚0.563618540763855 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.5738397240638733 6 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.808295488357544 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 ,jet/accuracy_4_of_4‚0.013899499550461769 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.16294758021831512 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5936055183410645 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6400655508041382 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 particle/accuracy_1_of_1‚1 particle/accuracy_1_of_3‚1 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7320296168327332 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6677135229110718 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.17369401454925537 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9814814329147339 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8304316401481628 6validation_average_jet_accuracy‚0.3319680988788605 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.15005344152450562 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.004261198919266462 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 jet/accuracy_4_of_5‚0 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8867430090904236 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9911041259765625 , Purity/0FRt2SRqqt*FBt/FBt_purity‚0.96875 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 6 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.803646981716156 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 8"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.648679792881012 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.3009641170501709 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 *"Purity/2FRt*SRqqt2FBt/event_purity‚0.5 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.1867813616991043 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7373812794685364 6 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.996728777885437 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6316797137260437 6 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.148982435464859 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9981850981712341 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5762588977813721 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.4512682557106018 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 _step‚956 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.011161720380187035 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 *jet/accuracy_2_of_2‚0.1448514610528946 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6700452566146851 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9940176606178284 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 jet/accuracy_5_of_5‚0 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3410474359989166 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.16363508999347687 )jet/accuracy_1_of_4‚0.968207597732544 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6539769768714905 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.33643069863319397 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8048753142356873 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9867116212844849 )loss/val_total_loss‚8.778519358771502 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7078481912612915 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5647892951965332 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.994884848594g®ñ¬6655 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9987687468528748 particle/accuracy_1_of_4‚1 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.11538461595773697 particle/accuracy_1_of_5‚1 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6311888694763184 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.007895393297076225 particle/accuracy_5_of_5‚1 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9860765337944031 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6707918047904968 +jet/accuracy_3_of_4‚0.09157896786928177 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7857142686843872<º>éû=û.õ{ 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6319804191589355 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6460762023925781 particle/accuracy_2_of_4‚1 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.05000000074505806 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.05000000074505806 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.10666666179895401 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.17349153757095337 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.02849261276423931 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.20186379551887512 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.03464503958821297 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.42824071645736694 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.017098452895879745 (particle/accuracy‚0.8401498198509216 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.44731512665748596 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9971989393234253 +particle/sensitivity‚0.6076356172561646 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.1694391965866089 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.4628221392631531 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 particle/accuracy_3_of_3‚1 6 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.942307710647583 jet/accuracy_1_of_5‚1 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8629205822944641 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 *jet/accuracy_2_of_4‚0.7478668689727783 particle/accuracy_2_of_3‚1 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7370359301567078 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.021154994145035744 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7976946234703064 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.18896345794200897 particle/accuracy_3_of_4‚1 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6517536640167236 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.5598958730697632 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.028119683265686035 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.03432755172252655 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.017098452895879745 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9970017671585083 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.18574312329292297 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.15343768894672394 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.11522343009710312 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.1402275264263153 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9960039854049683 , Purity/0FRt2SRqqt1FBt/FBt_purity‚0.96875 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.48245611786842346 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.5922490358352661 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9814814329147339 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 *jet/accuracy_3_of_3‚0.0718173235654831 +particle/specificity‚0.9037465453147888 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5798068046569824 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.02278808504343033 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8101851940155029 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8304316401481628 8"Purity/0FRt*SRqqt*FBt/event_purity‚0.620089054107666 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.5825408101081848 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8101851940155029 ) Purity/2FRt*SRqqt2FBt/FRt_purity‚0.75 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6959940791130066 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.18701235949993134 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.17369401454925537 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6187440752983093 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7375471591949463 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8071429133415222 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9904571771621704 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5762588977813721 8"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.660709023475647 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7175413966178894 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.42129629850387573 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 *jet/accuracy_1_of_2‚0.7177823185920715 /particle/accuracy_4_of_4‚0.9359927177429199 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6106952428817749 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9970346093177795 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.4933333396911621 *jet/accuracy_1_of_3‚0.8914276361465454 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.5714285969734192 9"Purity/*FRt2SRqqt*FBt/event_purity‚0.1981169879436493 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.43836358189582825 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.0974741205573082 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.15749499201774597 _runtime‚19253.028056656 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.17339469492435455 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6073070168495178 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9954171180725098 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9922067523002625 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 8"Purity/1FRt*SRqqt0FBt/event_purity‚0.130881667137146 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7384824752807617 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8638511896133423 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9981850981712341 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9787510633468628 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9893755316734314 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9937976002693176 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 particle/accuracy_2_of_2‚1 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.11538461595773697 particle/accuracy_3_of_5‚1 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.23942218720912933 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5633923411369324 /"Purity/2FRt0SRqqt1FBt/event_purity‚0.234375 particle/accuracy_2_of_5‚1 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.10572153329849243 particle/accuracy_4_of_5‚1 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9232968688011169 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.14205147325992584 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7455910444259644 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3499999940395355 particle/accuracy_1_of_2‚1 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9961785674095154 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.27886828780174255 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 6 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.997636616230011 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7134841680526733 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 jet/accuracy_2_of_5‚1 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.24048054218292236 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4296875 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9988662004470825 jet/accuracy_3_of_5‚1 trainer/global_step‚11599 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6681514978408813 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9944478273391724 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.03461533039808273 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 % _timestamp‚1.7843090688101702e+09 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9211474657058716 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6532559990882874 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 'particle/f_score‚0.6201565861701965 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.47985348105430603 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.16655097901821136 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7284769415855408 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 )jet/accuracy_1_of_1‚0.287670373916626 *jet/accuracy_2_of_3‚0.5208443999290466 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.12380415201187134 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 5 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.55209881067276 8"Purity/0FRt*SRqqt0FBt/event_purity‚0.563618540763855 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.5738397240638733 6 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.808295488357544 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 ,jet/accuracy_4_of_4‚0.013899499550461769 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.16294758021831512 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5936055183410645 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6400655508041382 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 particle/accuracy_1_of_1‚1 particle/accuracy_1_of_3‚1 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7320296168327332 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6677135229110718 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.17369401454925537 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9814814329147339 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8304316401481628 6validation_average_jet_accuracy‚0.3319680988788605 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.15005344152450562 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.004261198919266462 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 jet/accuracy_4_of_5‚0 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8867430090904236 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9911041259765625 , Purity/0FRt2SRqqt*FBt/FBt_purity‚0.96875 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 6 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.803646981716156 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 8"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.648679792881012 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.3009641170501709 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 *"Purity/2FRt*SRqqt2FBt/event_purity‚0.5 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.1867813616991043 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7373812794685364 6 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.996728777885437 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6316797137260437 6 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.148982435464859 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9981850981712341 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5762588977813721 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.4512682557106018 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 _step‚956 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.011161720380187035 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 *jet/accuracy_2_of_2‚0.1448514610528946 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6700452566146851 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9940176606178284 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 jet/accuracy_5_of_5‚0 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3410474359989166 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.16363508999347687 )jet/accuracy_1_of_4‚0.968207597732544 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6539769768714905 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.33643069863319397 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8048753142356873 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9867116212844849 )loss/val_total_loss‚8.778519358771502 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7078481912612915 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5647892951965332 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9948848485946655 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9987687468528748 particle/accuracy_1_of_4‚1 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.11538461595773697 particle/accuracy_1_of_5‚1 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6311888694763184 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.007895393297076225 particle/accuracy_5_of_5‚1 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9860765337944031 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6707918047904968 +jet/accuracy_3_of_4‚0.09157896786928177 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7857142686843872 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 epoch‚28¼iqÚ•ü.‚(:Š ÓÊéÒ”È÷ gpu.0.gpu‚100.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚346.941' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚69.3882 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Õª­Jý.‚(:¿ ÓÊéÒ¬ÿ¬ø" disk.nvme0n1.in‚45433.67578125$ memory_percent‚82.77781009674072 disk.rbd1.in‚ 0.52392578125 disk.sda.in‚ 106.21875& disk.rbd0.in‚1.759218604441423e+13 cpu‚25.592432658502254 disk.sda3.out‚ 1261.53515625 disk.drbd2165.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-1.in‚ 106.21875 proc.memory.rssMB‚2804.875 network.sent‚ 2.2378873e+07 disk.rbd3.in‚0 disk.rbd1.out‚ 50.3310546875 disk.dm-1.out‚ 1261.53515625$ disk.dm-0.out‚142375.29541015625) proc.memory.availableMB‚ 2821.68359375) disk./.usagePercent‚37.78571260045326 disk.dm-0.in‚45433.67578125 proc.cpu.threads‚17 disk.sda1.out‚0 disk.rbd0.out‚0 disk.rbd3.out‚ 0.22412109375 disk.drbd2165.out‚0' disk.nvme0n1.out‚142375.29541015625 disk.sda.out‚ 1261.53515625* proc.memory.percent‚17.119598388671875$ disk.nvme0c0n1.in‚45433.67578125) disk.nvme0c0n1.out‚142375.29541015625 network.recv‚ 4.797941e+06# disk./.usageGB‚563.034423828125 disk.sda1.in‚0 disk.sda3.in‚ 106.21875¤Mnžþ.‚(:“ âÊéÒµìüö gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚267.95' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚53.589999999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0b£·Dÿ.‚(:¹ âÊéÒÉ™³ù disk.sda3.out‚ 1261.74609375 disk.sda2.in‚0 disk.dm-1.out‚ 1261.74609375 disk.sda1.out‚0 disk.rbd3.out‚ 0.22412109375 disk.sda.in‚ 106.21875 disk.dm-0.in‚ 45434.078125 disk.sda2.out‚0 network.sent‚ 2.2436661e+07 disk.dm-1.in‚ 106.21875& disk.rbd0.in‚1.759218604441423e+13 disk.nvme0n1.in‚ 45434.078125' disk.nvme0n1.out‚142391.07275390625 disk.rbd1.in‚ 0.52392578125) proc.memory.availableMB‚ 2821.69140625 proc.cpu.threads‚17 proc.memory.rssMB‚2804.875 network.recv‚ 4.802151e+06) disk./.usagePercent‚37.78608381137791$ disk./.usageGB‚563.0399551391602 disk.drbd2165.in‚0 disk.rbd3.in‚0) disk.nvme0c0n1.out‚142391.07275390625 disk.sda.out‚ 1261.74609375" disk.nvme0c0n1.in‚ 45434.078125 disk.rbd1.out‚ 50.3310546875* proc.memory.percent‚17.119598388671875 cpu‚25.592267189828704$ disk.dm-0.out‚142391.07275390625 disk.sda1.in‚0 disk.sda3.in‚ 106.21875 disk.drbd2165.out‚0# memory_percent‚82.7777624130249 disk.rbd0.out‚0J$iØž€/‚(:“ ñÊéÒ¨á­÷ gpu.0.gpu‚90.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚264.23' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚52.846000000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ó Kÿ</‚(:± ñÊéÒÿ·Ùù disk.sda1.in‚0 disk.rbd3.in‚0) proc.memory.availableMB‚ 2821.58203125 network.recv‚ 4.805504e+06$ disk./.usageGB‚563.0468292236328* proc.memory.percent‚17.119598388671875 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.sda1.out‚0 disk.rbd1.in‚ 0.52392578125$ memory_percent‚82.77842998504639 proc.cpu.threads‚17 network.sent‚ 2.2441755e+07 disk.dm-1.out‚ 1261.98046875 disk.sda3.in‚ 106.21875 disk.sda3.out‚ 1261.98046875 disk.rbd3.out‚ 0.22412109375! disk.nvme0c0n1.in‚ 45434.28125 disk.rbd1.out‚ 50.3935546875 disk.dm-1.in‚ 106.21875& disk.rbd0.in‚1.759218604441423e+13 disk.rbd0.out‚0" disk.dm-0.out‚142399.119140625 disk.nvme0n1.in‚ 45434.28125 proc.memory.rssMB‚2804.875 cpu‚25.592104143146482 disk.dm-0.in‚ 45434.28125' disk.nvme0c0n1.out‚142399.119140625% disk.nvme0n1.out‚142399.119140625 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda.out‚ 1261.98046875) disk./.usagePercent‚37.78654513695461 disk.sda.in‚ 106.21875!ËcŸ‚/‚(:” €ËéÒü‰‚ø gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚301.808' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚60.361599999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0HìK7ƒ/‚(:¬ €ËéÒâ°âù network.recv‚ 4.808406e+06 disk.sda.out‚ 1262.390625 disk.dm-1.in‚ 106.21875 disk.drbd2165.in‚0 disk.rbd3.in‚0" disk.nvme0c0n1.in‚ 45434.703125 disk.sda1.out‚0 cpu‚25.59196777081014 network.sent‚ 2.2446884e+07' disk.nvme0c0n1.out‚142409.431640625 disk.rbd1.in‚ 0.52392578125 disk.sda3.in‚ 106.21875 disk.drbd2165.out‚0$ memory_percent‚82.77981281280518* proc.memory.percent‚17.119598388671875( disk./.usagePercent‚37.7870033904409 disk.rbd3.out‚ 0.22412109375 disk.sda2.out‚0 proc.memory.rssMB‚2804.875 proc.cpu.threads‚17% disk.nvme0n1.out‚142409.431640625 disk.sda2.in‚0 disk.dm-0.in‚ 45434.703125) proc.memory.availableMB‚ 2821.35546875 disk.sda3.out‚ 1262.390625 disk.rbd1.out‚ 50.3935546875 disk.sda.in‚ 106.21875& disk.rbd0.in‚1.759218604441423e+13 disk.rbd0.out‚0" disk.dm-0.out‚142409.431640625 disk.sda1.in‚0$ disk./.usageGB‚563.0536575317383 disk.nvme0n1.in‚ 45434.703125 disk.dm-1.out‚ 1262.390625Îà‚ÍŸ„/‚(:” ËéÒ•ö÷ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚280.575' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚56.114999999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ž€¯VA…/‚(:¶ ËéÒ†ÙÉù disk.sda.in‚ 106.21875* proc.memory.percent‚17.119598388671875 cpu‚25.591780632401786 disk.sda1.in‚0 disk.drbd2165.in‚0 disk.sda2.in‚0$ memory_percent‚82.77974128723145 network.recv‚ 4.811736e+06) disk./.usagePercent‚37.78729626306007 disk.rbd3.out‚ 0.22412109375' disk.nvme0n1.out‚142423.06982421875 disk.sda.out‚ 1262.6953125 disk.dm-1.in‚ 106.21875 disk.sda3.in‚ 106.21875 disk.nvme0n1.in‚ 45587.296875$ disk./.usageGB‚563.0580215454102 disk.dm-0.in‚ 45587.296875 disk.sda3.out‚ 1262.6953125 disk.rbd3.in‚0" disk.nvme0c0n1.in‚ 45587.296875 proc.cpu.threads‚17 disk.rbd0.out‚0 network.sent‚ 2.2451422e+07 disk.sda2.out‚0 disk.rbd1.in‚ 0.52392578125 disk.dm-1.out‚ 1262.6953125& disk.rbd0.in‚1.759218604441423e+13 proc.memory.rssMB‚2804.875$ disk.dm-0.out‚142423.06982421875 disk.drbd2165.out‚0) disk.nvme0c0n1.out‚142423.06982421875( proc.memory.availableMB‚ 2821.3671875 disk.sda1.out‚0 disk.rbd1.out‚ 50.4716796875*ü¢;4†// _wandbruntime‚19330  _runtime‚19330ST1jˇ/Å ' lr-AdamW/pg1‚0.00026630248084102807 ' lr-AdamW/pg2‚0.00026630248084102807 trainer/global_step‚11649 % _timestamp‚1.7843091458867216e+09 _runtime‚19330.103996979 _step‚957‚y¢^Ј/Ê ' lr-AdamW/pg1‚0.00026630248084102807 ' lr-AdamW/pg2‚0.00026630248084102807 trainer/global_step‚11649 % _timestamp‚1.7843091458867216e+09 _runtime‚19330.103996979 _step‚957½ͯËÇ4‰// _wandbruntime‚19330  _runtime‚19330.­åŽŠ/‹P F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10961688115200 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚3360767786210304 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 200349452 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚29888771 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚32439296 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14130856051200 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 trainer/global_step‚11649 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3349807469658112 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚99432439 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10960342725632 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚3349807443484672 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10960342725632 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚99432445 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚918434880131584 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10961688115200 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 100917007 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3360767786210304 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚99432439 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 M=DeviceStaõøyóƒtsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 100907034 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚245 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚19330.105217192 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚62916303 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚904303655989248 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 100907034 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3360769157773312 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 400530432 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚99432445 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10941586945389 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648ܤ™Í(‹/P aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚3348360588627968 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚3359302150145389 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14130823611904 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10960342725632 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚99432445 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚918434880131584 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10961688115200 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 100917007 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3360767786210304 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚99432439 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 100907034 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚245 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚19330.105217192 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚62916303 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚904303655989248 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 100907034 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3360769157773312 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 400530432 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚99432445 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10941586945389 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10961688115200 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚3360767786210304 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 200349452 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚29888771 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚32439296 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14130856051200 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 trainer/global_step‚11649 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3349807469658112 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚99432439 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10960342725632 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚3349807443484672 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 200356157 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 400523264 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚29891319 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚29891314 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚62921633 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 200366150 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚99440645 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚918513573658112 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14132063170048 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 200366150 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚99440645 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚3350085470388224 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10962613157888 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10942510411635 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚99440651 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚99440651 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚237 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚33030319 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 _step‚959 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚62921633 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 200366150 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚99440645 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚918513573658112 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14132063170048 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 200366150 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚99440645 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚3350085470388224 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10962613157888 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10942510411635 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚99440651 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚99440651 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚237 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚33030319 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 _step‚959 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 200356157 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 400523264 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚29891319 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚29891314 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚30015921 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3374714138646528 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚30015926 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14190997505536 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14190962969088 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11006658656256 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚33167527 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚3362254652108800 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3374715510209536 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚63183712 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚63183448 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚33167786 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3374715510209536 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚922369341969920 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚3374714138646528 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11008004045824 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11008004045824 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3363707506163712 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚3363707506163712 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 201185504 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 101342759 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11006658656256 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 101332786 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0-rÁ(¡/P SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11006658656256 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚33167527 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚3362254652108800 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3374715510209536 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚63183712 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚63183448 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚33167786 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3374715510209536 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚922369341969920 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚3374714138646528 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11008004045824 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11008004045824 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3363707506163712 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚3363707506163712 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 201185504 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 101342759 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11006658656256 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 101332786 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚99842739 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚30015921 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3374714138646528 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚30015926 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14190997505536 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14190962969088 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008­۴]Ÿ¢/‚(:” éËéÒöȈ÷ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚257.462' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚51.492399999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0RE1¶H£/‚(:½ éËéÒ·­Åù disk.sda3.out‚ 1264.29296875) disk./.usagePercent‚37.78971015810042 disk.sda.out‚ 1264.29296875 disk.dm-1.out‚ 1264.29296875 disk.sda1.in‚0 disk.rbd3.in‚0) disk.nvme0c0n1.out‚142490.71337890625" disk.nvme0n1.in‚45729.74609375 disk.sda2.out‚0$ disk.nvme0c0n1.in‚45729.74609375' disk.nvme0n1.out‚142490.71337890625 cpu‚25.590815360732492 network.sent‚ 2.2540902e+07 disk.dm-1.in‚ 106.21875 disk.dm-0.in‚45729.74609375$ disk.dm-0.out‚142490.71337890625 disk.sda3.in‚ 106.21875 disk.drbd2165.in‚0 disk.rbd1.in‚ 0.52392578125 disk.rbd1.out‚ 50.6748046875 disk.sda1.out‚0 disk.rbd3.out‚ 0.22412109375# memory_percent‚82.7812671661377 proc.memory.rssMB‚2804.875 disk.sda.in‚ 106.21875 disk.sda2.in‚0( proc.memory.availableMB‚ 2821.1171875* proc.memory.percent‚17.119598388671875$ disk./.usageGB‚563.0939903259277& disk.rbd0.in‚1.759218604441423e+13 disk.rbd0.out‚0 disk.drbd2165.out‚0 proc.cpu.threads‚17 network.recv‚ 4.83359e+06’;Ö04¤// _wandbruntime‚19410  _runtime‚19410ˆá´Y¥/”N I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10990092125177 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11008929088512 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 _runtime‚19410.239082196 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 KEDeviceStatsMonitor.on_trÍKjÁ"ain_batch_end/reserved_bytes.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11007583691776 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3363985506893824 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚30018469 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 201192209 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 101341264 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚3363985480720384 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚99850951 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 101341264 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚33170321 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚33170575 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚3373521281411719 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 $ _timestamp‚1.784309226021587e+09 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 101351251 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3363985480720384 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14192162574848 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3373522651440121 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚30018464 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 201202202 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10988747524743 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3374994435982336 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3374994435982336 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11007583691776 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚908255830872064 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14192197104128 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 201202202 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚3363985506893824 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3374993064412160 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 201192209 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚11699 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚259 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚99850951 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚63188785 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚908255462780928 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 101351251 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3362532559314944 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3362532533886976v¯ô'¦/™N trainer/global_step‚11699 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚259 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚99850951 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚63188785 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚908255462780928 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 101351251 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3362532559314944 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3362532533886976 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10990092125177 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11008929088512 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 _runtime‚19410.239082196 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11007583691776 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3363985506893824 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚30018469 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 201192209 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 101341264 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚3363985480720384 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚99850951 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 101341264 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚33170321 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚33170575 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚3373521281411719 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 $ _timestamp‚1.784309226021587e+09 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 101351251 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3363985480720384 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14192162574848 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3373522651440121 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚30018464 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 201202202 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10988747524743 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3374994435982336 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3374994435982336 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11007583691776 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚908255830872064 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14192197104128 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 201202202 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚3363985506893824 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3374993064412160 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 201192209 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0Ãf)äµ4§// _wandbruntime‚19410  _runtime‚19410“Òoj¨/Š _step‚964 0loss/FRt2/assignment_loss‚0.3207059287782853 3loss/SRqqt2/assignment_loss‚0.22735707557237617 3loss/FBt1/assignment_loss‚0.0031827203362174337 3loss/FBt2/assignment_loss‚0.0002043741065060658 1loss/SRqqt2/detection_loss‚0.5354763735811658 %loss/total_loss‚5.620491190184721 0loss/FRt1/assignment_loss‚1.1036375826707592 0loss/FRt1/detection_loss‚0.45705269163698353 /loss/FRt2/detection_loss‚1.2765921575952732 0loss/FBt2/detection_loss‚0.30686946948454713 $ _timestamp‚1.784309226107566e+09 _runtime‚19410.324579259 epoch‚29 2loss/SRqqt1/assignment_loss‚0.1576756990064686 1loss/SRqqt1/detection_loss‚0.6583352777169241 /loss/FBt1/detection_loss‚0.5734018396992141 trainer/global_step‚11699•[A^©/ 3loss/SRqqt2/assignment_loss‚0.22735707557237617 3loss/FBt1/assignment_loss‚0.0031827203362174337 3loss/FBt2/assignment_loss‚0.0002043741065060658 1loss/SRqqt2/detection_loss‚0.5354763735811658 %loss/total_loss‚5.620491190184721 0loss/FRt1/assignment_loss‚1.1036375826707592 0loss/FRt1/detection_loss‚0.45705269163698353 /loss/FRt2/detection_loss‚1.2765921575952732 0loss/FBt2/detection_loss‚0.30686946948454713 $ _timestamp‚1.784309226107566e+09 _runtime‚19410.324579259 epoch‚29 2loss/SRqqt1/assignment_loss‚0.1576756990064686 1loss/SRqqt1/detection_loss‚0.6583352777169241 /loss/FBt1/detection_loss‚0.5734018396992141 trainer/global_step‚11699 _step‚964 0loss/FRt2/assignment_loss‚0.3207059287782853Ä2-o~•ª/‚(:Š øËéÒ®àëö gpu.0.gpu‚100.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚343.993' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚68.7986 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0T)éßF«/‚(:» øËéÒþ‚±ù& disk.rbd0.in‚1.759218604441423e+13 disk.sda3.in‚ 106.21875$ disk.nvme0c0n1.in‚45729.94921875' disk.nvme0c0n1.out‚142501.681640625 disk.sda2.out‚0 proc.memory.rssMB‚2804.875 network.sent‚ 2.2593414e+07 disk.sda3.out‚ 1264.44921875 disk.sda1.out‚0 network.recv‚ 4.837769e+06 disk.dm-1.in‚ 106.21875 disk.dm-0.in‚45729.94921875 disk.sda2.in‚0$ memory_percent‚82.77995586395264 disk.rbd3.in‚0 disk.rbd3.out‚ 0.22412109375 cpu‚25.590668629286895 disk.rbd1.out‚ 50.6748046875$ disk./.usageGB‚563.1020164489746 disk.sda.out‚ 1264.44921875 disk.rbd1.in‚ 0.52392578125) proc.memory.availableMB‚ 2821.33203125* disk./.usagePercent‚37.790248797952465 disk.sda.in‚ 106.21875 disk.dm-1.out‚ 1264.44921875 disk.sda1.in‚0 disk.drbd2165.in‚0 disk.drbd2165.out‚0 proc.cpu.threads‚17 disk.rbd0.out‚0" disk.dm-0.out‚142501.681640625" disk.nvme0n1.in‚45729.94921875* proc.memory.percent‚17.119598388671875% disk.nvme0n1.out‚142501.681640625d ¶™ž¬/‚(:“ ‡ÌéÒõ’êö gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚262.054' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚52.41079999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Wé|F­/‚(:» ‡ÌéÒß°©ù disk.rbd1.in‚ 0.52392578125 disk.sda.in‚ 106.21875# memory_percent‚82.7815294265747 proc.memory.rssMB‚2804.875& disk.rbd0.in‚1.759218604441423e+13) disk.nvme0c0n1.out‚142514.90869140625 disk.rbd1.out‚ 50.7529296875 disk.rbd3.in‚0 disk.rbd3.out‚ 0.22412109375 disk.sda2.in‚0 cpu‚25.590496160147254 proc.cpu.threads‚17 disk.sda2.out‚0 disk.sda.out‚ 1264.58203125 disk.sda1.in‚0 disk.sda3.in‚ 106.21875 disk.dm-1.out‚ 1264.58203125# disk./.usageGB‚563.108512878418" disk.nvme0n1.in‚45867.19140625 disk.dm-1.in‚ 106.21875 network.recv‚ 4.842e+06 disk.dm-0.in‚45867.19140625 disk.sda1.out‚0 disk.sda3.out‚ 1264.58203125$ disk.nvme0c0n1.in‚45867.19140625* proc.memory.percent‚17.119598388671875 disk.drbd2165.out‚0 network.sent‚ 2.2645972e+07) disk./.usagePercent‚37.79068477878327 disk.rbd0.out‚0 disk.drbd2165.in‚0' disk.nvme0n1.out‚142514.90869140625) proc.memory.availableMB‚ 2821.07421875$ disk.dm-0.out‚142514.90869140625°¬·ž®/‚(:“ –ÌéÒöǯ÷ gpu.0.gpu‚99.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚335.114' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚67.02279999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0C*šG¯/‚(:¼ –ÌéÒè߶ù disk.rbd1.in‚ 0.52392578125 disk.dm-1.in‚ 106.21875 proc.cpu.threads‚17 network.recv‚ 4.845297e+06# disk.dm-0.out‚142525.1943359375 disk.sda2.out‚0 disk.sda.in‚ 106.21875 disk.sda3.out‚ 1264.85546875 disk.rbd3.out‚ 0.22412109375$ disk.nvme0c0n1.in‚45867.41015625$ memory_percent‚82.78002738952637* proc.memory.percent‚17.119598388671875 disk.rbd0.out‚0 disk.drbd2165.out‚0( proc.memory.availableMB‚ 2821.3203125 disk.rbd3.in‚0 proc.memory.rssMB‚2804.875& disk.nvme0n1.out‚142525.1943359375 cpu‚25.590349187934997 network.sent‚ 2.2651064e+07$ disk./.usageGB‚563.1130714416504( disk.nvme0c0n1.out‚142525.1943359375" disk.nvme0n1.in‚45867.41015625 disk.rbd1.out‚ 50.7529296875) disk./.usagePercent‚37.79099070778669& disk.rbd0.in‚1.759218604441423e+13 disk.dm-0.in‚45867.41015625 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.sda2.in‚0 disk.sda.out‚ 1264.85546875 disk.dm-1.out‚ 1264.85546875 disk.sda1.in‚0 disk.sda3.in‚ 106.21875ú‚=”°/‚(:‰ ¥ÌéÒÝþ÷ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚265.534' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.1068 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ô‹çÊJ±/‚(:¿ ¥ÌéÒ´ùºù disk.sda3.out‚ 1265.17578125 disk.rbd3.in‚0 disk.sda2.in‚0 disk.sda2.out‚0$ memory_percent‚82.78002738952637( proc.memory.availableMB‚ 2821.3203125 disk.rbd0.out‚0 disk.rbd1.in‚ 0.52392578125 proc.memory.rssMB‚2804.875* proc.memory.percent‚17.119598388671875 proc.cpu.threads‚17 disk.dm-1.out‚ 1265.17578125 disk.dm-0.in‚45867.83203125 disk.sda1.out‚0" disk.nvme0n1.in‚45867.83203125 network.recv‚ 4.848671e+06 disk.dm-1.in‚ 106.21875 network.sent‚ 2.2656203e+07 disk.sda.in‚ 106.21875' disk.nvme0n1.out‚142532.79052734375 disk.rbd1.out‚ 50.8154296875 cpu‚25.59017695606488$ disk./.usageGB‚563.1225242614746& disk.rbd0.in‚1.759218604441423e+13 disk.sda1.in‚0 disk.drbd2165.out‚0 disk.rbd3.out‚ 0.22412109375* disk./.usagePercent‚37.791625094456535$ disk.dm-0.out‚142532.79052734375 disk.drbd2165.in‚0$ disk.nvme0c0n1.in‚45867.83203125) disk.nvme0c0n1.out‚142532.79052734375 disk.sda.out‚ 1265.17578125 disk.sda3.in‚ 106.21875E²’²/‚(:‡ ´ÌéÒȺ£÷ gpu.0.gpu‚99.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚177.71' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.542 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0éõ‚J?³/‚(:´ ´ÌéÒ¦Œúø" disk.nvme0n1.in‚45868.03515625 disk.rbd1.in‚ 0.52392578125& disk.rbd0.in‚1.759218604441423e+13 disk.sda3.in‚ 106.21875 proc.memory.rssMB‚2804.875 disk.rbd3.in‚0) disk.nvme0c0n1.out‚142547.41552734375$ memory_percent‚82.77833461761475$ disk./.usageGB‚563.1266632080078 disk.rbd3.out‚ 0.22412109375 disk.sda.in‚ 106.21875 disk.sda1.out‚0* proc.memory.percent‚17.119598388671875 cpu‚25.59003211939607 disk.drbd2165.out‚0 network.recv‚ 4.8522e+06 disk.dm-1.out‚ 1265.28125 disk.sda3.out‚ 1265.28125) disk./.usagePercent‚37.79190286263119 disk.sda2.in‚0 disk.sda.out‚ 1265.28125$ disk.nvme0c0n1.in‚45868.03515625' disk.nvme0n1.out‚142547.41552734375 disk.rbd1.out‚ 50.8154296875 disk.rbd0.out‚0 disk.dm-0.in‚45868.03515625 disk.drbd2165.in‚0) proc.memory.availableMB‚ 2821.59765625 proc.cpu.threads‚17 network.sent‚ 2.2661388e+07 disk.sda2.out‚0 disk.dm-1.in‚ 106.21875$ disk.dm-0.out‚142547.41552734375 disk.sda1.in‚0Э4´// _wandbruntime‚19487  _runtime‚19487Þ'?ɵ/à & lr-AdamW/pg1‚0.0002609171544718216 & lr-AdamW/pg2‚0.0002609171544718216 trainer/global_step‚11749 % _timestamp‚1.7843093033848653e+09 _runtime‚19487.602057046 _step‚9659bCζ/È % _timestamp‚1.7843093033848653e+09 _runtime‚19487.602057046 _step‚965 & lr-AdamW/pg1‚0.0002609171544718216 & lr-AdamW/pg2‚0.0002609171544718216 trainer/global_step‚11749ÅïÂ:(4·// _wandbruntime‚19487  _runtime‚19487ãau4é¸/P VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚3388660491082752 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 202021556 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚63450532 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 101768511 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚63450254 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 100253045 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚3377607542669312 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3377607516495872 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3388660491082752 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 100253045 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚912052296757248 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 _step‚966 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404724736 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚30143076 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11035405771927 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 101758538 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚278 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚926302601714176 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3388661862645760 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.D;Ž,current‚ 1371563008 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚912052664848384 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚926303006438912 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚3377607516495872 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 100253039 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36633600 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11034061171605 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 100253039 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648ËX'p(¹/”P _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11054319976448 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚33307183 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚30143071 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 trainer/global_step‚11749 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11052974586880 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 % _timestamp‚1.7843093033859727e+09 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11054319976448 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11052974586880 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14250341590528 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚3376148741017600 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚3376148766445568 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 202011577 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚273 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14250304956928 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚3387182802189205 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 101758538 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3377607542669312 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚33307456 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3387184172217495 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 100253039 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚3388660491082752 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 202021556 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚63450532 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 101768511 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚63450254 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 100253045 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚3377607542669312 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3377607516495872 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3388660491082752 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 100253045 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚912052296757248 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 _step‚966 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404724736 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚30143076 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11035405771927 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 101758538 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚278 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚926302601714176 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3388661862645760 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚912052664848384 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚926303006438912 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚3377607516495872 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 100253039 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36633600 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11034061171605 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚3377885517225984 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚30145619 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14251489322496 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3377885517225984 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚912130151256064 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14251525948928 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 202038254 _runtime‚19488.991001189 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11055245019136 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚33309942 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404717568 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36626432 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚271 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚912129783164928 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚63455827 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11053899622400 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 202038254 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚63455556 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 trainer/global_step‚11749 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3377885517225984 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚912130151256064 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14251525948928 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 202038254 _runtime‚19488.991001189 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11055245019136 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚33309942 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404717568 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36626432 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚271 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚912129783164928 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚63455827 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11053899622400 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 202038254 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚63455556 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 trainer/global_step‚11749 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2Çâ´4½// _wandbruntime‚19489  _runtime‚19489vW:  ¾/„ /loss/FRt1/assignment_loss‚1.073304583719612 .loss/FRt2/detection_loss‚1.460125798190679 0loss/SRqqt1/detection_loss‚0.687345060760704 trainer/global_step‚11749 $ _timestamp‚1.784309304859188e+09 0loss/FRt2/assignment_loss‚0.2611686584040605 3loss/SRqqt1/assignment_loss‚0.16696651028215995 1loss/SRqqt2/detection_loss‚0.4892532042523785 %loss/total_loss‚5.643708496785778 2loss/SRqqt2/assignment_loss‚0.2572148384303311 /loss/FBt2/detection_loss‚0.2628770435887235 epoch‚29 _runtime‚19489.076092412 2loss/FBt1/assignment_loss‚0.008111143809519895 3loss/FBt2/assignment_loss‚0.0005953077211718934 /loss/FRt1/detection_loss‚0.4581118699079865 /loss/FBt1/detection_loss‚0.5186344777184511 _step‚968ÓÆ\x¿/‰ 2loss/SRqqt2/assignment_loss‚0.2572148384303311 /loss/FBt2/detection_loss‚0.2628770435887235 epoch‚29 _runtime‚19489.076092412 2loss/FBt1/assignment_loss‚0.008111143809519895 3loss/FBt2/assignment_loss‚0.0005953077211718934 /loss/FRt1/detection_loss‚0.4581118699079865 /loss/FBt1/detection_loss‚0.5186344777184511 _step‚968 /loss/FRt1/assignment_loss‚1.073304583719612 .loss/FRt2/detection_loss‚1.460125798190679 0loss/SRqqt1/detection_loss‚0.687345060760704 trainer/global_step‚11749 $ _timestamp‚1.784309304859188e+09 0loss/FRt2/assignment_loss‚0.2611686584040605 3loss/SRqqt1/assignment_loss‚0.16696651028215995 1loss/SRqqt2/detection_loss‚0.4892532042523785 %loss/total_loss‚5.643708496785778ÈN¯*”À/‚(:‰ ÃÌéÒññüö gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚284.051' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚56.8102 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Uëº`8Á/‚(:­ ÃÌéÒ§ÿÑù disk.sda3.in‚ 106.21875 disk.nvme0n1.in‚ 46009.28125 disk.sda.in‚ 106.21875 disk.sda3.out‚ 1265.77734375 disk.drbd2165.in‚0 disk.rbd3.out‚ 0.22412109375$ disk.nvme0n1.out‚142556.22265625 disk.rbd0.out‚0! disk.dm-0.out‚142556.22265625 disk.drbd2165.out‚0 disk.rbd1.in‚ 0.52392578125 disk.dm-1.in‚ 106.21875 disk.sda1.in‚0 cpu‚25.58985861974008 proc.cpu.threads‚17! disk.nvme0c0n1.in‚ 46009.28125& disk.nvme0c0n1.out‚142556.22265625 network.recv‚ 4.855655e+06$ disk./.usageGB‚563.1341972351074 disk.dm-1.out‚ 1265.77734375& disk.rbd0.in‚1.759218604441423e+13 disk.sda1.out‚0 proc.memory.rssMB‚2804.875* disk./.usagePercent‚37.792408477511316 disk.sda2.in‚0# memory_percent‚82.7800989151001) proc.memory.availableMB‚ 2821.30859375* proc.memory.percent‚17.119598388671875 disk.sda2.out‚0 disk.rbd1.out‚ 50.8935546875 disk.rbd3.in‚0 network.sent‚ 2.2666573e+07 disk.sda.out‚ 1265.77734375 disk.dm-0.in‚ 46009.28125oý«šžÂ/‚(:“ ÒÌéÒ¿¸Ç÷ gpu.0.gpu‚90.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚289.743' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚57.94859999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0!Â\À@Ã/‚(:µ ÒÌéÒÞ²†ù disk.rbd1.out‚ 50.8935546875) disk./.usagePercent‚37.79281245740035& disk.rbd0.in‚1.759218604441423e+13" disk.nvme0c0n1.in‚ 46009.703125 disk.nvme0n1.in‚ 46009.703125 disk.sda.in‚ 106.21875 disk.sda.out‚ 1265.9609375 disk.dm-1.out‚ 1265.9609375) disk.nvme0c0n1.out‚142567.37158203125 disk.sda2.in‚0 disk.rbd1.in‚ 0.52392578125 network.recv‚ 4.859445e+06$ disk./.usageGB‚563.1402168273926 disk.dm-1.in‚ 106.21875 disk.rbd3.in‚0 proc.cpu.threads‚17 disk.rbd3.out‚ 0.22412109375' disk.nvme0n1.out‚142567.37158203125 network.sent‚ 2.2729151e+07 disk.rbd0.out‚0$ disk.dm-0.out‚142567.37158203125 proc.memory.rssMB‚2804.875 disk.dm-0.in‚ 46009.703125 disk.sda1.out‚0 disk.sda3.out‚ 1265.9609375 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.sda1.in‚0 disk.sda2.out‚0( proc.memory.availableMB‚ 2821.8046875* proc.memory.percent‚17.119598388671875 cpu‚25.58970168472024 disk.sda3.in‚ 106.21875$ memory_percent‚82.77707099914551:UòÍŸÄ/‚(:” áÌéÒ¿å®÷ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚277.162' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚55.432399999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚04¤)¸8Å/‚(:­ áÌéÒšÀùø disk.drbd2165.in‚0 disk.rbd1.out‚ 50.9560546875 disk.dm-1.in‚ 106.21875& disk.rbd0.in‚1.759218604441423e+13$ memory_percent‚82.77585506439209) disk./.usagePercent‚37.79309662576335 disk.sda2.in‚0 disk.sda2.out‚0 proc.memory.rssMB‚2804.875 disk.sda3.in‚ 106.21875) proc.memory.availableMB‚ 2822.00390625 network.recv‚ 4.862564e+06$ disk./.usageGB‚563.1444511413574 disk.dm-0.in‚ 46009.90625 disk.sda.out‚ 1266.296875 disk.dm-1.out‚ 1266.296875 disk.sda1.out‚0 disk.rbd3.out‚ 0.22412109375 disk.nvme0n1.in‚ 46009.90625 disk.rbd0.out‚0 cpu‚25.589556430810532( disk.nvme0c0n1.out‚142581.2607421875& disk.nvme0n1.out‚142581.2607421875 disk.drbd2165.out‚0 disk.sda.in‚ 106.21875* proc.memory.percent‚17.119598388671875 disk.sda3.out‚ 1266.296875 disk.rbd3.in‚0! disk.nvme0c0n1.in‚ 46009.90625 disk.rbd1.in‚ 0.52392578125 proc.cpu.threads‚17 network.sent‚ 2.273423e+07# disk.dm-0.out‚142581.2607421875 disk.sda1.in‚0§‰¢žÆ/‚(:“ ðÌéÒ³âæö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚287.28' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚57.455999999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ô ¥~8Ç/‚(:­ ðÌéÒÔ•“ù disk.sda.in‚ 106.21875 disk.dm-1.in‚ 106.21875 disk.drbd2165.in‚0 proc.cpu.threads‚17) disk./.usagePercent‚37.79355539126471$ disk./.usageGB‚563.1512870788574 disk.sda.out‚ 1266.4609375 disk.sda1.out‚0 disk.sda2.out‚0) proc.memory.availableMB‚ 2821.75390625 disk.rbd1.in‚ 0.52392578125 disk.dm-1.out‚ 1266.4609375 disk.drbd2165.out‚0 disk.rbd3.out‚ 0.22412109375 disk.nvme0c0n1.in‚ 46010.125' disk.nvme0n1.out‚142592.53271484375 disk.sda2.in‚0 network.recv‚ 4.866233e+06& disk.rbd0.in‚1.759218604441423e+13 disk.dm-0.in‚ 46010.125) disk.nvme0c0n1.out‚142592.53271484375 network.sent‚ 2.273936e+07$ disk.dm-0.out‚142592.53271484375 proc.memory.rssMB‚2804.875 disk.sda3.in‚ 106.21875 disk.nvme0n1.in‚ 46010.125 cpu‚25.589385027936615 disk.rbd1.out‚ 50.9560546875 disk.rbd0.out‚0 disk.sda1.in‚0 disk.rbd3.in‚0$ memory_percent‚82.77738094329834* proc.memory.percent‚17.119598388671875 disk.sda3.out‚ 1266.4609375@Ô!ñŸÈ/‚(:” ÿÌéÒøÛø÷ gpu.0.gpu‚92.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚41.0 gpu.0.powerWatts‚225.979' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚45.195800000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0í,L±BÉ/‚(:· ÿÌéÒ®öŒù network.recv‚ 4.869222e+06 disk.sda1.in‚0 disk.rbd3.out‚ 0.22412109375 disk.sda.out‚ 1266.734375 disk.rbd0.out‚0$ disk.dm-0.out‚142605.88427734375) disk./.usagePercent‚37.79400980463799 cpu‚25.589240141735853 disk.drbd2165.out‚0 disk.rbd1.in‚ 0.52392578125 disk.dm-1.in‚ 106.21875 disk.dm-1.out‚ 1266.734375 disk.sda1.out‚0 disk.rbd3.in‚0$ memory_percent‚82.77585506439209 disk.drbd2165.in‚0 disk.dm-0.in‚ 46112.6171875$ disk./.usageGB‚563.1580581665039 disk.sda3.in‚ 106.21875! disk.nvme0n1.in‚ 46112.6171875' disk.nvme0n1.out‚142605.88427734375 disk.sda2.out‚0 disk.rbd1.out‚ 51.0341796875 disk.sda.in‚ 106.21875 network.sent‚ 2.2744441e+07 disk.sda3.out‚ 1266.734375# disk.nvme0c0n1.in‚ 46112.6171875& disk.rbd0.in‚1.759218604441423e+13) proc.memory.availableMB‚ 2822.00390625 proc.memory.rssMB‚2804.875* proc.memory.percent‚17.119598388671875 proc.cpu.threads‚17) disk.nvme0c0n1.out‚142605.88427734375 disk.sda2.in‚0kÎÕ4Ê// _wandbruntime‚19566  _runtime‚19566³.ÁÉË/à ' lr-AdamW/pg1‚0.00025823259466777856 ' lr-AdamW/pg2‚0.00025823259466777856 trainer/global_step‚11799 $ _timestamp‚1.784309381888918e+09 _runtime‚19566.10609502 _step‚969o ¢UÎÌ/È trainer/global_step‚11799 $ _timestamp‚1.784309381888918e+09 _runtime‚19566.10609502 _step‚969 ' lr-AdamW/pg1‚0.00025823259466777856 ' lr-AdamW/pg2‚0.00025823259466777856ÉQÇåÎ4Í// _wandbruntime‚19566  _runtime‚19566qz }(Î/P =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚3401123128211113 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3402608215081984 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 100663339 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 100663339 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 202857608 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚33446195 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚915926985232384 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11099290517504 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3401124498239403 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14309790179840 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚3391507553001472 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11100635907072 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚33445942 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 trainer/global_step‚11799 _runtime‚19566.107316746 _step‚970 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚30270226 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚930236372784640 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚3390042855354368 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 % _timestamp‚1.7843093818899956e+09 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3402608215081984 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3402606843518976 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3391507553001472 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚258 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚30270221 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504AOIùû"Ï/”P C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 202857608 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚33446195 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚915926985232384 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11099290517504 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3401124498239403 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14309790179840 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚3391507553001472 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11100635907072 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚33445942 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 trainer/global_step‚11799 _runtime‚19566.107316746 _step‚970 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚30270226 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚930236372784640 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚3390042855354368 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 % _timestamp‚1.7843093818899956e+09 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3402608215081984 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3402606843518976 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3391507553001472 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚258 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚30270221 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚3401123128211113 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3402608215081984 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 100663339 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 100663339 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 BhŸin_batch_start/active_bytes.small_pool.freed‚11099290517504 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚63716163 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚3390042829926400 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11080298284713 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216Ê"Ç©€4Ð// _wandbruntime‚19567  _runtime‚19567ƒœ+'Ñ/™N I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11101560949760 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚930315454086656 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 102192768 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14310947917312 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚916004103548928 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _runtime‚19567.491878027 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 100671551 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 102192768 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 202874306 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 202864313 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3402887140854784 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚243 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11081221750959 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚3401401933455535 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚63721706 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3401403303483937 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚63721458 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14310982446592 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3390320737132544 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3390320711704576 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚248 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 102202755 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 202874306 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7843093832745383e+09 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 102202755 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3402885769284608 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 202864313 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 trainer/global_step‚11799 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11100215553024 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3391785579905024 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11082566351393 _step‚971 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 100671551 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11101560949760 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚916004471640064 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 100671545 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3402887140854784 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚3391785579905024 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3391785553731584 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚30272764 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216»’Î$'Ò/žN E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11101560949760 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚916004471640064 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 100671545 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3402887140854784 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚3391785579905024 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3391785553731584 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚30272764 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11101560949760 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚930315454086656 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 102192768 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14310947917312 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚916004103548928 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _runtime‚19567.491878027 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 100671551 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 102192768 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 202874306 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 202864313 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3402887140854784 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚243 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11081221750959 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚3401401933455535 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚63721706 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3401403303483937 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚63721458 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14310982446592 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3390320737132544 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3390320711704576 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚248 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 102202755 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 202874306 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7843093832745383e+09 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 102202755 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3402885769284608 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 202864313 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 trainer/global_step‚11799 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11100215553024 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3391785579905024 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11082566351393 _step‚971 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 100671551ËÖÕ›4Ó// _wandbruntime‚19567  _runtime‚19567È:øÔ/ 4loss/FBt1/assignment_loss‚3.3553276553528336e-06 1loss/SRqqt1/detection_loss‚0.6748190182237849 _step‚972 0loss/FRt2/assignment_loss‚0.2774385245913107 0loss/FRt1/detection_loss‚0.47028086745222275 /loss/FRt2/detection_loss‚1.2704689521530375 %loss/total_loss‚5.828655344795279 epoch‚29 0loss/FRt1/assignment_loss‚1.1002810103600564 3loss/SRqqt1/assignment_loss‚0.19735470272598796 3loss/FBt2/assignment_loss‚0.0007449837927479392 /loss/FBt1/detection_loss‚0.5950137050069102 trainer/global_step‚11799 3loss/SRqqt2/assignment_loss‚0.23131493287677163 1loss/SRqqt2/detection_loss‚0.5264409525849246 0loss/FBt2/detection_loss‚0.48449433969986877 % _timestamp‚1.7843093833612726e+09 _runtime‚19567.578201711β>Õ/’ 0loss/FRt1/assignment_loss‚1.1002810103600564 3loss/SRqqt1/assignment_loss‚0.19735470272598796 3loss/FBt2/assignment_loss‚0.0007449837927479392 /loss/FBt1/detection_loss‚0.5950137050069102 trainer/global_step‚11799 3loss/SRqqt2/assignment_loss‚0.23131493287677163 1loss/SRqqt2/detection_loss‚0.5264409525849246 0loss/FBt2/detection_loss‚0.48449433969986877 % _timestamp‚1.7843093833612726e+09 _runtime‚19567.578201711 4loss/FBt1/assignment_loss‚3.3553276553528336e-06 1loss/SRqqt1/detection_loss‚0.6748190182237849 _step‚972 0loss/FRt2/assignment_loss‚0.2774385245913107 0loss/FRt1/detection_loss‚0.47028086745222275 /loss/FRt2/detection_loss‚1.2704689521530375 %loss/total_loss‚5.828655344795279 epoch‚29Ì5ž„Ý”Ö/‚(:‰ ŽÍéÒŸ·ñö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚270.943' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.1886 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ºö§Õ6×/‚(:« ŽÍéÒã×Õù network.sent‚ 2.274902e+07 disk.dm-1.out‚ 1266.9375 disk.drbd2165.in‚0 disk.rbd3.out‚ 0.22412109375 disk.sda2.in‚0# disk.dm-0.out‚142616.0283203125 disk.drbd2165.out‚0# disk.nvme0c0n1.in‚ 46112.8203125( disk.nvme0c0n1.out‚142616.0283203125 disk.sda2.out‚0 disk.dm-0.in‚ 46112.8203125* proc.memory.percent‚17.119598388671875! disk.nvme0n1.in‚ 46112.8203125 disk.sda1.out‚0 cpu‚25.589080948717406 disk.sda3.out‚ 1266.9375 disk.rbd3.in‚0& disk.nvme0n1.out‚142616.0283203125 disk.rbd1.in‚ 0.52392578125& disk.rbd0.in‚1.759218604441423e+13 disk.rbd0.out‚0' proc.memory.availableMB‚ 2821.734375 proc.cpu.threads‚17 disk.sda3.in‚ 106.21875 disk.sda.in‚ 106.21875 disk.dm-1.in‚ 106.21875 proc.memory.rssMB‚2804.875 network.recv‚ 4.872772e+06 disk.rbd1.out‚ 51.0341796875 disk.sda1.in‚0$ memory_percent‚82.77750015258789) disk./.usagePercent‚37.79345990045444$ disk./.usageGB‚563.1498641967773 disk.sda.out‚ 1266.9375Ö»¹[“Ø/‚(:ˆ ÍéÒé®–÷ gpu.0.gpu‚12.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚340.072' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚68.0144 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0@NL#>Ù/‚(:³ ÍéÒç±ïù$ disk./.usageGB‚563.1594352722168& disk.rbd0.in‚1.759218604441423e+13 disk.sda1.out‚0 disk.sda3.out‚ 1267.3359375 disk.drbd2165.in‚0 disk.rbd3.out‚ 0.25927734375 network.recv‚ 4.877244e+06 disk.sda2.in‚0 disk.rbd1.out‚ 51.0966796875 proc.cpu.threads‚17 disk.sda2.out‚0 disk.sda.out‚ 1267.3359375 disk.rbd0.out‚0' disk.nvme0c0n1.out‚142623.814453125 disk.rbd3.in‚0 disk.dm-1.out‚ 1267.3359375" disk.dm-0.out‚142623.814453125 disk.sda1.in‚0 disk.sda3.in‚ 106.21875 proc.memory.rssMB‚2804.875 network.sent‚ 2.281294e+07* disk./.usagePercent‚37.794102223357854 disk.sda.in‚ 106.21875# disk.nvme0c0n1.in‚ 46113.2421875% disk.nvme0n1.out‚142623.814453125# memory_percent‚82.7775239944458* proc.memory.percent‚17.119598388671875 disk.rbd1.in‚ 0.52392578125 disk.dm-1.in‚ 106.21875) proc.memory.availableMB‚ 2821.73046875 cpu‚25.588923099686298 disk.dm-0.in‚ 46113.2421875 disk.drbd2165.out‚0! disk.nvme0n1.in‚ 46113.2421875÷|HÍ”Ú/‚(:‰ ¬ÍéÒèó÷ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚273.366' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.6732 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0™Â‹Î?Û/‚(:´ ¬ÍéÒн¬ù network.sent‚ 2.281805e+07 disk.sda1.out‚0* proc.memory.percent‚17.119598388671875 disk.rbd3.out‚ 0.34521484375( disk.nvme0c0n1.out‚142639.2177734375 proc.cpu.threads‚17* disk./.usagePercent‚37.794390487841405# disk.nvme0c0n1.in‚ 46113.4609375 disk.sda2.in‚0( proc.memory.availableMB‚ 2821.4453125 network.recv‚ 4.880785e+06 disk.sda3.out‚ 1267.4609375! disk.nvme0n1.in‚ 46113.4609375 disk.dm-1.out‚ 1267.4609375& disk.rbd0.in‚1.759218604441423e+13& disk.nvme0n1.out‚142639.2177734375$ disk./.usageGB‚563.1637306213379 disk.drbd2165.out‚0 disk.rbd3.in‚0 disk.sda2.out‚0 disk.rbd1.in‚ 0.52392578125 disk.sda.out‚ 1267.4609375 disk.dm-1.in‚ 106.21875# disk.dm-0.out‚142639.2177734375$ memory_percent‚82.77926445007324 proc.memory.rssMB‚2804.875 cpu‚25.5887546577062 disk.sda3.in‚ 106.21875 disk.drbd2165.in‚0 disk.sda.in‚ 106.21875 disk.rbd0.out‚0 disk.sda1.in‚0 disk.rbd1.out‚ 51.0966796875 disk.dm-0.in‚ 46113.4609375&²3m”Ü/‚(:‰ »ÍéÒŸ»êö gpu.0.gpu‚96.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚338.056' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚67.6112 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚09µ?Ý/‚(:´ »ÍéÒËé§ù disk.dm-1.out‚ 1267.71484375 disk.rbd3.in‚0* disk./.usagePercent‚37.795629308299596 disk.sda.out‚ 1267.71484375 disk.sda1.out‚0 disk.drbd2165.out‚0* proc.memory.percent‚17.119598388671875 network.recv‚ 4.884091e+06 disk.rbd1.out‚ 51.1748046875& disk.rbd0.in‚1.759218604441423e+13 disk.dm-0.in‚ 46270.0625$ disk.dm-0.out‚142649.26025390625 disk.sda1.in‚0 disk.sda3.in‚ 106.21875 proc.memory.rssMB‚2804.875 disk.sda.in‚ 106.21875 disk.dm-1.in‚ 106.21875 disk.rbd3.out‚ 0.34521484375 network.sent‚ 2.2823308e+07 disk.nvme0n1.in‚ 46270.0625' disk.nvme0n1.out‚142649.26025390625 disk.rbd1.in‚ 0.52392578125 disk.sda3.out‚ 1267.71484375 disk.nvme0c0n1.in‚ 46270.0625) disk.nvme0c0n1.out‚142649.26025390625$ disk./.usageGB‚563.1821899414062 disk.sda2.out‚0$ memory_percent‚82.78143405914307 proc.cpu.threads‚17 disk.sda2.in‚0 disk.rbd0.out‚0 disk.drbd2165.in‚0) proc.memory.availableMB‚ 2821.08984375 cpu‚25.58861058882248Œ¼æòŸÞ/‚(:” ÊÍéÒþÑ™÷ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚285.446' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚57.089200000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¥) –Aß/‚(:¶ ÊÍéÒŽðúù proc.memory.rssMB‚2804.875$ disk./.usageGB‚563.1911964416504 disk.sda1.out‚0) proc.memory.availableMB‚ 2821.08984375) disk./.usagePercent‚37.79623374208793& disk.rbd0.in‚1.759218604441423e+13% disk.nvme0n1.out‚142661.740234375* proc.memory.percent‚17.119598388671875 disk.dm-1.in‚ 106.21875" disk.nvme0n1.in‚46278.34765625 disk.sda.in‚ 106.21875 disk.sda2.in‚0 disk.sda3.in‚ 106.21875 disk.drbd2165.in‚0 disk.sda2.out‚0$ memory_percent‚82.78143405914307 disk.sda.out‚ 1267.88671875 disk.dm-0.in‚46278.34765625 disk.rbd1.in‚ 0.52392578125 cpu‚25.58845219853605 proc.cpu.threads‚17 disk.dm-1.out‚ 1267.88671875 disk.rbd0.out‚0 disk.sda1.in‚0 disk.drbd2165.out‚0 disk.rbd3.in‚0 disk.rbd3.out‚ 0.369140625$ disk.nvme0c0n1.in‚46278.34765625' disk.nvme0c0n1.out‚142661.740234375 network.sent‚ 2.282842e+07 network.recv‚ 4.887933e+06" disk.dm-0.out‚142661.740234375 disk.sda3.out‚ 1267.88671875 disk.rbd1.out‚ 51.1748046875:g44à// _wandbruntime‚19644  _runtime‚19644D©bÆËá/Å % _timestamp‚1.7843094605888646e+09 _runtime‚19644.806992168 _step‚973 ' lr-AdamW/pg1‚0.00025555366713334704 ' lr-AdamW/pg2‚0.00025555366713334704 trainer/global_step‚11849Î’ð Ðâ/Ê % _timestamp‚1.7843094605888646e+09 _runtime‚19644.806992168 _step‚973 ' lr-AdamW/pg1‚0.00025555366713334704 ' lr-AdamW/pg2‚0.00025555366713334704 trainer/global_step‚11849Íä(U±4ã// _wandbruntime‚19644  _runtime‚19644i7hhMä/P TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚30397376 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚63982610 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3405407615680512 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 203683681 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚3415063454233021 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11126535397821 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚934171130058240 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 101073639 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 102620015 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚33585234 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 101073645 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11146951837696 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚235 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚33584999 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚919801305616384 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 102610042 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3416553195955200 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚11849 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 102610042 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 _step‚974 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3416554567518208 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚3403936918835200 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 _runtime‚19644.807106754 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚3416553195955200 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚63982370 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 102620015 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 400530432 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008®pûÞ(å/”P ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚934171130058240 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 101073639 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 102620015 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚33585234 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 101073645 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11146951837696 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚235 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚33584999 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚919801305616384 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 102610042 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3416553195955200 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚11849 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 102610042 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 _step‚974 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3416554567518208 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚3403936918835200 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 _runtime‚19644.807106754 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚3416553195955200 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚63982370 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 102620015 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 400530432 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚30397376 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚63982610 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3405407615680512 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 203683681 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚3415063454233021 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11126535397821 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175Îèy¸ï4æ// _wandbruntime‚19646  _runtime‚19646N 0'ç/—N P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚3405685590237184 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚33588006 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11146531483648 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚30399914 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚919878792024064 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3404214826041344 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14371032060416 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 101081845 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 203700365 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3416833493291008 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3416832121720832 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3405685590237184 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 203710358 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚3415342259477443 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚934249824084480 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3404214800613376 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚233 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11147876880384 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚228 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 101081845 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚934249423561216 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 102628507 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚63987692 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚30399919 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 400523264 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3415343629505845 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 trainer/global_step‚11849 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11127458864067 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 101081851 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3416833493291008 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚32432128 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 102628507 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11147876880384 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 203710358 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _runtime‚19646.19042314 _step‚975 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 101081845 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 203700365 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3416833493291008 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 FDeviceStatsMonitor.on_train_batch_end/act³\ÜÈxive_bytes.all.current‚ 1371570176 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3416832121720832 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3405685590237184 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 203710358 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚3415342259477443 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚934249824084480 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3404214800613376 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚233 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11147876880384 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚228 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 101081845 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚934249423561216 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 102628507 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚63987692 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚30399919 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 400523264 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3415343629505845 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 trainer/global_step‚11849 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11127458864067 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 101081851 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3416833493291008 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚32432128 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 102628507 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11147876880384 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 203710358 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0Ï*Ñ4é// _wandbruntime‚19646  _runtime‚19646!ÕÁ%ê/Š %loss/total_loss‚5.539969689956886 0loss/FRt1/assignment_loss‚1.0989720705621828 3loss/SRqqt2/assignment_loss‚0.23312506695933916 3loss/FBt1/assignment_loss‚0.0023009125363360637 .loss/FRt2/detection_loss‚1.292648733540273 1loss/SRqqt1/detection_loss‚0.6578993822987593 % _timestamp‚1.7843094620587244e+09 _step‚976 3loss/SRqqt1/assignment_loss‚0.23906359072839242 3loss/FBt2/assignment_loss‚3.899046171704833e-05 1loss/SRqqt2/detection_loss‚0.4920825321451608 trainer/global_step‚11849 _runtime‚19646.275661536 1loss/FRt2/assignment_loss‚0.22976325288578703 /loss/FBt1/detection_loss‚0.5029740695666988 epoch‚29 /loss/FRt1/detection_loss‚0.4876151092975459 /loss/FBt2/detection_loss‚0.3034859789746938=Öë/ 3loss/SRqqt1/assignment_loss‚0.23906359072839242 3loss/FBt2/assignment_loss‚3.899046171704833e-05 1loss/SRqqt2/detection_loss‚0.4920825321451608 trainer/global_step‚11849 _runtime‚19646.275661536 1loss/FRt2/assignment_loss‚0.22976325288578703 /loss/FBt1/detection_loss‚0.5029740695666988 epoch‚29 /loss/FRt1/detection_loss‚0.4876151092975459 /loss/FBt2/detection_loss‚0.3034859789746938 %loss/total_loss‚5.539969689956886 0loss/FRt1/assignment_loss‚1.0989720705621828 3loss/SRqqt2/assignment_loss‚0.23312506695933916 3loss/FBt1/assignment_loss‚0.0023009125363360637 .loss/FRt2/detection_loss‚1.292648733540273 1loss/SRqqt1/detection_loss‚0.6578993822987593 % _timestamp‚1.7843094620587244e+09 _step‚976ÐkÛ _“ì/‚(:ˆ ÙÍéÒîì™÷ gpu.0.gpu‚100.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚329.76' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚65.952 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0nà”Cí/‚(:¸ ÙÍéÒáÄ ù& disk.rbd0.in‚1.759218604441423e+13 disk.sda3.out‚ 1268.20703125" disk.nvme0n1.in‚46278.56640625% disk.nvme0n1.out‚142675.322265625 disk.rbd0.out‚0 disk.drbd2165.in‚0 proc.cpu.threads‚17$ disk./.usageGB‚563.1956558227539 disk.sda.in‚ 106.21875 disk.dm-0.in‚46278.56640625 disk.sda1.out‚0 disk.rbd3.in‚0 disk.dm-1.in‚ 106.21875 network.recv‚ 4.890998e+06' disk.nvme0c0n1.out‚142675.322265625* proc.memory.percent‚17.119598388671875 cpu‚25.588310430425338 disk.sda2.in‚0 disk.sda1.in‚0$ memory_percent‚82.78210163116455) proc.memory.availableMB‚ 2820.98046875 network.sent‚ 2.2833502e+07 disk.sda.out‚ 1268.20703125 disk.drbd2165.out‚0 proc.memory.rssMB‚2804.875) disk./.usagePercent‚37.79653301489546" disk.dm-0.out‚142675.322265625 disk.sda3.in‚ 106.21875$ disk.nvme0c0n1.in‚46278.56640625 disk.dm-1.out‚ 1268.20703125 disk.sda2.out‚0 disk.rbd1.in‚ 0.52392578125 disk.rbd1.out‚ 51.2998046875 disk.rbd3.out‚ 0.369140625“é-&’î/‚(:‡ èÍéÒëí„÷ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚276.31' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.262 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ù¶/@ï/‚(:µ èÍéÒùÏÒù& disk.rbd0.in‚1.759218604441423e+13$ memory_percent‚82.78176784515381 disk.rbd1.in‚ 0.52392578125 disk.rbd0.out‚0 disk.sda1.out‚0( disk.nvme0c0n1.out‚142685.4189453125 disk.sda2.in‚0 disk.sda.in‚ 106.21875# disk.dm-0.out‚142685.4189453125 disk.sda3.out‚ 1268.3359375 disk.drbd2165.out‚0) disk./.usagePercent‚37.79698896431395# disk.nvme0c0n1.in‚ 46278.8828125 disk.sda3.in‚ 106.21875 disk.drbd2165.in‚0 disk.rbd3.in‚0! disk.nvme0n1.in‚ 46278.8828125& disk.nvme0n1.out‚142685.4189453125 disk.sda.out‚ 1268.3359375 disk.dm-1.out‚ 1268.3359375 disk.dm-0.in‚ 46278.8828125 network.recv‚ 4.895747e+06 disk.rbd3.out‚ 0.43212890625 disk.sda2.out‚0 disk.dm-1.in‚ 106.21875) proc.memory.availableMB‚ 2821.03515625 disk.rbd1.out‚ 51.2998046875 proc.memory.rssMB‚2804.875* proc.memory.percent‚17.119598388671875 cpu‚25.588127817943736 proc.cpu.threads‚17 network.sent‚ 2.289691e+07# disk./.usageGB‚563.202449798584 disk.sda1.in‚0ß{Džð/‚(:“ ÷Íé҃͂÷ gpu.0.gpu‚90.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚269.905' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚53.98099999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÀU"Dñ/‚(:¹ ÷ÍéÒÎàäù network.recv‚ 4.899079e+06& disk.rbd0.in‚1.759218604441423e+13 disk.drbd2165.out‚0! disk.nvme0n1.in‚ 46279.3046875$ memory_percent‚82.78200626373291 disk.sda3.out‚ 1268.58203125 disk.dm-1.out‚ 1268.58203125 disk.drbd2165.in‚0( disk.nvme0c0n1.out‚142693.2587890625& disk.nvme0n1.out‚142693.2587890625# disk.nvme0c0n1.in‚ 46279.3046875) proc.memory.availableMB‚ 2820.99609375 proc.memory.rssMB‚2804.875 disk.sda3.in‚ 106.21875 disk.rbd3.in‚0 disk.rbd3.out‚ 0.43212890625 disk.sda.in‚ 106.21875$ disk./.usageGB‚563.2094497680664 disk.dm-1.in‚ 106.21875 disk.rbd0.out‚0 disk.sda1.in‚0 disk.rbd1.in‚ 0.52392578125) disk./.usagePercent‚37.79745873813927 disk.dm-0.in‚ 46279.3046875# disk.dm-0.out‚142693.2587890625 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd1.out‚ 51.3779296875* proc.memory.percent‚17.119598388671875 cpu‚25.58798418511747 network.sent‚ 2.2902083e+07 disk.sda1.out‚0 disk.sda.out‚ 1268.58203125 proc.cpu.threads‚17`äÝÔ”ò/‚(:‰ †ÎéÒ‘èö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚296.693' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚59.3386 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚05›Â@ó/‚(:µ †ÎéÒðç™ù network.recv‚ 4.902495e+06$ disk./.usageGB‚563.2140884399414 disk.drbd2165.out‚0) disk./.usagePercent‚37.79777004330091 disk.rbd1.in‚ 0.52392578125 disk.sda3.in‚ 106.21875 disk.rbd3.in‚0( proc.memory.availableMB‚ 2820.9921875 disk.sda.in‚ 106.21875 disk.dm-1.in‚ 106.21875 disk.drbd2165.in‚0' disk.nvme0n1.out‚142707.29833984375* proc.memory.percent‚17.119598388671875 cpu‚25.587817424093508 network.sent‚ 2.2907208e+07 disk.sda2.out‚0$ disk.dm-0.out‚142707.29833984375) disk.nvme0c0n1.out‚142707.29833984375 proc.memory.rssMB‚2804.875 disk.sda.out‚ 1268.72265625 disk.dm-0.in‚ 46739.28125 disk.sda1.in‚0 disk.rbd3.out‚ 0.4951171875 proc.cpu.threads‚17 disk.rbd1.out‚ 51.3779296875& disk.rbd0.in‚1.759218604441423e+13 disk.sda1.out‚0 disk.nvme0n1.in‚ 46739.28125$ memory_percent‚82.78203010559082 disk.sda2.in‚0 disk.dm-1.out‚ 1268.72265625 disk.rbd0.out‚0 disk.sda3.out‚ 1268.72265625! disk.nvme0c0n1.in‚ 46739.28125ùöe”ô/‚(:‰ •ÎéÒã”Òö gpu.0.gpu‚90.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚270.099' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.0198 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0þ ,<õ/‚(:± •ÎéÒ¤ôÌø disk.rbd0.out‚0$ disk.dm-0.out‚142714.91650390625" disk.nvme0c0n1.in‚ 46739.484375) disk./.usagePercent‚37.79821908051596 disk.dm-1.out‚ 1269.140625 disk.sda1.in‚0 disk.nvme0n1.in‚ 46739.484375* proc.memory.percent‚17.119598388671875 disk.dm-0.in‚ 46739.484375 disk.rbd3.in‚0' disk.nvme0n1.out‚142714.91650390625 proc.cpu.threads‚17 disk.sda.in‚ 106.21875 disk.dm-1.in‚ 106.21875 disk.sda2.out‚0 proc.memory.rssMB‚2804.875 network.sent‚ 2.291228e+07 network.recv‚ 4.905711e+06 disk.sda3.out‚ 1269.140625 disk.sda2.in‚0 disk.rbd1.out‚ 51.4404296875 disk.rbd1.in‚ 0.52392578125 disk.sda.out‚ 1269.140625 disk.sda1.out‚0 disk.sda3.in‚ 106.21875 disk.rbd3.out‚ 0.4951171875$ disk./.usageGB‚563.2207794189453 disk.drbd2165.in‚0 disk.drbd2165.out‚0) disk.nvme0c0n1.out‚142714.91650390625& disk.rbd0.in‚1.759218604441423e+13$ memory_percent‚82.78355598449707( proc.memory.availableMB‚ 2820.7421875 cpu‚25.587663611244423\v¹4ö// _wandbruntime‚19723  _runtime‚19723@;mË÷/Å ' lr-AdamW/pg1‚0.00025288054393103675 ' lr-AdamW/pg2‚0.00025288054393103675 trainer/global_step‚11899 % _timestamp‚1.7843095393325865e+09 _runtime‚19723.549914024 _step‚977TkÐø/Ê ' lr-AdamW/pg2‚0.00025288054393103675 trainer/global_step‚11899 % _timestamp‚1.7843095393325865e+09 _runtime‚19723.549914024 _step‚977 ' lr-AdamW/pg1‚0.00025288054393103675Ñ»%vE4ù// _wandbruntime‚19723  _runtime‚19723,Ž8æ(ú/P LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 103035794 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 103035794 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 % _timestamp‚1.7843095393338716e+09 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 101483939 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚3429003780254929 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11191922378752 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚30524521 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3419307626012672 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3430500919954432 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11191922378752 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚3417831033171968 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11174117111251 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚254 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚3417831007744000 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚30524526 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 204519733 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14429755065856 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14429787505152 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11193267768320 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚32439296 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3430499548391424 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚3419307626012672 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 103045767 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚33723708 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚64248234 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648n]û/”P JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚254 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚3417831007744000 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚30524526 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 204519733 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14429755065856 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14429787505152 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11193267768320 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚32439296 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3430499548391424 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚3419307626012672 MFDeviceStatsMonitor.on_train_`†=½batch_start/inactive_split.large_pool.peak‚51 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 103045767 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚33723708 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚64248234 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 103035794 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 103035794 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 % _timestamp‚1.7843095393338716e+09 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 101483939 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚3429003780254929 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11191922378752 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚30524521 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3419307626012672 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3430500919954432 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11191922378752 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚3417831033171968 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11174117111251 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154ÒÒæ4ü// _wandbruntime‚19724  _runtime‚19724º÷.<'ý/™N XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚938183695810048 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3418108889522176 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3418108914950144 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3419585626742784 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚3419585652916224 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚923752744316928 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3429283955527753 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 101492145 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 204546410 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14430986022400 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 103044272 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚64253281 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 204546410 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 204536417 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚64253524 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 103054259 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11173695977175 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 103044272 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚3419585626742784 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11194192811008 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 101492151 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 204536417 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚33726455 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚33726217 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3430778474157056 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3419585652916224 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 % _timestamp‚1.7843095407230232e+09 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 103054259 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚923753112408064 _step‚979 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 101492145 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚238 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _runtime‚19724.940379441 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 101492151 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3430779845727232 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11175040577609 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11192847414272 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11194192811008 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416y½¬d$'þ/žN ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11194192811008 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 101492151 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 204536417 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚33726455 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚33726217 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3430778474157056 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3419585652916224 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 % _timestamp‚1.7843095407230232e+09 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 103054259 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚923753112408064 _step‚979 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 101492145 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚238 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _runtime‚19724.940379441 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 101492151 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3430779845727232 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11175040577609 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11192847414272 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11194192811008 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚938183695810048 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3418108889522176 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3418108914950144 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3419585626742784 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚3419585652916224 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚923752744316928 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3429283955527753 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 101492145 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 204546410 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14430986022400 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 103044272 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚64253281 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 204546410 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 204536417 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚64253524 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 103054259 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11173695977175 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 103044272 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚3419585626742784 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0Ó’\¡jŸÿ/‚(:” ¤ÎéÒÔ¶Åù gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚287.815' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚57.562999999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0öô¸m7€0‚(:¬ ¤ÎéÒâÊËú* proc.memory.percent‚17.119598388671875 disk.dm-0.in‚ 46739.90625 disk.sda3.in‚ 106.21875# memory_percent‚82.7770471572876 disk.sda.in‚ 106.21875 disk.dm-1.out‚ 1269.4609375 disk.sda1.in‚0 disk.sda3.out‚ 1269.4609375' disk.nvme0c0n1.out‚142729.205078125 cpu‚25.587503024882043 disk.rbd3.out‚ 0.4951171875 disk.sda2.out‚0) disk./.usagePercent‚37.79871112699678$ disk./.usageGB‚563.2281112670898 disk.sda.out‚ 1269.4609375 disk.rbd0.out‚0 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.rbd3.in‚0) proc.memory.availableMB‚ 2821.80859375 network.sent‚ 2.2917394e+07 network.recv‚ 4.909026e+06 disk.rbd1.out‚ 51.4404296875! disk.nvme0c0n1.in‚ 46739.90625 proc.memory.rssMB‚2804.875 proc.cpu.threads‚17& disk.rbd0.in‚1.759218604441423e+13" disk.dm-0.out‚142729.205078125 disk.nvme0n1.in‚ 46739.90625 disk.dm-1.in‚ 106.21875% disk.nvme0n1.out‚142729.205078125 disk.sda2.in‚0 disk.rbd1.in‚ 0.52392578125 disk.drbd2165.out‚0r*Q*40/ _wandbruntime‚19725  _runtime‚19725Xcf' ‚0‡ 2loss/SRqqt2/detection_loss‚0.49167864913456216 /loss/FBt1/detection_loss‚0.4424517857320704 %loss/total_loss‚5.766333797348723 epoch‚29 /loss/FRt1/assignment_loss‚1.134037299392734 0loss/FRt2/assignment_loss‚0.2445933429475635 4loss/FBt2/assignment_loss‚4.2900235241419856e-05 trainer/global_step‚11899 _step‚980 3loss/SRqqt1/assignment_loss‚0.23668858964380363 0loss/SRqqt2/assignment_loss‚0.26633964369597 .loss/FRt2/detection_loss‚1.255066893800414 1loss/SRqqt1/detection_loss‚0.7123538363209527 % _timestamp‚1.7843095408079896e+09 _runtime‚19725.024920832 3loss/FBt1/assignment_loss‚0.0024037567416724435 0loss/FRt1/detection_loss‚0.49509230528709613 .loss/FBt2/detection_loss‚0.485584794416643Cu»ƒ0Œ 2loss/SRqqt2/detection_loss‚0.49167864913456216 /loss/FBt1/detection_loss‚0.4424517857320704 %loss/total_loss‚5.766333797348723 epoch‚29 /loss/FRt1/assignment_loss‚1.134037299392734 0loss/FRt2/assignment_loss‚0.2445933429475635 4loss/FBt2/assignment_loss‚4.2900235241419856e-05 trainer/global_step‚11899 _step‚980 3loss/SRqqt1/assignment_loss‚0.23668858964380363 0loss/SRqqt2/assignment_loss‚0.26633964369597 .loss/FRt2/detection_loss‚1.255066893800414 1loss/SRqqt1/detection_loss‚0.7123538363209527 % _timestamp‚1.7843095408079896e+09 _runtime‚19725.024920832 3loss/FBt1/assignment_loss‚0.0024037567416724435 0loss/FRt1/detection_loss‚0.49509230528709613 .loss/FBt2/detection_loss‚0.485584794416643Ô¼CÀž„0‚(:“ ³ÎéÒ½ëø gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚272.83' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚54.565999999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ú_,…0‚(:¡ ³ÎéÒÍ–ýú& disk.rbd0.in‚1.759218604441423e+13 disk.sda1.in‚0 disk.sda2.in‚0 disk.dm-0.in‚ 46740.125 disk.sda3.in‚ 106.21875 disk.drbd2165.in‚0 network.sent‚ 2.297936e+07$ disk./.usageGB‚563.2323150634766 disk.dm-1.in‚ 106.21875 disk.rbd3.in‚0' disk.nvme0c0n1.out‚142737.564453125 cpu‚25.58735968945507 disk.dm-1.out‚ 1269.859375" disk.dm-0.out‚142737.564453125$ memory_percent‚82.77711868286133* proc.memory.percent‚17.119598388671875 disk.sda.in‚ 106.21875 disk.sda1.out‚0 disk.nvme0c0n1.in‚ 46740.125 disk.rbd1.out‚ 51.5185546875' proc.memory.availableMB‚ 2821.796875 network.recv‚ 4.913064e+06* disk./.usagePercent‚37.798993247299514 disk.nvme0n1.in‚ 46740.125% disk.nvme0n1.out‚142737.564453125 disk.rbd1.in‚ 0.52392578125 proc.memory.rssMB‚2804.875 disk.sda.out‚ 1269.859375 disk.rbd0.out‚0 disk.drbd2165.out‚0 disk.rbd3.out‚ 0.4951171875 disk.sda2.out‚0 disk.sda3.out‚ 1269.859375 proc.cpu.threads‚17Çp¶Àž†0‚(:“ ÂÎéÒ¯éòö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚332.801' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚66.56020000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ët‚ìB‡0‚(:· ÂÎéÒŽ¯ù disk.drbd2165.in‚0 disk.drbd2165.out‚0' disk.nvme0n1.out‚142756.69482421875 disk.sda2.out‚0) disk./.usagePercent‚37.69802438781021 disk.sda1.in‚0 disk.rbd3.out‚ 0.4951171875) disk.nvme0c0n1.out‚142756.69482421875" disk.nvme0n1.in‚46829.98828125 disk.sda.in‚ 106.21875' proc.memory.availableMB‚ 2822.046875* proc.memory.percent‚17.119598388671875 network.recv‚ 4.917285e+06& disk.rbd0.in‚1.759218604441423e+13 disk.sda2.in‚0 proc.memory.rssMB‚2804.875 disk.rbd1.out‚ 51.5185546875 disk.dm-0.in‚46829.98828125 disk.sda3.in‚ 106.21875 disk.rbd3.in‚0 disk.dm-1.in‚ 106.21875 disk.dm-1.out‚ 1272.640625 proc.cpu.threads‚17$ disk./.usageGB‚561.7278060913086$ disk.dm-0.out‚142756.69482421875 disk.sda3.out‚ 1272.640625$ disk.nvme0c0n1.in‚46829.98828125 disk.rbd1.in‚ 0.52392578125 disk.rbd0.out‚0 cpu‚25.587221916942905 network.sent‚ 2.3021937e+07 disk.sda1.out‚0 disk.sda.out‚ 1272.640625$ memory_percent‚82.77559280395508un}”ˆ0‚(:‰ ÑÎéÒÛ¸ûö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚270.152' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.0304 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÑdMD‰0‚(:¹ ÑÎéÒÿ™†ú$ disk./.usageGB‚561.7337417602539 disk.rbd1.in‚ 0.52392578125 proc.cpu.threads‚17 network.sent‚ 2.3027543e+07! disk.nvme0n1.in‚ 46830.3984375# disk.dm-0.out‚142802.5146484375 disk.sda3.in‚ 106.21875 disk.rbd3.in‚0 proc.memory.rssMB‚2804.875 disk.sda.in‚ 106.21875 disk.sda3.out‚ 1273.43359375 disk.drbd2165.out‚0# disk.nvme0c0n1.in‚ 46830.3984375 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda.out‚ 1273.43359375 disk.dm-1.out‚ 1273.43359375( disk.nvme0c0n1.out‚142802.5146484375) disk./.usagePercent‚37.69842273553349 disk.rbd1.out‚ 51.5810546875 disk.dm-1.in‚ 106.21875 disk.sda1.in‚0& disk.rbd0.in‚1.759218604441423e+13 disk.dm-0.in‚ 46830.3984375 disk.sda1.out‚0 disk.rbd3.out‚ 0.4951171875 cpu‚25.587052929705546& disk.nvme0n1.out‚142802.5146484375 disk.drbd2165.in‚0) proc.memory.availableMB‚ 2822.29296875* proc.memory.percent‚17.119598388671875 network.recv‚ 4.920526e+06 disk.rbd0.out‚0$ memory_percent‚82.77409076690674+¾Ã䔊0‚(:‰ àÎéÒ™‰ëö gpu.0.gpu‚92.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚334.874' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚66.9748 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0wt¾'‹0‚(:¶ àÎéÒ£¤§ù network.sent‚ 2.3032658e+07 network.recv‚ 4.924069e+06 disk.sda3.in‚ 106.21875! disk.nvme0n1.in‚ 46830.6171875 disk.rbd1.out‚ 51.5810546875$ memory_percent‚82.77406692504883 disk.drbd2165.in‚0( disk.nvme0c0n1.out‚142837.1611328125 disk.sda.in‚ 106.21875' proc.memory.availableMB‚ 2822.296875 disk.sda1.in‚0 disk.rbd0.out‚0 proc.memory.rssMB‚2804.875# disk.dm-0.out‚142837.1611328125# disk.nvme0c0n1.in‚ 46830.6171875$ disk./.usageGB‚561.7734603881836 disk.sda3.out‚ 1273.91015625* proc.memory.percent‚17.119598388671875 proc.cpu.threads‚17) disk./.usagePercent‚37.70108828597999 disk.sda1.out‚0 disk.drbd2165.out‚0& disk.nvme0n1.out‚142837.1611328125 cpu‚25.58691’Ík_+331736998 disk.rbd3.in‚0 disk.rbd3.out‚ 0.4951171875 disk.sda2.out‚0 disk.rbd1.in‚ 0.52392578125 disk.dm-1.in‚ 106.21875 disk.dm-1.out‚ 1273.91015625& disk.rbd0.in‚1.759218604441423e+13 disk.dm-0.in‚ 46830.6171875 disk.sda2.in‚0 disk.sda.out‚ 1273.91015625%ÉùñŸŒ0‚(:” ïÎéÒ€Œ·ù gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚293.084' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚58.616800000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ŸvY‡A0‚(:¶ ïÎéÒ¥¶û proc.cpu.threads‚17 disk.dm-0.in‚ 46833.9765625 disk.sda1.out‚0 disk.sda3.out‚ 1274.2578125) disk./.usagePercent‚37.72144421305004# disk.nvme0c0n1.in‚ 46833.9765625 network.sent‚ 2.3037833e+07! disk.nvme0n1.in‚ 46833.9765625 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda1.in‚0$ memory_percent‚82.77614116668701* proc.memory.percent‚17.119598388671875 network.recv‚ 4.927521e+06 disk.dm-1.in‚ 106.21875 disk.rbd1.in‚ 0.52392578125$ disk./.usageGB‚562.0767784118652 disk.sda.out‚ 1274.2578125# disk.dm-0.out‚142983.3662109375 disk.drbd2165.out‚0 disk.rbd3.in‚0 disk.rbd3.out‚ 0.4951171875 proc.memory.rssMB‚2804.875 disk.sda.in‚ 106.21875 disk.dm-1.out‚ 1274.2578125 disk.sda3.in‚ 106.21875 disk.rbd1.out‚ 51.7216796875) proc.memory.availableMB‚ 2821.95703125& disk.rbd0.in‚1.759218604441423e+13 disk.rbd0.out‚0 disk.drbd2165.in‚0( disk.nvme0c0n1.out‚142983.3662109375& disk.nvme0n1.out‚142983.3662109375 cpu‚25.586728323857425Ù]@4Ž0/ _wandbruntime‚19802  _runtime‚19802"äæ~Ë0Å ' lr-AdamW/pg1‚0.00025021339675055575 ' lr-AdamW/pg2‚0.00025021339675055575 trainer/global_step‚11949 % _timestamp‚1.7843096178100562e+09 _runtime‚19802.027248476 _step‚981ânVùÐ0Ê ' lr-AdamW/pg2‚0.00025021339675055575 trainer/global_step‚11949 % _timestamp‚1.7843096178100562e+09 _runtime‚19802.027248476 _step‚981 ' lr-AdamW/pg1‚0.00025021339675055575ÕÅ©4‘0/ _wandbruntime‚19802  _runtime‚198026+l(’0ŽP ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚32439296 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 205365764 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚236 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3442945476305127 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11239583698944 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3444447272390656 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚64513804 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 103471519 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚942039836996096 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 $ _timestamp‚1.784309617811162e+09 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 400530432 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3444445900827648 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3433207662518272 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 103471519 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 103461546 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚3431725122080768 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 205355785 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14489858172416 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚64513563 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 101894239 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚927549946384384 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290;‚½á(“0“P LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚32439296 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 205365764 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚236 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3442945476305127 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11239583698944 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3444447272390656 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚64513804 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 103471519 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚942039836996096 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 $ _timestamp‚1.784309617811162e+09 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 400530432 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3444445900827648 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3433207662518272 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 103471519 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 103461546 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚3431725122080768 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 205355785 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14489858172416 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚64513563 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 101894239 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚927549946384384 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640Öæ4”0/ _wandbruntime‚19803  _runtime‚19803.Fp+D&•0™N H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 205382462 _runtime‚19803.413275839 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11240508741632 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚983 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11239163344896 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 103480011 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 103480011 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11221277690717 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚942118116376064 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚64519080 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚228 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚33864633 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚942118516899328 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 101902445 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11240508741632 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3432003003858944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚233 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚927627432792064 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚3433485663248384 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3433485689421824 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 101902451 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚64518847 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3432002978430976 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3433485663248384 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 101902445 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚3433485689421824 WHDeviceStatsMonj*µ–Ûitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 103470024 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0É{@¬$'–0žN A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚64519080 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚228 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚33864633 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚942118516899328 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 101902445 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11240508741632 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3432003003858944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚233 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚927627432792064 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚3433485663248384 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3433485689421824 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 101902451 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚64518847 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3432002978430976 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3433485663248384 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 101902445 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚3433485689421824 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 103470024 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 205382462 _runtime‚19803.413275839 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11240508741632 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚983 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11239163344896 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 103480011 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 103480011 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11221277690717 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚942118116376064 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352×dŸÔ4—0/ _wandbruntime‚19803  _runtime‚19803ë¶ ˜0‡ _step‚984 4loss/FBt2/assignment_loss‚1.8230259413931255e-06 /loss/FRt1/detection_loss‚0.4914412203658917 /loss/FRt1/assignment_loss‚1.223512044566149 /loss/FRt2/detection_loss‚1.3362960232479215 1loss/SRqqt1/detection_loss‚0.7731000071852192 % _timestamp‚1.7843096192825124e+09 epoch‚29 0loss/FRt2/assignment_loss‚0.3457929534899851 2loss/SRqqt1/assignment_loss‚0.1863288337355686 2loss/FBt1/assignment_loss‚0.012809122474261659 1loss/SRqqt2/detection_loss‚0.5469197650578035 /loss/FBt1/detection_loss‚0.3371349438449316 %loss/total_loss‚5.935999075321709 trainer/global_step‚11949 2loss/SRqqt2/assignment_loss‚0.2682556032732289 /loss/FBt2/detection_loss‚0.4144067350548069 _runtime‚19803.499522459^É5Õ™0Œ 1loss/SRqqt1/detection_loss‚0.7731000071852192 % _timestamp‚1.7843096192825124e+09 epoch‚29 0loss/FRt2/assignment_loss‚0.3457929534899851 2loss/SRqqt1/assignment_loss‚0.1863288337355686 2loss/FBt1/assignment_loss‚0.012809122474261659 1loss/SRqqt2/detection_loss‚0.5469197650578035 /loss/FBt1/detection_loss‚0.3371349438449316 %loss/total_loss‚5.935999075321709 trainer/global_step‚11949 2loss/SRqqt2/assignment_loss‚0.2682556032732289 /loss/FBt2/detection_loss‚0.4144067350548069 _runtime‚19803.499522459 _step‚984 4loss/FBt2/assignment_loss‚1.8230259413931255e-06 /loss/FRt1/detection_loss‚0.4914412203658917 /loss/FRt1/assignment_loss‚1.223512044566149 /loss/FRt2/detection_loss‚1.3362960232479215ØF¥û’š0‚(:‡ þÎéÒèË€÷ gpu.0.gpu‚1.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚332.412' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚66.4824 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0öµ…@›0‚(:µ þÎéÒ³³ú disk.sda.in‚ 106.21875& disk.rbd0.in‚1.759218604441423e+13 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd3.out‚ 0.4951171875 proc.memory.rssMB‚2804.875 disk.sda3.in‚ 106.21875 disk.sda3.out‚ 1274.8828125# memory_percent‚82.7775239944458) proc.memory.availableMB‚ 2821.73046875$ disk./.usageGB‚564.0924949645996* proc.memory.percent‚17.119598388671875( disk.nvme0c0n1.out‚143103.0849609375! disk.nvme0n1.in‚ 46956.0703125 disk.sda2.out‚0 disk.rbd1.out‚ 51.7216796875 disk.drbd2165.out‚0 disk.sda.out‚ 1274.8828125 disk.drbd2165.in‚0# disk.nvme0c0n1.in‚ 46956.0703125 network.recv‚ 4.931186e+06 disk.dm-1.out‚ 1274.8828125 disk.dm-0.in‚ 46956.0703125# disk.dm-0.out‚143103.0849609375& disk.nvme0n1.out‚143103.0849609375 network.sent‚ 2.3042417e+07) disk./.usagePercent‚37.85672064220289 disk.sda2.in‚0 disk.dm-1.in‚ 106.21875 disk.rbd0.out‚0 disk.rbd3.in‚0 cpu‚25.586590699861024 proc.cpu.threads‚17 disk.rbd1.in‚ 0.52392578125–±Sü”œ0‚(:‰ ÏéÒ›Õ‹÷ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚267.733' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.5466 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0À×0q>0‚(:³ ÏéÒîùÃù disk.dm-1.out‚ 1275.4609375 disk.sda1.out‚0$ memory_percent‚82.78157711029053* proc.memory.percent‚17.119598388671875& disk.rbd0.in‚1.759218604441423e+13 disk.dm-0.in‚ 46956.2890625" disk.dm-0.out‚144976.064453125 disk.rbd3.in‚0# disk.nvme0c0n1.in‚ 46956.2890625$ disk./.usageGB‚566.0356063842773 disk.rbd3.out‚ 0.4951171875' disk.nvme0c0n1.out‚144976.064453125 disk.sda2.in‚0 disk.rbd1.out‚ 51.7841796875) proc.memory.availableMB‚ 2821.06640625 disk.rbd1.in‚ 0.52392578125 disk.sda.in‚ 106.21875 cpu‚25.586413475243983* disk./.usagePercent‚37.987124479956535 disk.rbd0.out‚0% disk.nvme0n1.out‚144976.064453125 proc.cpu.threads‚17 disk.sda2.out‚0 disk.sda.out‚ 1275.4609375 network.sent‚ 2.310572e+07 disk.sda3.in‚ 106.21875 disk.drbd2165.out‚0! disk.nvme0n1.in‚ 46956.2890625 proc.memory.rssMB‚2804.875 network.recv‚ 4.935445e+06 disk.sda1.in‚0 disk.sda3.out‚ 1275.4609375 disk.drbd2165.in‚0 disk.dm-1.in‚ 106.21875‡Y}Æžž0‚(:“ œÏéÒ®ŠÝö gpu.0.gpu‚69.0 gpu.0.memory‚22, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚328.257' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚65.65140000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ö<ßÞ=Ÿ0‚(:² œÏéÒì’Âú& disk.nvme0n1.out‚145010.4052734375 disk.sda2.in‚0) disk./.usagePercent‚37.90478759280171 disk.dm-0.in‚ 46956.765625 disk.rbd1.in‚ 0.52392578125 proc.memory.rssMB‚2804.875 disk.rbd0.out‚0 disk.sda1.out‚0 disk.dm-1.out‚ 1275.7265625" disk.nvme0c0n1.in‚ 46956.765625( disk.nvme0c0n1.out‚145010.4052734375$ disk./.usageGB‚564.8087272644043 disk.dm-1.in‚ 106.21875$ memory_percent‚82.78324604034424 cpu‚25.58628269703661 disk.sda.in‚ 106.21875# disk.dm-0.out‚145010.4052734375 disk.sda3.out‚ 1275.7265625 disk.sda.out‚ 1275.7265625 disk.sda1.in‚0 disk.sda3.in‚ 106.21875 disk.drbd2165.out‚0 disk.rbd3.in‚0 disk.sda2.out‚0 disk.rbd1.out‚ 51.7841796875 network.sent‚ 2.3111366e+07 network.recv‚ 4.938743e+06 disk.drbd2165.in‚0 disk.nvme0n1.in‚ 46956.765625) proc.memory.availableMB‚ 2820.79296875* proc.memory.percent‚17.119598388671875 proc.cpu.threads‚17& disk.rbd0.in‚1.759218604441423e+13 disk.rbd3.out‚ 0.4951171875Ä6Už 0‚(:“ «ÏéÒ¬àÒö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚275.782' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚55.15639999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0p*O9¡0‚(:® «ÏéÒ¹æçø disk.nvme0n1.in‚ 46963.046875 disk.sda2.out‚0 disk.rbd1.out‚ 51.8623046875 disk.dm-1.in‚ 106.21875 disk.rbd0.out‚0 disk.dm-0.in‚ 46963.046875 network.sent‚ 2.3115906e+07 disk.rbd3.out‚ 0.4951171875 disk.sda2.in‚0 disk.rbd1.in‚ 0.52392578125 disk.sda.out‚ 1275.9765625" disk.dm-0.out‚146137.201171875' disk.nvme0c0n1.out‚146137.201171875( proc.memory.availableMB‚ 2821.0078125* proc.memory.percent‚17.119598388671875 cpu‚25.586121751661068 disk.drbd2165.in‚0" disk.nvme0c0n1.in‚ 46963.046875 disk.dm-1.out‚ 1275.9765625 proc.memory.rssMB‚2804.875 network.recv‚ 4.942184e+06& disk.rbd0.in‚1.759218604441423e+13$ memory_percent‚82.78193473815918 proc.cpu.threads‚17 disk.sda3.in‚ 106.21875 disk.rbd3.in‚0% disk.nvme0n1.out‚146137.201171875 disk.sda.in‚ 106.21875( disk./.usagePercent‚37.9048989560791$ disk./.usageGB‚564.8103866577148 disk.sda1.in‚0 disk.sda3.out‚ 1275.9765625 disk.sda1.out‚0 disk.drbd2165.out‚0üô/•¢0‚(:Š ºÏéÒÄÞ÷ö gpu.0.gpu‚100.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚224.914' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚44.9828 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0KÆEMA£0‚(:¶ ºÏéÒ×°¼ù) disk./.usagePercent‚37.90476327208596 disk.sda.out‚ 1276.19921875 disk.sda1.out‚0 network.sent‚ 2.3121397e+07 disk.sda.in‚ 106.21875# disk.dm-0.out‚146170.8134765625# disk.nvme0c0n1.in‚ 47047.6015625& proc.memory.availableMB‚ 2821.21875* proc.memory.percent‚17.119598388671875 network.recv‚ 4.945165e+06 disk.rbd1.out‚ 51.8623046875 disk.sda1.in‚0 disk.drbd2165.in‚0 disk.rbd3.in‚0! disk.nvme0n1.in‚ 47047.6015625& disk.nvme0n1.out‚146170.8134765625 disk.sda2.in‚0$ disk./.usageGB‚564.8083648681641 disk.dm-1.in‚ 106.21875( disk.nvme0c0n1.out‚146170.8134765625 disk.sda2.out‚0 proc.cpu.threads‚17 disk.dm-1.out‚ 1276.19921875 disk.dm-0.in‚ 47047.6015625 disk.drbd2165.out‚0$ memory_percent‚82.78064727783203 disk.rbd1.in‚ 0.52392578125 disk.rbd0.out‚0 disk.sda3.in‚ 106.21875 disk.sda3.out‚ 1276.19921875 disk.rbd3.out‚ 0.4951171875 proc.memory.rssMB‚2804.875 cpu‚25.585979492134726& disk.rbd0.in‚1.759218604441423e+13Ìðw¸4¤0/ _wandbruntime‚19880  _runtime‚19880Ò6Ë¥0Å trainer/global_step‚11999 % _timestamp‚1.7843096965444422e+09 _runtime‚19880.763065547 _step‚985 ' lr-AdamW/pg1‚0.00024755239689778347 ' lr-AdamW/pg2‚0.00024755239689778347%…&Ц0Ê % _timestamp‚1.7843096965444422e+09 _runtime‚19880.763065547 _step‚985 ' lr-AdamW/pg1‚0.00024755239689778347 ' lr-AdamW/pg2‚0.00024755239689778347 trainer/global_step‚11999Ù8âE=4§0/ _wandbruntime‚19880  _runtime‚198807nî(¨0ŽP VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 102304545 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚931424266768384 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3447107699023872 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚3447107699023872 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚64778866 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11284554240000 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚235 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 trainer/global_step‚11999 _step‚986 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚931423898677248 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚32439296 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11265246737145 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚30778821 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚3445619210989568 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 _runtime‚19880.76320489 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11266591337467 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚3447107725197312 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚3445619185561600 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 206191837 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 206201816 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚34000040 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14549963135488 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚30778826 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 102304539 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚3456884432298745 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚945974262343168 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚945973861812736 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 103887298 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3458392253263872 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚3458392253263872 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11285899629568 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 103897271 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚240 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3456885802327035 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11285899629568 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 400530432 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 % _timestamp‚1.7843096965452428e+09 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14549995574784 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 102304545 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 103897271 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11284554240000 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 103887298 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3458393624826880 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚64778626 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3458393624826880 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0Yï›Ø ©0“P E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 103887298 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3458392253263872 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚3458392253263872 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11285899629568 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 103897271 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚240 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3456885802327035 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11285899629568 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 400530432 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 % _timestamp‚1.7843096965452428e+09 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14549995574784 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 H?DeviceStatsMonitor.on_train_batch_start/segment.laÜíÙArge_pool.peak‚1175 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 102304545 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 103897271 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11284554240000 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 103887298 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3458393624826880 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚64778626 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3458393624826880 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 102304545 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚931424266768384 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3447107699023872 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚3447107699023872 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚64778866 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11284554240000 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚235 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 trainer/global_step‚11999 _step‚986 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚931423898677248 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚32439296 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11265246737145 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚30778821 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚3445619210989568 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 _runtime‚19880.76320489 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11266591337467 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚3447107725197312 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚3445619185561600 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 206191837 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 206201816 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚34000040 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14549963135488 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚30778826 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 102304539 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚3456884432298745 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚945974262343168 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚945973861812736Úé0ê„4ª0/ _wandbruntime‚19882  _runtime‚19882R¤:'«0™N ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚3457163237543167 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚64784151 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11267514803825 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 103905763 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚3447385725927424 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 102312745 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 206218514 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14551196279296 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14551163847168 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3457164607571569 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3445897092767744 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 % _timestamp‚1.7843096979294903e+09 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚30781369 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚987 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 103905763 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 206208521 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11285479275520 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚34002782 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11266170203391 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚931501385084928 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 trainer/global_step‚11999 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚30781364 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚64783915 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚931501753176064 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 102312745 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 206218514 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3445897067339776 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3447385699753984 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 400523264 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚946052949455360 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11285479275520 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 103895776 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640Æ™²Å$'¬0žN OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 102312751 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 103895776 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3458672550599680 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 206208521 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚32432128 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 206218514 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3445897067339776 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3447385699753984 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 400523264 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚946052949455360 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11285479275520 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 103895776 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚3457163237543167 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚64784151 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11267514803825 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 103905763 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚3447385725927424 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 102312745 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 206218514 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14551196279296 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14551163847168 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3457164607571569 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3445897092767744 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 % _timestamp‚1.7843096979294903e+09 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚30781369 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚987 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 103905763 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 206208521 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11285479275520 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚34002782 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11266170203391 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚931501385084928 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 trainer/global_step‚11999 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚30781364 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚64783915 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚931501753176064 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 102312745 Q&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7546296119689941 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6694518327713013 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 +particle/specificity‚0.8955490589141846 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.16280612349510193 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.42426395416259766 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.26670071482658386 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.03603396937251091 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6529145240783691 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7010915875434875 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5716547966003418 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.16207164525985718 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 particle/accuracy_1_of_5‚1 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.40740740299224854 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9675925374031067 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7384542226791382 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 +particle/sensitivity‚0.5896269083023071 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.44212961196899414 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.8571428656578064 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7985260486602783 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6509882807731628 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6513180732727051 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9937976002693176 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.05000000074505806 *jet/accuracy_2_of_4‚0.6649686098098755 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6002768874168396 8"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.644493043422699 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9877882599830627 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.12976807355880737 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.02274966798722744 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.5911458730697632 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.7783293128013611 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 jet/accuracy_4_of_5‚0 epoch‚29 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9947019219398499 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 7validation_average_jet_accuracy‚0.32875925302505493 jet/accuracy_2_of_5‚1 jet/accuracy_3_of_5‚1 % _timestamp‚1.7843097243776171e+09 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9259259104728699 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.46530410647392273 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9814428687095642 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.03228423744440079 +jet/accuracy_4_of_4‚0.01628045178949833 /particle/accuracy_4_of_4‚0.9452607035636902 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6006007194519043 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5518558621406555 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6581449508666992 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 particle/accuracy_2_of_3‚1 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.03568393364548683 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7405298352241516 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.1447967141866684 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 particle/accuracy_3_of_4‚1 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7067535519599915 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 particle/accuracy_5_of_5‚0 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9879798889160156 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.7834089398384094 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.2083333283662796 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.7967120409011841 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 _runtime‚19908.595408241 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9179728627204895 8"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.637731671333313 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.23608845472335815 particle/accuracy_1_of_3‚1 9"Purity/*FRt2SRqqt*FBt/event_purity‚0.1863406002521515 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5882753133773804 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9970017671585083 8"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.655674397945404 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.08508814126253128 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 +jet/accuracy_3_of_3‚0.04386269301176071 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8623236417770386 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9994588494300842 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.22913235425949097 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 particle/accuracy_2_of_2‚1 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3349742591381073 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 *jet/accuracy_1_of_4‚0.9523818492889404 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.12654951214790344 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6761233806610107 *jet/accuracy_3_of_4‚0.0460347905755043 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6176846623420715 8"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.624143123626709 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9038461446762085 jet/accuracy_5_of_5‚0 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.15900808572769165 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 trainer/global_step‚11999 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.12962964177131653 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.14557117223739624 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 +"Purity/2FRt*SRqqt2FBt/event_purity‚0.75 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.16153262555599213 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.13994207978248596 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5905072093009949 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.16207164525985718 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.1538461595773697 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.1786903738975525 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 - Purity/0FRt2SRqqt1FBt/FBt_purity‚0.953125 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.46493375301361084 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.011742228642106056 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0044373576529324055 8"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.796413779258728 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9976726770401001 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9955974817276001 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.03712765499949455 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.1995995193719864 )loss/val_total_loss‚8.909951667995687 6 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.990075409412384 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9929051995277405 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 * Purity/2FRt*SRqqt2FBt/FRt_purity‚0.875 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.17124022543430328 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7309466004371643 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 particle/accuracy_4_of_5‚1 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.05000000074505806 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 6 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.512499988079071 particle/accuracy_1_of_2‚1 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.14573544263839722 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 particle/accuracy_2_of_5‚1 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.1593962460756302 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6199240684509277 - Purity/0FRt2SRqqt*FBt/FBt_purity‚0.953125 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 *jet/accuracy_1_of_3‚0.9081710577011108 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6763877868652344 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 +jet/accuracy_2_of_2‚0.11252303421497345 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9940670728683472 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8025002479553223 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9992673993110657 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.16661392152309418 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 6 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.996728777885437 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9971821308135986 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4140625 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9992673993110657 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.5682171583175659 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.12298078835010529 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9988662004470825 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3440580368041992 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.16253705322742462 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.03670666366815567 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9962545037269592 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.18180403113365173 particle/accuracy_3_of_5‚1 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.12962964177131653 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7302516102790833 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 particle/accuracy_2_of_4‚1 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.17541857063770294 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.12350355833768845 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5882753133773804 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.024619849398732185 7"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.66740882396698 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.22363540530204773 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.17516835033893585 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.21935106813907623 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.09666667133569717 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7546296119689941 8"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.796413779258728 *jet/accuracy_1_of_2‚0.7306475639343262 'particle/f_score‚0.5981227159500122 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.5962218642234802 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.1563953012228012 particle/accuracy_1_of_1‚1 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6801548004150391 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.11367584764957428 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.15541177988052368 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 (particle/accuracy‚0.8298495411872864 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6377679705619812 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 6 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.617996335029602 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6109147071838379 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9891344904899597 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.41208139061927795 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.008677415549755096 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.03228423744440079 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9960039854049683 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.1538461595773697 _step‚989 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.18984173238277435 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9755765199661255 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7924225330352783 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9234426617622375 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9675925374031067 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 *jet/accuracy_1_of_1‚0.2882997989654541 particle/accuracy_3_of_3‚1 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6134642362594604 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7119367122650146 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.027268458157777786 jet/accuracy_1_of_5‚1 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9971989393234253 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9285714030265808L­”>¶0Œ| 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9962545037269592 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.18180403113365173 particle/accuracy_3_of_5‚1 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.12962964177131653 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7302516102790833 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 particle/accuracy_2_of_4‚1 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.17541857063770294 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.12350355833768845 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5882753133773804 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.024619849398732185 7"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.66740882396698 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.22363540530204773 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.17516835033893585 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.21935106813907623 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.09666667133569717 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7546296119689941 8"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.796413779258728 *jet/accuracy_1_of_2‚0.7306475639343262 'particle/f_score‚0.5981227159500122 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.5962218642234802 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.1563953012228012 particle/accuracy_1_of_1‚1 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6801548004150391 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.11367584764957428 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.15541177988052368 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 (particle/accuracy‚0.8298495411872864 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6377679705619812 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 6 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.617996335029602 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6109147071838379 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9891344904899597 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.41208139061927795 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.008677415549755096 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.03228423744440079 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9960039854049683 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.1538461595773697 _step‚989 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.18984173238277435 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9755765199661255 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7924225330352783 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9234426617622375 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9675925374031067 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 *jet/accuracy_1_of_1‚0.2882997989654541 particle/accuracy_3_of_3‚1 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6134642362594604 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7119367122650146 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.027268458157777786 jet/accuracy_1_of_5‚1 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9971989393234253 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.9285714030265808 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3499999940395355 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5427258610725403 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9832489490509033 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7173658609390259 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7834089398384094 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7353813052177429 +jet/accuracy_2_of_3‚0.45976975560188293 particle/accuracy_1_of_4‚1 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7546296119689941 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6694518327713013 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 +particle/specificity‚0.8955490589141846 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.16280612349510193 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.42426395416259766 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.26670071482658386 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.03603396937251091 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6529145240783691 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7010915875434875 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5716547966003418 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.16207164525985718 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 particle/accuracy_1_of_5‚1 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.40740740299224854 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9675925374031067 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7384542226791382 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 +particle/sensitivity‚0.5896269083023071 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.44212961196899414 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.8571428656578064 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7985260486602783 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6509882807731628 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6513180732727051 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9937976002693176 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.05000000074505806 *jet/accuracy_2_of_4‚0.6649686098098755 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6002768874168396 8"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.644493043422699 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9877882599830627 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.12976807355880737 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.02274966798722744 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.5911458730697632 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.7783293128013611 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 jet/accuracy_4_of_5‚0 epoch‚29 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9947019219398499 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 7validation_average_jet_accuracy‚0.32875925302505493 jet/accuracy_2_of_5‚1 jet/accuracy_3_of_5‚1 % _timestamp‚1.7843097243776171e+09 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9259259104728699 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.46530410647392273 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9814428687095642 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.03228423744440079 +jet/accuracy_4_of_4‚0.01628045178949833 /particle/accuracy_4_of_4‚0.9452607035636902 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6006007194519043 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5518558621406555 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6581449508666992 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 particle/accuracy_2_of_3‚1 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.03568393364548683 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7405298352241516 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.1447967141866684 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 particle/accuracy_3_of_4‚1 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7067535519599915 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 particle/accuracy_5_of_5‚0 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9879798889160156 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.7834089398384094 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.2083333283662796 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.7967120409011841 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 _runtime‚19908.595408241 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9179728627204895 8"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.637731671333313 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.23608845472335815 particle/accuracy_1_of_3‚1 9"Purity/*FRt2SRqqt*FBt/event_purity‚0.1863406002521515 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5882753133773804 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9970017671585083 8"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.655674397945404 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.08508814126253128 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 +jet/accuracy_3_of_3‚0.04386269301176071 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8623236417770386 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9994588494300842 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.22913235425949097 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 particle/accuracy_2_of_2‚1 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3349742591381073 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 *jet/accuracy_1_of_4‚0.9523818492889404 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.12654951214790344 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6761233806610107 *jet/accuracy_3_of_4‚0.0460347905755043 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6176846623420715 8"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.624143123626709 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9038461446762085 jet/accuracy_5_of_5‚0 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.15900808572769165 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 trainer/global_step‚11999 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.12962964177131653 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.14557117223739624 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 +"Purity/2FRt*SRqqt2FBt/event_purity‚0.75 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.16153262555599213 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.13994207978248596 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5905072093009949 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.16207164525985718 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.1538461595773697 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.1786903738975525 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 - Purity/0FRt2SRqqt1FBt/FBt_purity‚0.953125 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.46493375301361084 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.011742228642106056 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0044373576529324055 8"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.796413779258728 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9976726770401001 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9955974817276001 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.03712765499949455 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.1995995193719864 )loss/val_total_loss‚8.909951667995687 6 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.990075409412384 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9929051995277405 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 * Purity/2FRt*SRqqt2FBt/FRt_purity‚0.875 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.17124022543430328 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7309466004371643 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 particle/accuracy_4_of_5‚1 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.05000000074505806 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 6 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.512499988079071 particle/accuracy_1_of_2‚1 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.14573544263839722 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 particle/accuracy_2_of_5‚1 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.1593962460756302 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6199240684509277 - Purity/0FRt2SRqqt*FBt/FBt_purity‚0.953125 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 *jet/accuracy_1_of_3‚0.9081710577011108 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6763877868652344 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 +jet/accuracy_2_of_2‚0.11252303421497345 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9940670728683472 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8025002479553223 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9992673993110657 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.16661392152309418 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 6 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.996728777885437 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9971821308135986 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4140625 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9992673993110657 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.5682171583175659 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.12298078835010529 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9988662004470825 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3440580368041992 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.16253705322742462 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.03670666366815567 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0ݯO5ÝŸ·0‚(:” çÏéÒ¯ª¤÷ gpu.0.gpu‚39.0 gpu.0.memory‚12, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚38.0 gpu.0.powerWatts‚109.997' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.999399999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0̸Â;ï¸0‚(:½ çÏéÒ×Òûø# memory_percent‚82.7838659286499$ disk./.usageGB‚564.8279457092285) disk.nvme0c0n1.out‚146204.45849609375' disk.nvme0n1.out‚146204.45849609375 network.sent‚ 2.3194232e+07) disk./.usagePercent‚37.90607735875921 disk.rbd3.out‚ 0.4951171875 disk.sda2.in‚0 disk.dm-0.in‚47072.94140625 disk.sda1.out‚0) proc.memory.availableMB‚ 2820.69140625 disk.drbd2165.in‚0 disk.rbd3.in‚0 disk.rbd1.in‚ 0.52392578125 disk.dm-1.out‚ 1277.0859375& disk.rbd0.in‚1.759218604441423e+13 cpu‚25.58907116960493 proc.cpu.threads‚17 disk.drbd2165.out‚0$ disk.nvme0c0n1.in‚47072.94140625 disk.sda.in‚ 108.921875$ disk.dm-0.out‚146204.45849609375* proc.memor d(ýYy.percent‚17.119598388671875 network.recv‚ 4.955843e+06 disk.sda2.out‚0 disk.sda.out‚ 1277.0859375 disk.rbd0.out‚0 disk.dm-1.in‚ 108.921875 proc.memory.rssMB‚2804.875 disk.sda3.out‚ 1277.0859375" disk.nvme0n1.in‚47072.94140625 disk.rbd1.out‚ 52.0029296875 disk.sda1.in‚0 disk.sda3.in‚ 108.921875d'«Ÿ¹0‚(:” öÏéÒ­¶‡÷ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚283.559' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚56.711800000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0V £uEº0‚(:º öÏéÒ¡¨¼ù disk.sda2.out‚0 disk.sda1.out‚0 disk.rbd3.out‚ 0.4951171875 cpu‚25.588915203176594 disk.rbd1.in‚ 0.52392578125" disk.dm-0.out‚146222.123046875 disk.sda3.in‚ 108.921875$ disk./.usageGB‚564.8325347900391 disk.rbd1.out‚ 52.0029296875 proc.memory.rssMB‚2804.875* proc.memory.percent‚17.119598388671875 disk.sda.in‚ 108.921875 disk.sda.out‚ 1277.2890625 disk.dm-1.out‚ 1277.2890625" disk.nvme0n1.in‚47229.03515625% disk.nvme0n1.out‚146222.123046875 proc.cpu.threads‚17* disk./.usagePercent‚37.906385335822904 disk.sda2.in‚0 disk.sda1.in‚0 disk.drbd2165.out‚0$ disk.nvme0c0n1.in‚47229.03515625' disk.nvme0c0n1.out‚146222.123046875 network.sent‚ 2.3252064e+07& disk.rbd0.in‚1.759218604441423e+13 disk.dm-0.in‚47229.03515625 disk.drbd2165.in‚0 disk.rbd3.in‚0 disk.dm-1.in‚ 108.921875 disk.rbd0.out‚0 disk.sda3.out‚ 1277.2890625$ memory_percent‚82.78405666351318) proc.memory.availableMB‚ 2820.66015625 network.recv‚ 4.960311e+06c$õ÷»0‚(:’ …ÐéÒ…ðîö gpu.0.gpu‚13.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚337.288' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚67.45760000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0k­Í =¼0‚(:² …ÐéÒçÐßø disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.rbd3.in‚0& disk.nvme0c0n1.out‚146231.42578125 disk.rbd1.out‚ 52.0654296875 disk.sda1.out‚0$ memory_percent‚82.78422355651855) disk./.usagePercent‚37.90682618079686 disk.sda3.out‚ 1277.671875 network.sent‚ 2.3257199e+07$ disk./.usageGB‚564.8391036987305$ disk.nvme0c0n1.in‚47229.23828125 disk.sda.out‚ 1277.671875! disk.dm-0.out‚146231.42578125 disk.sda3.in‚ 108.921875$ disk.nvme0n1.out‚146231.42578125 disk.sda.in‚ 108.921875 disk.sda1.in‚0( proc.memory.availableMB‚ 2820.6328125 proc.memory.rssMB‚2804.875 disk.dm-1.out‚ 1277.671875 disk.dm-0.in‚47229.23828125 proc.cpu.threads‚17 network.recv‚ 4.963632e+06 disk.rbd3.out‚ 0.4951171875" disk.nvme0n1.in‚47229.23828125 disk.sda2.out‚0 disk.dm-1.in‚ 108.921875 disk.rbd0.out‚0 cpu‚25.588762741842395 disk.sda2.in‚0 disk.rbd1.in‚ 0.52392578125& disk.rbd0.in‚1.759218604441423e+13* proc.memory.percent‚17.119598388671875Ú· ^”½0‚(:‰ ”ÐéÒŠ˜´÷ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚270.441' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.0882 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ËŽmlF¾0‚(:» ”ÐéÒ§œù disk.sda1.in‚0 disk.sda.in‚ 108.921875* disk./.usagePercent‚37.907362004566025" disk.nvme0n1.in‚47229.66015625 disk.rbd0.out‚0 network.sent‚ 2.326233e+07 network.recv‚ 4.967262e+06 disk.drbd2165.in‚0$ disk.nvme0c0n1.in‚47229.66015625$ disk.dm-0.out‚146242.50146484375* proc.memory.percent‚17.119598388671875 proc.cpu.threads‚17 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd1.in‚ 0.52392578125 disk.dm-0.in‚47229.66015625 proc.memory.rssMB‚2804.875$ disk./.usageGB‚564.8470878601074 disk.sda3.in‚ 108.921875 disk.drbd2165.out‚0 disk.rbd3.out‚ 0.4951171875 disk.dm-1.out‚ 1277.890625) disk.nvme0c0n1.out‚146242.50146484375 disk.sda.out‚ 1277.890625 disk.dm-1.in‚ 108.921875 disk.sda1.out‚0 disk.sda3.out‚ 1277.890625$ memory_percent‚82.78584480285645( proc.memory.availableMB‚ 2820.3671875 cpu‚25.588594802060776 disk.rbd3.in‚0' disk.nvme0n1.out‚146242.50146484375 disk.rbd1.out‚ 52.0654296875& disk.rbd0.in‚1.759218604441423e+13èþ×&”¿0‚(:‰ £ÐéÒîàüö gpu.0.gpu‚93.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚46.0 gpu.0.powerWatts‚331.878' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚66.3756 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0åÉ[<À0‚(:± £ÐéÒà…Èù disk.sda2.out‚0 disk.sda.out‚ 1278.171875 disk.dm-1.out‚ 1278.171875$ disk.nvme0c0n1.in‚47229.87890625 cpu‚25.58845245316892 disk.sda1.out‚0( disk./.usagePercent‚37.9076548771852 disk.sda2.in‚0 disk.rbd1.out‚ 52.1435546875 disk.dm-1.in‚ 108.921875 disk.drbd2165.in‚0 proc.memory.rssMB‚2804.875 network.recv‚ 4.970648e+06$ disk./.usageGB‚564.8514518737793! disk.dm-0.out‚146255.87890625 disk.sda1.in‚0 disk.drbd2165.out‚0& disk.nvme0c0n1.out‚146255.87890625) proc.memory.availableMB‚ 2819.93359375* proc.memory.percent‚17.119598388671875 network.sent‚ 2.3267395e+07 disk.rbd1.in‚ 0.52392578125& disk.rbd0.in‚1.759218604441423e+13 disk.sda3.in‚ 108.921875 disk.sda3.out‚ 1278.171875 proc.cpu.threads‚17 disk.sda.in‚ 108.921875 disk.dm-0.in‚47229.87890625" disk.nvme0n1.in‚47229.87890625$ memory_percent‚82.78849124908447 disk.rbd3.in‚0 disk.rbd3.out‚ 0.4951171875 disk.rbd0.out‚0$ disk.nvme0n1.out‚146255.87890625Â6÷4Á0/ _wandbruntime‚19985  _runtime‚19985Êš©+ÈÂ0 & lr-AdamW/pg1‚0.0002448977152837679 & lr-AdamW/pg2‚0.0002448977152837679 trainer/global_step‚12049 $ _timestamp‚1.784309801702203e+09 _runtime‚19985.919335607 _step‚990Í.¡\ÍÃ0Ç trainer/global_step‚12049 $ _timestamp‚1.784309801702203e+09 _runtime‚19985.919335607 _step‚990 & lr-AdamW/pg1‚0.0002448977152837679 & lr-AdamW/pg2‚0.0002448977152837679ÞŸ·£4Ä0/ _wandbruntime‚19985  _runtime‚19985)Þ¾O(Å0P QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 104372032 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚3463339183403008 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚34168849 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11318657582012 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11317312981690 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 207218877 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3476173801309696 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚32439296 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3464835743137792 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚3476172429746688 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3476172429746688 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚3464835743137792 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11336712782336 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 102846845 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚935346699399168 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3476173801309696 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 102846845 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 104382005 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 trainer/global_step‚12049 _step‚991 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚65083743 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚34168612 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 104382005 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11338058171904 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚935346331308032 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚949963524303360 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 104372032Ñ©ì(Æ0”P M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 102846845 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 104382005 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 trainer/global_step‚12049 _step‚991 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚65083743 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚34168612 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 104382005 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11338058171904 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚935346331308032 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚949963524303360 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 104372032 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 104372032 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚3463339183403008 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚34168849 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11318657582012 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11317312981690 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 207218877 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3476173801309696 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚32439296 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3464835743137792 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚3476172429746688 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3476172429746688 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚3464835743137792 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11336712782336 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 102846845 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚935346699399168 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3476173801309696 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0ß:uO,4Ç0/ _wandbruntime‚19987  _runtime‚19987#ñ½hÈ0™N XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3476452727082496 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 104390497 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 102855051 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 104390497 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3465113717694464 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚950042225668608 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚65088710 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 ?6DeviceStatsMonitor.on_trêŠAt·ain_batch_end/segment.all.peak‚1995 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11338983214592 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 trainer/global_step‚12049 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 104380510 _step‚992 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚226 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚3465113717694464 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚34171504 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚935424185806848 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11337637817856 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚65088941 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 207245554 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0û $H$'É0žN L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 207245554 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3476452727082496 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 104390497 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 102855051 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 104390497 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3465113717694464 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚950042225668608 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚65088710 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11338983214592 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 trainer/global_step‚12049 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 104380510 _step‚992 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚226 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚3465113717694464 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚34171504 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚935424185806848 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11337637817856 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚65088941 QDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 104797784 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3490120153745920 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 104797784 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚953897891390976 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚3477233272311808 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 208054929 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 103257151 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11384374102528 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14677239698944 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3478735779643392 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚259 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 _step‚995 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3478735753469952 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 103257145 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚3477233297739776 trainer/global_step‚12099 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11384374102528 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚65346146 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 400530432 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11364894695120 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peaö9Ê $k‚12097 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3490120153745920 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11383028712960 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 104807757 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3490118782182912 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚3478735779643392 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚939220651692032 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚34304102¢µÀ8(Ü0”P JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 104797784 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3490120153745920 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 104797784 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚953897891390976 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚3477233272311808 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 208054929 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 103257151 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11384374102528 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14677239698944 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3478735779643392 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚259 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 _step‚995 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3478735753469952 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 103257145 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚3477233297739776 trainer/global_step‚12099 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11384374102528 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚65346146 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 400530432 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11364894695120 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3490120153745920 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11383028712960 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 104807757 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3490118782182912 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚3478735779643392 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚939220651692032 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚34304102 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11363550094798 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 103257151 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚32439296 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3488598192434896 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 104807757 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚3488596822406606 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚953898291921408 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚3478735753469952 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚31042039 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚34303843 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚939221019783168 _runtime‚20064.582141437 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚31042044 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820ãÉox4Ý0/ _wandbruntime‚20065  _runtime‚20065m¬º¡'Þ0—N UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚953976973618688 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚953976570998272 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚3488875627651028 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 208071613 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 208081606 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3488876997679430 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11383953748480 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14678432898560 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 _runtime‚20065.968943685 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚3479013780373504 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3490399079518720 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11385299145216 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚65351441 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 104816249 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3479013780373504 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11383953748480 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 trainer/global_step‚12099 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 103265351 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11385299145216 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3490399079518720 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 103265357 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚34306854 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚31044587 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚34306600 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚65351182 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3477511154089984 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 104806262 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3490397707948544 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11365818161478 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 208071613 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚939298138099712 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 104816249 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 208081606 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3488876997679430 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11383953748480 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14678432898560 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 _runtime‚20065.968943685 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚3479013780373504 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3490399079518720 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11385299145216 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚65351441 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 104816249 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3479013780373504 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11383953748480 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 trainer/global_step‚12099 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 103265351 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11385299145216 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3490399079518720 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 103265357 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚34306854 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚31044587 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚34306600 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚65351182 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3477511154089984 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 104806262 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3490397707948544 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11365818161478 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0ä i4à0/ _wandbruntime‚20066  _runtime‚20066r ¦êá0‰ 2loss/FBt1/assignment_loss‚0.012698520611241425 3loss/FBt2/assignment_loss‚0.0002560330565745081 /loss/FBt2/detection_loss‚0.3329027932153211 epoch‚30 _runtime‚20066.054037179 3loss/SRqqt1/assignment_losSíÚ%s‚0.20801443633738242 3loss/SRqqt2/assignment_loss‚0.22866613480851103 0loss/FRt1/detection_loss‚0.46625341654634556 $ _timestamp‚1.784309881837083e+09 /loss/FRt2/detection_loss‚1.3425543469200094 1loss/SRqqt1/detection_loss‚0.5940941884919556 %loss/total_loss‚5.714326811596573 _step‚997 0loss/FRt1/assignment_loss‚1.0405559556972053 0loss/FRt2/assignment_loss‚0.2858815379166193 1loss/SRqqt2/detection_loss‚0.4799571075028895 /loss/FBt1/detection_loss‚0.7224923404925182 trainer/global_step‚12099“‘áèâ0Ž /loss/FRt2/detection_loss‚1.3425543469200094 1loss/SRqqt1/detection_loss‚0.5940941884919556 %loss/total_loss‚5.714326811596573 _step‚997 0loss/FRt1/assignment_loss‚1.0405559556972053 0loss/FRt2/assignment_loss‚0.2858815379166193 1loss/SRqqt2/detection_loss‚0.4799571075028895 /loss/FBt1/detection_loss‚0.7224923404925182 trainer/global_step‚12099 2loss/FBt1/assignment_loss‚0.012698520611241425 3loss/FBt2/assignment_loss‚0.0002560330565745081 /loss/FBt2/detection_loss‚0.3329027932153211 epoch‚30 _runtime‚20066.054037179 3loss/SRqqt1/assignment_loss‚0.20801443633738242 3loss/SRqqt2/assignment_loss‚0.22866613480851103 0loss/FRt1/detection_loss‚0.46625341654634556 $ _timestamp‚1.784309881837083e+09å‹È-¶ ã0‚(:• ýÐéÒóüû÷ gpu.0.gpu‚100.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚201.501' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚40.300200000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ÿx‚÷Mä0‚(: ýÐéÒ¤ˆ“ú network.sent‚ 2.3356463e+07$ disk./.usageGB‚564.8877143859863 disk.rbd3.in‚0" disk.nvme0n1.in‚47506.23828125 disk.sda2.out‚0 disk.rbd0.out‚0 disk.dm-0.in‚47506.23828125 disk.sda3.in‚ 108.921875& disk.rbd0.in‚1.759218604441423e+13$ memory_percent‚82.78849124908447) disk./.usagePercent‚37.91008848480569 disk.sda.out‚ 1280.53515625$ disk.dm-0.out‚146324.39794921875 disk.sda3.out‚ 1280.53515625 network.recv‚ 4.992199e+06) disk.nvme0c0n1.out‚146324.39794921875 proc.memory.rssMB‚2804.875* proc.memory.percent‚17.119598388671875 disk.drbd2165.out‚0 disk.rbd3.out‚ 0.4951171875 disk.sda1.in‚0 disk.drbd2165.in‚0$ disk.nvme0c0n1.in‚47506.23828125' disk.nvme0n1.out‚146324.39794921875 disk.rbd1.out‚ 52.3466796875 disk.dm-1.in‚ 108.921875 disk.dm-1.out‚ 1280.53515625 cpu‚25.587532138281823 disk.rbd1.in‚ 0.52392578125 disk.sda.in‚ 108.921875 proc.cpu.threads‚17 disk.sda2.in‚0 disk.sda1.out‚0) proc.memory.availableMB‚ 2819.93359375ëEƒŸå0‚(:” ŒÑéÒãýçö gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚269.302' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚53.860400000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0l ªº9æ0‚(:® ŒÑéÒúÜÕù network.sent‚ 2.3419757e+07 disk.sda3.out‚ 1280.76953125 disk.drbd2165.in‚0# disk.nvme0c0n1.in‚ 47565.2578125 disk.dm-1.out‚ 1280.76953125 proc.memory.rssMB‚2804.875& disk.rbd0.in‚1.759218604441423e+13 disk.dm-0.in‚ 47565.2578125 disk.rbd3.in‚0 disk.rbd3.out‚ 0.4951171875) proc.memory.availableMB‚ 2819.92578125* proc.memory.percent‚17.119598388671875$ disk./.usageGB‚564.8969917297363 disk.drbd2165.out‚0 proc.cpu.threads‚17) disk./.usagePercent‚37.91071109512896 disk.sda3.in‚ 108.921875! disk.nvme0n1.in‚ 47565.2578125 disk.sda2.in‚0 disk.rbd1.out‚ 52.3466796875 disk.sda1.in‚0" disk.nvme0n1.out‚ 146335.171875 cpu‚25.58736865094365 network.recv‚ 4.996692e+06 disk.sda1.out‚0 disk.sda.out‚ 1280.76953125$ disk.nvme0c0n1.out‚ 146335.171875 disk.sda2.out‚0 disk.rbd1.in‚ 0.52392578125 disk.rbd0.out‚0 disk.dm-0.out‚ 146335.171875 disk.sda.in‚ 108.921875 disk.dm-1.in‚ 108.921875# memory_percent‚82.7885389328003iˆ¢”ç0‚(:‰ ›ÑéÒž”õö gpu.0.gpu‚89.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚166.909' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚33.3818 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¯€PR@è0‚(:µ ›ÑéÒ¡‹Îù network.recv‚ 5.000051e+06 disk.sda2.out‚0& disk.rbd0.in‚1.759218604441423e+13 disk.sda3.out‚ 1281.03125 disk.drbd2165.out‚0$ disk./.usageGB‚564.9003601074219 disk.rbd1.in‚ 0.52392578125 disk.dm-1.out‚ 1281.03125 disk.rbd0.out‚0 disk.sda1.out‚0 disk.sda3.in‚ 108.921875 disk.rbd3.in‚0 disk.rbd1.out‚ 52.4248046875 disk.dm-1.in‚ 108.921875 disk.sda1.in‚0 disk.drbd2165.in‚0! disk.nvme0n1.in‚ 47565.4609375' disk.nvme0n1.out‚146348.06103515625$ memory_percent‚82.78717994689941 disk.dm-0.in‚ 47565.4609375$ disk.dm-0.out‚146348.06103515625 disk.rbd3.out‚ 0.4951171875 cpu‚25.58722914350068 disk.sda.in‚ 108.921875) disk.nvme0c0n1.out‚146348.06103515625 proc.cpu.threads‚17 disk.sda2.in‚0 disk.sda.out‚ 1281.03125# disk.nvme0c0n1.in‚ 47565.4609375* proc.memory.percent‚17.119598388671875 network.sent‚ 2.3424999e+07* disk./.usagePercent‚37.910937149781695( proc.memory.availableMB‚ 2820.1484375 proc.memory.rssMB‚2804.875¢îÖ®”é0‚(:‰ ªÑéÒÿ¨€÷ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚273.606' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.7212 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0gðÄ€<ê0‚(:± ªÑéÒ§ÏÅù! disk.nvme0n1.in‚ 47565.6796875' disk.nvme0n1.out‚146360.64404296875 disk.rbd1.out‚ 52.4248046875 disk.sda1.out‚0) disk.nvme0c0n1.out‚146360.64404296875 proc.cpu.threads‚17& disk.rbd0.in‚1.759218604441423e+13 disk.sda3.out‚ 1281.1875 disk.rbd3.out‚ 0.4951171875 disk.sda2.in‚0 network.recv‚ 5.003464e+06 disk.sda.out‚ 1281.1875 disk.rbd0.out‚0# disk.nvme0c0n1.in‚ 47565.6796875$ memory_percent‚82.78722763061523 disk.dm-1.out‚ 1281.1875 disk.dm-0.in‚ 47565.6796875 disk.sda3.in‚ 108.921875' proc.memory.availableMB‚ 2820.140625 cpu‚25.587065035922254 network.sent‚ 2.3430067e+07) disk./.usagePercent‚37.91140026741113 disk.rbd1.in‚ 0.52392578125$ disk./.usageGB‚564.9072608947754 disk.dm-1.in‚ 108.921875 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.sda2.out‚0* proc.memory.percent‚17.119598388671875 disk.sda.in‚ 108.921875$ disk.dm-0.out‚146360.64404296875 disk.sda1.in‚0 disk.rbd3.in‚0 proc.memory.rssMB‚2804.875£«ÌLŸë0‚(:” ¹ÑéÒØ¥‚÷ gpu.0.gpu‚90.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚258.574' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚51.714800000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÜÑÈŽIì0‚(:¾ ¹ÑéÒÞ·¼ú disk.dm-1.in‚ 108.921875 disk.drbd2165.in‚0 disk.rbd3.in‚0( disk.nvme0c0n1.out‚146374.8564453125$ memory_percent‚82.78610706329346) proc.memory.availableMB‚ 2820.32421875 cpu‚25.586935796157814# disk.dm-0.out‚146374.8564453125 disk.sda3.in‚ 108.921875 disk.drbd2165.out‚0" disk.nvme0n1.in‚47866.52734375 disk.sda2.out‚0* proc.memory.percent‚17.119598388671875 disk.sda3.out‚ 1281.64453125$ disk.nvme0c0n1.in‚47866.52734375 disk.sda.out‚ 1281.64453125& disk.rbd0.in‚1.759218604441423e+13 proc.memory.rssMB‚2804.875 disk.sda1.in‚0& disk.nvme0n1.out‚146374.8564453125 network.recv‚ 5.006724e+06 disk.rbd0.out‚0 proc.cpu.threads‚17) disk./.usagePercent‚37.91185928892003 disk.sda1.out‚0 disk.dm-1.out‚ 1281.64453125 network.sent‚ 2.343518e+07 disk.dm-0.in‚47866.52734375 disk.rbd3.out‚ 0.4951171875 disk.rbd1.in‚ 0.52392578125 disk.rbd1.out‚ 52.5498046875$ disk./.usageGB‚564.9141006469727 disk.sda2.in‚0 disk.sda.in‚ 108.921875Öæí¡4í0/ _wandbruntime‚20143  _runtime‚20143¼ò_Ëî0Å trainer/global_step‚12149 % _timestamp‚1.7843099592880778e+09 _runtime‚20143.505249552 _step‚998 ' lr-AdamW/pg1‚0.00023960798837620502 ' lr-AdamW/pg2‚0.00023960798837620502 Â)Ðï0Ê % _timestamp‚1.7843099592880778e+09 _runtime‚20143.505249552 _step‚998 ' lr-AdamW/pg1‚0.00023960798837620502 ' lr-AdamW/pg2‚0.00023960798837620502 trainer/global_step‚12149æ­U|4ð0/ _wandbruntime‚20143  _runtime‚20143°`öB(ñ0P NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚3492635789975552 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 103667451 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 208900960 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14737323106816 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚3492635816148992 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚3502537148428514 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3492635789975552 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3492635816148992 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚65610740 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 105233509 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3504066506182144 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚3491127386648576 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3504066506182144 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3504065134619136 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 105223536 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 103667451 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 208890981 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚31169189 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 trainer/global_step‚12149 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚943094972076032 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚65610991 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚32439296 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 % _timestamp‚1.7843099592891657e+09 _runtime‚20143.509938381 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11429344643584 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 103667445 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚34441797 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11411131808228 _step‚999 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚957832262743552 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3504065134619136 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 105223536 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 103667451 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 208890981 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚31169189 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 trainer/global_step‚12149 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚943094972076032 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚65610991 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚32439296 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 % _timestamp‚1.7843099592891657e+09 _runtime‚20143.509938381 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11429344643584 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 103667445 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚34441797 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11411131808228 _step‚999 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚957832262743552 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 208900960 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14737323106816 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚3492635816148992 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚3502537148428514 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3492635789975552 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3492635816148992 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚65610740 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 105233509 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3504066506182144 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚3491127386648576 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3504066506182144 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 103675657 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3504344060384768 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11431615075840 _step‚1000 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14738525482496 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3491405242998784 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11412055274586 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 _runtime‚20144.893580691 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚957911352057344 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11431615075840 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3504345431954944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 105242001 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 105232014 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚957910951534080 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 105242001 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚3492913790705664 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚31171732 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11410710674152 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 208917658 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 103675651 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 208907665 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚65616047 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚245 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚31171737 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚943172826574848 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14738493050368 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 208917658 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 % _timestamp‚1.7843099606761687e+09 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 103675651 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3491405268426752 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11430269679104 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚34444555 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚34444315 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 208907665 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚3502815953672936 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3492913790705664 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚943172458483712 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 105232014 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 208917658 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 103675651 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 208907665 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚65616047 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚245 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚31171737 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚943172826574848 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14738493050368 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 208917658 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 % _timestamp‚1.7843099606761687e+09 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 103675651 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3491405268426752 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11430269679104 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚34444555 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚34444315 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 208907665 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚3502815953672936 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3492913790705664 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚943172458483712 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 105232014 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 103675657 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3504344060384768 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11431615075840 _step‚1000 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14738525482496 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3491405242998784 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11412055274586 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 _runtime‚20144.893580691 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚957911352057344 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11431615075840 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3504345431954944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 105242001 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 105232014 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚957910951534080 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 105242001 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚3492913790705664 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚31171732 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11410710674152 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640èÿeê!4ö0/ _wandbruntime‚20144  _runtime‚20144ËüdÐ÷0 1loss/SRqqt2/detection_loss‚0.5098194527533076 % _timestamp‚1.7843099607609684e+09 _runtime‚20144.977884918 _step‚1001 0loss/FRt2/assignment_loss‚0.2091647066079177 1loss/SRqqt1/detection_loss‚0.6088292062623077 0loss/FBt1/detection_loss‚0.48774419830685173 /loss/FBt2/detection_loss‚0.3287325539418905 %loss/total_loss‚5.394814083591164 3loss/SRqqt1/assignment_loss‚0.18349133558455424 3loss/SRqqt2/assignment_loss‚0.19787870480425174 2loss/FBt1/assignment_loss‚0.016559703594760618 trainer/global_step‚12149 4loss/FBt2/assignment_loss‚1.3238261414201817e-06 epoch‚30 0loss/FRt1/assignment_loss‚1.1197427546808505 0loss/FRt1/detection_loss‚0.44669750249607193 /loss/FRt2/detection_loss‚1.2861526407322592^¥Ê ø0’ 0loss/FRt1/detection_loss‚0.44669750249607193 /loss/FRt2/detection_loss‚1.2861526407322592 1loss/SRqqt2/detection_loss‚0.5098194527533076 % _timestamp‚1.7843099607609684e+09 _runtime‚20144.977884918 _step‚1001 0loss/FRt2/assignment_loss‚0.2091647066079177 1loss/SRqqt1/detection_loss‚0.6088292062623077 0loss/FBt1/detection_loss‚0.48774419830685173 /loss/FBt2/detection_loss‚0.3287325539418905 %loss/total_loss‚5.394814083591164 3loss/SRqqt1/assignment_loss‚0.18349133558455424 3loss/SRqqt2/assignment_loss‚0.19787870480425174 2loss/FBt1/assignment_loss‚0.016559703594760618 trainer/global_step‚12149 4loss/FBt2/assignment_loss‚1.3238261414201817e-06 epoch‚30 0loss/FRt1/assignment_loss‚1.1197427546808505éX ¿žù0‚(:“ ÈÑéÒäý…÷ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚281.143' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚56.22859999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0­„>´Jú0‚(:¿ ÈÑéÒ’öù network.recv‚ 5.010186e+06 disk.drbd2165.in‚0( disk.nvme0c0n1.out‚146385.4365234375& disk.nvme0n1.out‚146385.4365234375 disk.dm-1.out‚ 1281.84765625 proc.cpu.threads‚17 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd2165.out‚0 disk.sda2.in‚0 disk.sda.out‚ 1281.84765625# disk.dm-0.out‚146385.4365234375 network.sent‚ 2.3440257e+07) disk./.usagePercent‚37.91216982605906 disk.sda3.out‚ 1281.84765625 disk.sda2.out‚0 disk.rbd1.in‚ 0.52392578125$ disk.nvme0c0n1.in‚47866.73046875 disk.rbd1.out‚ 52.5498046875& disk.rbd0.in‚1.759218604441423e+13 disk.dm-0.in‚47866.73046875* proc.memory.percent‚17.119598388671875 proc.memory.rssMB‚2804.875 disk.sda3.in‚ 108.921875$ disk./.usageGB‚564.9187278747559 disk.rbd3.in‚0" disk.nvme0n1.in‚47866.73046875 disk.sda.in‚ 108.921875 disk.dm-1.in‚ 108.921875) proc.memory.availableMB‚ 2819.83984375 cpu‚25.586773374135593 disk.rbd3.out‚ 0.4951171875 disk.rbd0.out‚0$ memory_percent‚82.78906345367432o?Ÿû0‚(:” ×ÑéÒ“ç½ø gpu.0.gpu‚90.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚292.642' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚58.528400000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Î'ÒMü0‚(: ×ÑéÒßü§ú disk.dm-1.in‚ 108.921875& disk.rbd0.in‚1.759218604441423e+13 disk.sda3.out‚ 1282.09765625 disk.drbd2165.out‚0 disk.rbd3.out‚ 0.4951171875 network.recv‚ 5.014843e+06 disk.rbd0.out‚0) disk.nvme0c0n1.out‚146393.50244140625' disk.nvme0n1.out‚146393.50244140625 network.sent‚ 2.3503611e+07) proc.memory.availableMB‚ 2820.39453125 cpu‚25.586633584357646# disk./.usageGB‚564.925952911377 disk.dm-0.in‚47866.94921875 disk.dm-1.out‚ 1282.09765625$ disk.nvme0c0n1.in‚47866.94921875 disk.sda2.out‚0 disk.rbd1.in‚ 0.52392578125 disk.rbd1.out‚ 52.6279296875 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda.in‚ 108.921875 disk.sda.out‚ 1282.09765625 disk.sda2.in‚0 proc.memory.rssMB‚2804.875 disk.sda3.in‚ 108.921875 disk.rbd3.in‚0" disk.nvme0n1.in‚47866.94921875$ memory_percent‚82.78567790985107* proc.memory.percent‚17.119598388671875 proc.cpu.threads‚17* disk./.usagePercent‚37.912654704328915$ disk.dm-0.out‚146393.50244140625 disk.drbd2165.in‚0ÖÕß”ý0‚(:‰ æÑéÒÆ”ðö gpu.0.gpu‚92.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚40.0 gpu.0.powerWatts‚233.012' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚46.6024 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0-}$Ó?þ0‚(:´ æÑéÒ¿µù network.sent‚ 2.3508261e+07$ disk./.usageGB‚564.9331970214844) disk./.usagePercent‚37.91314086263643 disk.rbd1.in‚ 0.52392578125 disk.sda2.out‚0 disk.rbd1.out‚ 52.6279296875 network.recv‚ 5.018428e+06 disk.drbd2165.in‚0' disk.nvme0c0n1.out‚146408.943359375$ memory_percent‚82.78732299804688$ proc.memory.availableMB‚2820.125" disk.nvme0n1.in‚47867.37109375 disk.sda.in‚ 108.921875 disk.dm-1.in‚ 108.921875 disk.rbd0.out‚0" disk.dm-0.out‚146408.943359375 disk.drbd2165.out‚0 proc.memory.rssMB‚2804.875& disk.rbd0.in‚1.759218604441423e+13 disk.dm-0.in‚47867.37109375 disk.sda1.in‚0* proc.memory.percent‚17.119598388671875 proc.cpu.threads‚17 disk.sda2.in‚0 disk.sda1.out‚0 disk.sda3.in‚ 108.921875 disk.rbd3.out‚ 0.4951171875$ disk.nvme0c0n1.in‚47867.37109375 cpu‚25.586460188243258% disk.nvme0n1.out‚146408.943359375 disk.sda.out‚ 1282.3515625 disk.dm-1.out‚ 1282.3515625 disk.sda3.out‚ 1282.3515625 disk.rbd3.in‚0úÖ# žÿ0‚(:“ õÑéÒ¨‚Øö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚273.684' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚54.73680000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0èû¥=E€1‚(:º õÑéÒ¤áø& disk.rbd0.in‚1.759218604441423e+13 disk.sda3.out‚ 1282.63671875$ memory_percent‚82.78720378875732 disk.sda1.out‚0$ disk.nvme0c0n1.in‚48106.69140625 proc.memory.rssMB‚2804.875 network.recv‚ 5.021003e+06$ disk./.usageGB‚564.9398574829102* proc.memory.percent‚17.119598388671875 disk.dm-0.in‚48106.69140625 disk.drbd2165.in‚0% disk.nvme0n1.out‚146416.517578125 disk.rbd1.in‚ 0.52392578125 disk.sda.out‚ 1282.63671875 network.sent‚ 2.3513017e+07" disk.dm-0.out‚146416.517578125 disk.sda3.in‚ 108.921875 disk.sda2.out‚0 proc.cpu.threads‚17) disk./.usagePercent‚37.91358785179122' disk.nvme0c0n1.out‚146416.517578125 disk.sda2.in‚0 disk.rbd1.out‚ 52.6904296875 disk.dm-1.in‚ 108.921875 disk.dm-1.out‚ 1282.63671875 cpu‚25.5863235544489 disk.rbd0.out‚0 disk.rbd3.out‚ 0.4951171875" disk.nvme0n1.in‚48106.69140625 disk.sda.in‚ 108.921875) proc.memory.availableMB‚ 2820.14453125 disk.sda1.in‚0 disk.drbd2165.out‚0 disk.rbd3.in‚0înÇ1‚(:’ „ÒéÒ¡’‰÷ gpu.0.gpu‚4.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚40.0 gpu.0.powerWatts‚109.757' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.951400000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0žâˆöF‚1‚(:» „ÒéÒÉÄù disk.rbd3.out‚ 0.4951171875$ disk.nvme0c0n1.in‚48107.12890625 disk.sda2.in‚0 disk.sda.out‚ 1282.79296875 disk.sda1.out‚0 disk.drbd2165.in‚0 network.sent‚ 2.3518202e+07 network.recv‚ 5.024483e+06 disk.sda3.out‚ 1282.79296875" disk.nvme0n1.in‚48107.12890625% disk.nvme0n1.out‚146426.794921875 proc.cpu.threads‚17 disk.sda1.in‚0 disk.rbd3.in‚0) disk./.usagePercent‚37.91405839363915 disk.rbd1.out‚ 52.6904296875 disk.sda.in‚ 108.921875$ memory_percent‚82.78164863586426( proc.memory.availableMB‚ 2821.0546875 proc.memory.rssMB‚2804.875* proc.memory.percent‚17.119598388671875 disk.rbd0.out‚0 disk.dm-0.in‚48107.12890625 disk.drbd2165.out‚0 disk.rbd1.in‚ 0.52392578125" disk.dm-0.out‚146426.794921875 disk.sda3.in‚ 108.921875' disk.nvme0c0n1.out‚146426.794921875 disk.sda2.out‚0 disk.dm-1.out‚ 1282.79296875 cpu‚25.586183337039348$ disk./.usageGB‚564.9468688964844& disk.rbd0.in‚1.759218604441423e+13 disk.dm-1.in‚ 108.9218752éUvžƒ1‚(:“ “ÒéÒã’÷ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚288.679' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚57.73579999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0\üÍùA„1‚(:¶ “ÒéÒú¼“ú disk.dm-1.out‚ 1283.203125 disk.rbd0.out‚0 disk.rbd3.in‚0 disk.sda2.out‚0 disk.rbd1.in‚ 0.52392578125 disk.sda.out‚ 1283.203125 disk.drbd2165.out‚0 network.sent‚ 2.3523332e+07 disk.sda1.in‚0 disk.sda3.in‚ 108.921875" disk.nvme0n1.in‚48107.34765625 disk.rbd3.out‚ 0.4951171875 proc.memory.rssMB‚2804.875& disk.rbd0.in‚1.759218604441423e+13* proc.memory.percent‚17.119598388671875) disk./.usagePercent‚37.91436048252955$ disk.nvme0c0n1.in‚48107.34765625' disk.nvme0c0n1.out‚146440.736328125 disk.dm-1.in‚ 108.921875 network.recv‚ 5.027542e+06$ disk./.usageGB‚564.9513702392578% disk.nvme0n1.out‚146440.736328125 disk.rbd1.out‚ 52.7685546875 disk.sda.in‚ 108.921875" disk.dm-0.out‚146440.736328125) proc.memory.availableMB‚ 2821.22265625 disk.dm-0.in‚48107.34765625 disk.sda1.out‚0 disk.sda3.out‚ 1283.203125 disk.drbd2165.in‚0$ memory_percent‚82.78062343597412 cpu‚25.586019062912428 proc.cpu.threads‚17 disk.sda2.in‚0Š±s4…1/ _wandbruntime‚20222  _runtime‚20222¾æîôˆ1Å $ _timestamp‚1.784310037819947e+09 _runtime‚20222.037144883 _step‚1002 ' lr-AdamW/pg1‚0.00023697328283193232 ' lr-AdamW/pg2‚0.00023697328283193232 trainer/global_step‚12199"Ì!&Ї1Ê _runtime‚20222.037144883 _step‚1002 ' lr-AdamW/pg1‚0.00023697328283193232 ' lr-AdamW/pg2‚0.00023697328283193232 trainer/global_step‚12199 $ _timestamp‚1.784310037819947e+09ê µ4ˆ1/ _wandbruntime‚20222  _runtime‚20222 –þ&÷‰1P S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3518011487055360 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚3506535826481152 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11477005963776 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚255 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚961766793049600 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3516478844478712 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11475660574208 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 104077745 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3506535826481152 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14797132498432 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚31296344 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 209727033 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 105649288 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 _runtime‚20222.038386146 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11456024321014 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 104077751 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚3505021450129408 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _step‚1003 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚65874670 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚260 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 104077751 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VADeviceStatsMonitor.on_train_batch_start/requesteèÊÏqd_bytes.all.freed‚3516477474450422 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚31296339 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚65874410 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚3518011487055360 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 105659261 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3518012858618368 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚946969292460032 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚32439296 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚3505021475557376 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11475660574208 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3518012858618368 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚961766392519168 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 trainer/global_step‚12199 % _timestamp‚1.7843100378210425e+09 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚34578326 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚34578071 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14797100059136 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11457368921336 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 104077745 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3506535852654592 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 209737012 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3506535852654592 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 209737012 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3518011487055360 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚3506535826481152 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11477005963776 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚255 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚961766793049600 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3516478844478712 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11475660574208 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 104077745 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3506535826481152 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14797132498432 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚31296344 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 209727033 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 105649288 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 _runtime‚20222.038386146 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11456024321014 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 104077751 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚3505021450129408 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _step‚1003 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚65874670 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚260 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 104077751 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚3516477474450422 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚31296339 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚65874410 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚3518011487055360 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 105659261 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3518012858618368 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008ëö¢ùØ4‹1/ _wandbruntime‚20223  _runtime‚20223<ň 'Œ1šN H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 trainer/global_step‚12199 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 104085951 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚3506813827211264 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11476585609728 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 _runtime‚20223.424023079 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚251 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3518291784391168 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 104085957 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚31298882 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚961845482316288 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚65879728 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚31298887 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14798335357440 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3518291784391168 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 105657766 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3505299357335552 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚34581092 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚947047146958848 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3505299331907584 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11476585609728 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 209743717 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 104085957 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚32432128 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11477931006464 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 400523264 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3516757649723246 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 209753710 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 105667753 _step‚1004 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3506813853384704 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 105667753 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚65879979 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚34580846 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11456947787260 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3518290412820992 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3506813827211264 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11458292387694 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 % _timestamp‚1.7843100392065506e+09 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987@ñO1ŸN D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 209743717 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11477931006464 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 400523264 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3516757649723246 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 209753710 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 105667753 _step‚1004 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3506813853384704 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 105667753 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚65879979 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚34580846 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11456947787260 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3518290412820992 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3506813827211264 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11458292387694 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 % _timestamp‚1.7843100392065506e+09 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 trainer/global_step‚12199 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 104085951 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚3506813827211264 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11476585609728 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 _runtime‚20223.424023079 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚251 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3518291784391168 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 104085957 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚31298882 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚961845482316288 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚65879728 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚31298887 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14798335357440 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3518291784391168 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 105657766 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3505299357335552 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚34581092 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚947047146958848 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3505299331907584 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11476585609728 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 209743717 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 104085957 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚32432128 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F_Å1Ž /loss/FBt1/detection_loss‚0.4658992028871209 /loss/FBt2/detection_loss‚0.2882892841497793 trainer/global_step‚12199 % _timestamp‚1.7843100392932284e+09 1loss/FRt2/assignment_loss‚0.26449868355345163 0loss/FRt1/detection_loss‚0.45834569358120103 _runtime‚20223.510163707 3loss/SRqqt2/assignment_loss‚0.26461650788825836 _step‚1005 0loss/FRt1/assignment_loss‚1.2061536859911541 4loss/FBt2/assignment_loss‚0.00013756827983607282 /loss/FRt2/detection_loss‚1.3976027385152965 2loss/SRqqt2/detection_loss‚0.48349488719934774 %loss/total_loss‚5.659793782956704 epoch‚30 3loss/SRqqt1/assignment_loss‚0.18444972110629743 2loss/FBt1/assignment_loss‚0.005069782378905104 1loss/SRqqt1/detection_loss‚0.6412360274260568èƘ1“ % _timestamp‚1.7843100392932284e+09 1loss/FRt2/assignment_loss‚0.26449868355345163 0loss/FRt1/detection_loss‚0.45834569358120103 _runtime‚20223.510163707 3loss/SRqqt2/assignment_loss‚0.26461650788825836 _step‚1005 0loss/FRt1/assignment_loss‚1.2061536859911541 4loss/FBt2/assignment_loss‚0.00013756827983607282 /loss/FRt2/detection_loss‚1.3976027385152965 2loss/SRqqt2/detection_loss‚0.48349488719934774 %loss/total_loss‚5.659793782956704 epoch‚30 3loss/SRqqt1/assignment_loss‚0.18444972110629743 2loss/FBt1/assignment_loss‚0.005069782378905104 1loss/SRqqt1/detection_loss‚0.6412360274260568 /loss/FBt1/detection_loss‚0.4658992028871209 /loss/FBt2/detection_loss‚0.2882892841497793 trainer/global_step‚12199í×6ú”‘1‚(:‰ ¢ÒéÒ‹éêö gpu.0.gpu‚93.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚339.166' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚67.8332 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0]M¨á=’1‚(:² ¢ÒéÒ´½Þù disk.rbd1.out‚ 52.7685546875 disk.sda.in‚ 108.921875 disk.sda3.out‚ 1283.4375 disk.sda2.out‚0 disk.rbd0.out‚0 network.recv‚ 5.031166e+06 disk.sda2.in‚0 disk.rbd1.in‚ 0.52392578125 disk.sda1.in‚0# disk./.usageGB‚564.944263458252" disk.nvme0n1.in‚48107.55078125 disk.sda.out‚ 1283.4375# disk.dm-0.out‚146452.9970703125* proc.memory.percent‚17.119598388671875 disk.drbd2165.out‚0 disk.rbd3.out‚ 0.4951171875$ disk.nvme0c0n1.in‚48107.55078125( disk.nvme0c0n1.out‚146452.9970703125& disk.nvme0n1.out‚146452.9970703125 disk.dm-1.in‚ 108.921875 disk.dm-0.in‚48107.55078125$ memory_percent‚82.78241157531738 disk.drbd2165.in‚0 disk.dm-1.out‚ 1283.4375( proc.memory.availableMB‚ 2820.9296875 proc.memory.rssMB‚2804.875 cpu‚25.585882399366053 proc.cpu.threads‚17 disk.rbd3.in‚0& disk.rbd0.in‚1.759218604441423e+13 disk.sda1.out‚0 network.sent‚ 2.3528419e+07* disk./.usagePercent‚37.913883540493266 disk.sda3.in‚ 108.921875&bÔiž“1‚(:“ ±ÒéÒùý÷ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚261.525' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚52.30499999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0÷ÍÞþL”1‚(:Á ±ÒéÒªÿ¡ù" disk.nvme0n1.in‚48285.35546875 disk.rbd0.out‚0 disk.sda.out‚ 1283.70703125* proc.memory.percent‚17.119598388671875 disk.sda3.out‚ 1283.70703125' disk.nvme0n1.out‚146466.41357421875 disk.sda2.out‚0 disk.rbd1.out‚ 52.8310546875$ memory_percent‚82.78248310089111 cpu‚25.585721468559832 proc.cpu.threads‚17 disk.sda3.in‚ 108.921875 disk.drbd2165.in‚0) disk./.usagePercent‚37.91434640211517$ disk./.usageGB‚564.9511604309082 disk.dm-1.out‚ 1283.70703125 disk.rbd3.out‚ 0.4951171875 disk.sda2.in‚0 disk.rbd1.in‚ 0.52392578125 proc.memory.rssMB‚2804.875 disk.dm-0.in‚48285.35546875$ disk.dm-0.out‚146466.41357421875 disk.sda1.in‚0$ disk.nvme0c0n1.in‚48285.35546875 network.recv‚ 5.035556e+06 disk.sda.in‚ 108.921875 disk.dm-1.in‚ 108.921875& disk.rbd0.in‚1.759218604441423e+13 disk.drbd2165.out‚0 disk.rbd3.in‚0) disk.nvme0c0n1.out‚146466.41357421875) proc.memory.availableMB‚ 2820.91796875 network.sent‚ 2.359196e+07 disk.sda1.out‚0ù:'ž•1‚(:“ ÀÒéÒÃèãö gpu.0.gpu‚97.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚345.878' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚69.17559999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0VžÁfH–1‚(:½ ÀÒéÒ‡Ù´ù! disk.nvme0n1.in‚ 48293.0234375 disk.sda2.in‚0$ memory_percent‚82.78393745422363$ disk./.usageGB‚564.9557876586914 disk.rbd3.in‚0 disk.sda.out‚ 1283.93359375( proc.memory.availableMB‚ 2820.6796875* proc.memory.percent‚17.119598388671875 disk.rbd1.out‚ 52.8310546875 proc.memory.rssMB‚2804.875 cpu‚25.58558374854239# disk.nvme0c0n1.in‚ 48293.0234375 disk.dm-0.in‚ 48293.0234375 disk.sda3.out‚ 1283.93359375 proc.cpu.threads‚17 network.sent‚ 2.3597037e+07 disk.dm-1.out‚ 1283.93359375& disk.rbd0.in‚1.759218604441423e+13 disk.rbd0.out‚0 disk.dm-1.in‚ 108.921875 disk.drbd2165.out‚0 disk.sda.in‚ 108.921875) disk./.usagePercent‚37.91465693925421 disk.sda1.out‚0 disk.sda3.in‚ 108.921875 disk.drbd2165.in‚0' disk.nvme0n1.out‚146477.02197265625 disk.sda2.out‚0 disk.rbd1.in‚ 0.52392578125 network.recv‚ 5.038859e+06$ disk.dm-0.out‚146477.02197265625 disk.sda1.in‚0 disk.rbd3.out‚ 0.4951171875) disk.nvme0c0n1.out‚146477.02197265625Ž8òÛ”—1‚(:‰ ÏÒéÒ䎳ø gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚268.238' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.6476 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ŠŽE˜1‚(:º ÏÒéÒá™ù disk.rbd1.in‚ 0.52392578125 disk.dm-1.in‚ 108.921875 disk.dm-1.out‚ 1284.2265625& disk.rbd0.in‚1.759218604441423e+13 disk.drbd2165.out‚0 disk.rbd3.in‚0# disk.nvme0c0n1.in‚ 48293.4453125( proc.memory.availableMB‚ 2820.6796875 network.sent‚ 2.3602176e+07$ disk./.usageGB‚564.9836120605469 disk.sda.in‚ 108.921875 disk.sda.out‚ 1284.2265625 network.recv‚ 5.04206e+06 disk.sda2.in‚0* proc.memory.percent‚17.119598388671875 cpu‚25.585422578783387 disk.dm-0.in‚ 48293.4453125$ disk.dm-0.out‚146486.59033203125 disk.sda2.out‚0 disk.rbd0.out‚0 disk.sda3.in‚ 108.921875 disk.sda3.out‚ 1284.2265625) disk.nvme0c0n1.out‚146486.59033203125$ memory_percent‚82.78393745422363 proc.cpu.threads‚17 disk.rbd3.out‚ 0.4951171875! disk.nvme0n1.in‚ 48293.4453125) disk./.usagePercent‚37.91652425820896 disk.sda1.in‚0 disk.sda1.out‚0 proc.memory.rssMB‚2804.875 disk.drbd2165.in‚0' disk.nvme0n1.out‚146486.59033203125 disk.rbd1.out‚ 52.8310546875j»•™1‚(:Š ÞÒéÒ½úåö gpu.0.gpu‚100.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚210.352' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚42.0704 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0/””¼Gš1‚(:¼ ÞÒéÒü°¡ù network.sent‚ 2.3606723e+07 disk.rbd3.out‚ 0.4951171875$ disk.nvme0c0n1.in‚48294.25390625 disk.sda2.out‚0 disk.sda.in‚ 108.921875 cpu‚25.58528553625799 disk.rbd0.out‚0 disk.drbd2165.out‚0) disk.nvme0c0n1.out‚146504.71826171875$ memory_percent‚82.78341293334961$ disk.dm-0.out‚146504.71826171875 disk.sda3.out‚ 1285.1953125 disk.drbd2165.in‚0" disk.nvme0n1.in‚48294.25390625 disk.sda1.out‚0 disk.sda3.in‚ 108.921875' disk.nvme0n1.out‚146504.71826171875 network.recv‚ 5.045339e+06$ disk./.usageGB‚564.9936561584473& disk.rbd0.in‚1.759218604441423e+13 disk.dm-0.in‚48294.25390625 proc.memory.rssMB‚2804.875 proc.cpu.threads‚17 disk.dm-1.in‚ 108.921875 disk.sda1.in‚0 disk.sda.out‚ 1285.1953125) disk./.usagePercent‚37.91719832604661 disk.sda2.in‚0* proc.memory.percent‚17.119598388671875 disk.dm-1.out‚ 1285.1953125 disk.rbd3.in‚0 disk.rbd1.in‚ 0.52392578125 disk.rbd1.out‚ 52.9716796875' proc.memory.availableMB‚ 2820.765625µ[Í©4›1/ _wandbruntime‚20300  _runtime‚20300‡¦0AÊœ1Ä & lr-AdamW/pg1‚0.0002343455750031441 & lr-AdamW/pg2‚0.0002343455750031441 trainer/global_step‚12249 % _timestamp‚1.7843101167084928e+09 _runtime‚20300.925775634 _step‚1006´S® Ï1É & lr-AdamW/pg1‚0.0002343455750031441 & lr-AdamW/pg2‚0.0002343455750031441 trainer/global_step‚12249 % _timestamp‚1.7843101167084928e+09 _runtime‚20300.925775634 _step‚1006îèjêý4ž1/ _wandbruntime‚20300  _runtime‚20300eÔúB(Ÿ1P YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 400530432 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚3520435889160192 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 104488045 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 _runtime‚20300.927338704 _step‚1007 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 106085013 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚3520435862986752 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚34714424 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 trainer/global_step‚12249 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3530419170500620 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚3518915564466176 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚245 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚950843612844032 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 104488045 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11521976504832 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚31423489 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚34714664 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚3530417800472330 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3531957839491584 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 106075040 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚31423494 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚965700796760576 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14857216355840 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 106075040 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚32439296 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3531959211054592 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3531959211054592 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚965701197291008 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 104488051 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚66137913 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11503606034444 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚3518915539038208 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008y=(Ð" 1•P ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚3520435862986752 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚34714424 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 trainer/global_step‚12249 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3530419170500620 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚3518915564466176 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚245 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚950843612844032 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 104488045 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11521976504832 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚31423489 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚34714664 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚3530417800472330 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3531957839491584 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 106075040 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚31423494 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚965700796760576 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14857216355840 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 106075040 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚32439296 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3531959211054592 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3531959211054592 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚965701197291008 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 104488051 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚66137913 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11503606034444 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚3518915539038208 C:DeviceStatsMonitor.on_train_batch_start/active.all.currenüî¹_Kt‚9979 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 400530432 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚3520435889160192 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 104488045 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 _runtime‚20300.927338704 _step‚1007 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 106085013ïS[U54¡1/ _wandbruntime‚20302  _runtime‚20302wÖ˜Ã'¢1˜N OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚66143325 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚238 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚66143087 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚950921467342848 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 _step‚1008 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3519193420816384 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚950921099251712 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14858385717760 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 _runtime‚20302.36314769 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 210589762 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 104496257 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 104496257 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 210579769 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚32432128 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3532238136827392 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11503184900368 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14858418149888 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3532236765257216 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11522901540352 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 106093505 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚31426037 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 400523264 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11504529500802 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 210589762 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚3520713889890304 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11524246937088 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚965779484969472 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 106083518 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3520713889890304 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11522901540352 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 trainer/global_step‚12249 $ _timestamp‚1.784310118145636e+09 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚31426032 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚3520713863716864 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11524246937088 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3519193446244352 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚66143087 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚950921467342848 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 _step‚1008 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3519193420816384 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚950921099251712 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14858385717760 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 _runtime‚20302.36314769 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 210589762 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 104496257 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 104496257 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 210579769 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚32432128 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3532238136827392 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11503184900368 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14858418149888 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3532236765257216 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11522901540352 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 106093505 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚31426037 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 400523264 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11504529500802 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 210589762 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚3520713889890304 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11524246937088 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚965779484969472 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 106083518 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3520713889890304 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11522901540352 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 trainer/global_step‚12249 $ _timestamp‚1.784310118145636e+09 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚31426032 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚3520713863716864 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11524246937088 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3519193446244352 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0ðjra4¤1/ _wandbruntime‚20302  _runtime‚20302WXÁƒ¥1‰ 1loss/SRqqt1/detection_loss‚0.5611069464706625 %loss/total_loss‚5.954039769226604 3loss/SRqqt1/assignment_loss‚0.17800536151204646 /loss/FRt1/detection_loss‚0.4972201525504731 .loss/FRt2/detection_loss‚1.275129028462457 1loss/SRqqt2/detection_loss‚0.5078364510600666 /loss/FBt2/detection_loss‚0.3251352110294192 _runtime‚20302.419756357 3loss/FBt1/assignment_loss‚0.0005088132616457575 4loss/FBt2/assignment_loss‚0.00024287868920236377 % _timestamp‚1.7843101182025478e+09 epoch‚30 .loss/FRt2/assignment_loss‚0.26617357845766 3loss/SRqqt2/assignment_loss‚0.26053364602677603 /loss/FBt1/detection_loss‚1.0213987869228731 trainer/global_step‚12249 _step‚1009 0loss/FRt1/assignment_loss‚1.0607489147833222KƒI·¦1Ž 0loss/FRt1/assignment_loss‚1.0607489147833222 1loss/SRqqt1/detection_loss‚0.5611069464706625 %loss/total_loss‚5.954039769226604 3loss/SRqqt1/assignment_loss‚0.17800536151204646 /loss/FRt1/detection_loss‚0.4972201525504731 .loss/FRt2/detection_loss‚1.275129028462457 1loss/SRqqt2/detection_loss‚0.5078364510600666 /loss/FBt2/detection_loss‚0.3251352110294192 _runtime‚20302.419756357 3loss/FBt1/assignment_loss‚0.0005088132616457575 4loss/FBt2/assignment_loss‚0.00024287868920236377 % _timestamp‚1.7843101182025478e+09 epoch‚30 .loss/FRt2/assignment_loss‚0.26617357845766 3loss/SRqqt2/assignment_loss‚0.26053364602677603 /loss/FBt1/detection_loss‚1.0213987869228731 trainer/global_step‚12249 _step‚1009ñ>ºpŸ§1‚(:” íÒéÒËÛ¸ø gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚258.298' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚51.659600000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ï{é@¨1‚(:µ íÒéÒ¡é®ù( disk./.usagePercent‚37.9176079381014$ disk./.usageGB‚564.9997596740723 disk.rbd0.out‚0 disk.sda3.in‚ 108.921875 disk.sda2.out‚0 disk.sda3.out‚ 1287.015625 disk.rbd1.in‚ 0.52392578125 disk.rbd1.out‚ 52.9716796875 cpu‚25.585136286616564 network.sent‚ 2.3612378e+07 disk.drbd2165.in‚0 disk.rbd3.out‚ 0.4951171875) proc.memory.availableMB‚ 2821.02734375 proc.memory.rssMB‚2804.875 disk.dm-1.in‚ 108.921875 disk.nvme0n1.in‚ 48295.109375 disk.sda2.in‚0$ memory_percent‚82.78181552886963 network.recv‚ 5.048365e+06 disk.dm-1.out‚ 1287.015625$ disk.dm-0.out‚146525.37939453125 disk.sda1.out‚0 disk.rbd3.in‚0) disk.nvme0c0n1.out‚146525.37939453125 disk.sda.in‚ 108.921875 disk.sda.out‚ 1287.015625& disk.rbd0.in‚1.759218604441423e+13 disk.dm-0.in‚ 48295.109375 disk.drbd2165.out‚0' disk.nvme0n1.out‚146525.37939453125 proc.cpu.threads‚17" disk.nvme0c0n1.in‚ 48295.109375* proc.memory.percent‚17.119598388671875 disk.sda1.in‚0RÁh©1‚(:’ üÒéÒåºðö gpu.0.gpu‚22.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚330.288' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚66.05760000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0~w£‡9ª1‚(:® üÒéÒåçÏù cpu‚25.58499910737713 disk.dm-1.out‚ 1290.6640625 disk.rbd3.out‚ 0.4951171875 proc.cpu.threads‚17 disk.sda3.out‚ 1290.6640625) disk.nvme0c0n1.out‚146541.88037109375 disk.rbd1.in‚ 0.52392578125* proc.memory.percent‚17.119598388671875 disk.sda.out‚ 1290.6640625 disk.rbd0.out‚0 disk.drbd2165.in‚0' disk.nvme0n1.out‚146541.88037109375 disk.sda2.out‚0 proc.memory.rssMB‚2804.875 disk.nvme0n1.in‚ 48390.625 disk.sda1.out‚0 network.recv‚ 5.052949e+06 disk.dm-0.in‚ 48390.625 disk.rbd3.in‚0$ memory_percent‚82.78658390045166) proc.memory.availableMB‚ 2820.24609375 network.sent‚ 2.367524e+07) disk./.usagePercent‚37.91787879407262 disk.sda.in‚ 108.921875 disk.dm-1.in‚ 108.921875 disk.rbd1.out‚ 53.0341796875$ disk./.usageGB‚565.0037956237793 disk.nvme0c0n1.in‚ 48390.625 disk.sda2.in‚0 disk.sda1.in‚0& disk.rbd0.in‚1.759218604441423e+13$ disk.dm-0.out‚146541.88037109375 disk.sda3.in‚ 108.921875 disk.drbd2165.out‚0`Ì]Ô”«1‚(:‰ ‹ÓéÒºŠ‹÷ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚260.181' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.0362 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0YûC¬1‚(:¸ ‹ÓéÒå®Ãù disk.sda2.out‚0 network.recv‚ 5.056259e+06 disk.sda3.in‚ 108.921875 disk.sda.out‚ 1291.2421875 cpu‚25.584826265480558 disk.sda1.in‚0 disk.drbd2165.out‚0 disk.rbd1.out‚ 53.0341796875 disk.rbd3.out‚ 0.4951171875 disk.sda2.in‚0 disk.dm-1.in‚ 108.921875$ memory_percent‚82.78660774230957 disk.sda3.out‚ 1291.2421875) disk./.usagePercent‚37.91816065836782 disk.rbd0.out‚0 disk.dm-0.in‚ 48390.828125 disk.dm-1.out‚ 1291.2421875 proc.memory.rssMB‚2804.875$ disk.dm-0.out‚146555.46630859375' disk.nvme0n1.out‚146555.46630859375( proc.memory.availableMB‚ 2820.2421875 proc.cpu.threads‚17 disk.drbd2165.in‚0 network.sent‚ 2.3680905e+07$ disk./.usageGB‚565.0079956054688& disk.rbd0.in‚1.759218604441423e+13 disk.nvme0n1.in‚ 48390.828125 disk.rbd1.in‚ 0.52392578125 disk.sda.in‚ 108.921875* proc.memory.percent‚17.119598388671875 disk.sda1.out‚0 disk.rbd3.in‚0" disk.nvme0c0n1.in‚ 48390.828125) disk.nvme0c0n1.out‚146555.466308593752¡9.”­1‚(:‰ šÓéÒ…úö gpu.0.gpu‚61.0 gpu.0.memory‚20, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚333.229' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚66.6458 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Osd:®1‚(:¯ šÓéÒÀÂâù# disk.nvme0n1.out‚146566.1953125 disk.sda2.in‚0 disk.dm-1.out‚ 1292.02734375 proc.memory.rssMB‚2804.875* proc.memory.percent‚17.119598388671875 network.sent‚ 2.3686069e+07 network.recv‚ 5.060077e+06 disk.dm-0.out‚146566.1953125 disk.rbd3.in‚0 disk.sda.out‚ 1292.02734375' proc.memory.availableMB‚ 2820.734375 disk.sda1.out‚0 disk.sda3.in‚ 108.921875 disk.rbd1.in‚ 0.52392578125 disk.sda.in‚ 108.921875* disk./.usagePercent‚37.918535965413014 disk.dm-0.in‚ 48391.046875 disk.drbd2165.in‚0 disk.nvme0n1.in‚ 48391.046875 disk.sda2.out‚0 disk.rbd1.out‚ 53.1123046875 disk.rbd0.out‚0$ memory_percent‚82.78360366821289 disk.rbd3.out‚ 0.4951171875& disk.rbd0.in‚1.759218604441423e+13 disk.sda3.out‚ 1292.02734375" disk.nvme0c0n1.in‚ 48391.046875 disk.dm-1.in‚ 108.921875 disk.drbd2165.out‚0% disk.nvme0c0n1.out‚146566.1953125 cpu‚25.584701837632267 proc.cpu.threads‚17$ disk./.usageGB‚565.0135879516602 disk.sda1.in‚0z©(ž¯1‚(:“ ©ÓéÒ¿Á¶÷ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚269.376' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚53.87519999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0íµ5)9°1‚(:® ©ÓéÒƃµù disk.sda3.out‚ 1294.01953125! disk.dm-0.out‚146586.65234375 network.recv‚ 5.063239e+06 disk.drbd2165.in‚0 disk.rbd3.in‚0 disk.rbd3.out‚ 0.4951171875 disk.dm-1.out‚ 1294.01953125$ memory_percent‚82.78665542602539 disk.sda1.out‚0 disk.nvme0n1.in‚ 48391.46875$ disk.nvme0n1.out‚146586.65234375 disk.sda.in‚ 108.921875 disk.sda1.in‚0! disk.nvme0c0n1.in‚ 48391.46875 disk.rbd1.out‚ 53.1123046875 proc.memory.rssMB‚2804.875& disk.nvme0c0n1.out‚146586.65234375 disk.sda2.in‚0 network.sent‚ 2.3691187e+07* disk./.usagePercent‚37.919201585002035 disk.drbd2165.out‚0 disk.sda2.out‚0 disk.rbd1.in‚ 0.52392578125 disk.sda.out‚ 1294.01953125 disk.rbd0.out‚0 disk.dm-0.in‚ 48391.46875$ disk./.usageGB‚565.0235061645508 disk.dm-1.in‚ 108.921875& disk.rbd0.in‚1.759218604441423e+13' proc.memory.availableMB‚ 2820.234375* proc.memory.percent‚17.119598388671875 cpu‚25.58453014018229 proc.cpu.threads‚17 disk.sda3.in‚ 108.9218752 Ã4±1/ _wandbruntime‚20379  _runtime‚20379©G$ŽË²1Å trainer/global_step‚12299 $ _timestamp‚1.784310195576677e+09 _runtime‚20379.794076966 _step‚1010 ' lr-AdamW/pg1‚0.00023172503366260377 ' lr-AdamW/pg2‚0.00023172503366260377V­äг1Ê ' lr-AdamW/pg1‚0.00023172503366260377 ' lr-AdamW/pg2‚0.00023172503366260377 trainer/global_step‚12299 $ _timestamp‚1.784310195576677e+09 _runtime‚20379.794076966 _step‚1010òo‘ —4´1/ _wandbruntime‚20379  _runtime‚20379Ê^òµ1P XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚31550644 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3534335899492352 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11568292435456 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚969635052463616 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 trainer/global_step‚12299 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3545904191927808 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚3534335925665792 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚34850903 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11569637825024 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 104898351 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚3544358126494238 _runtime‚20379.798919638 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 104898351 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚256 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14917119235584 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚31550639 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚251 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14917153772032 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 211399137 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 106510765 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚954718301319168 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚3534335899492352 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚66401542 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚1011 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 $ _timestamp‚1.784310195578018e+09 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚3545904191927808 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3534335899492352 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11568292435456 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚969635052463616 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 trainer/global_step‚12299 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3545904191927808 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚3534335925665792 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚34850903 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11569637825024 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 104898351 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚3544358126494238 _runtime‚20379.798919638 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 104898351 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚256 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14917119235584 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚31550639 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚251 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14917153772032 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 211399137 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 106510765 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚954718301319168 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚3534335899492352 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚66401542 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚1011 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 $ _timestamp‚1.784310195578018e+09 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚3545904191927808 JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 % _timestamp‚1.7843101969639719e+09 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚31553187 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3534613926395904 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚34853439 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚969714137804288 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3546183117693440 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3533087535153152 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11550766613910 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 106519257 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚246 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚954795787726848 trainer/global_step‚12299 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11569217470976 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 _step‚1012 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 211425814 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚3534613926395904 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3534613900222464 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 106519257 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚34853685 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11549422013476 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 211415821 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚3534613900222464 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3546184489263616 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11570562867712 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11569217470976 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚251 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14918350077440 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 211425814 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3533087509725184 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11570562867712 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚969713735183872 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚66406872 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 _runtime‚20381.181373079 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 104906551 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 104906557 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3546184489263616 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚66406621 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0  u¹1ŸN A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚251 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14918350077440 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 211425814 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3533087509725184 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11570562867712 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚969713735183872 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚66406872 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 _runtime‚20381.181373079 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 104906551 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 104906557 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3546184489263616 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚66406621 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14918315548160 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 % _timestamp‚1.7843101969639719e+09 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚31553187 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3534613926395904 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚34853439 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚969714137804288 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3546183117693440 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3533087535153152 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11550766613910 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 106519257 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚246 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚954795787726848 trainer/global_step‚12299 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11569217470976 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 _step‚1012 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 211425814 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚3534613926395904 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3534613900222464 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 106519257 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚34853685 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11549422013476 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 211415821 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚3534613900222464 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3546184489263616 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11570562867712 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11569217470976 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648ôд\Ñ4º1/ _wandbruntime‚20381  _runtime‚203810ä»1 3loss/FBt1/assignment_loss‚7.918079401138455e-06 1loss/SRqqt2/detection_loss‚0.5417471991818517 _runtime‚20381.266440443 _step‚1013 0loss/FRt1/detection_loss‚0.44447802242476825 1loss/SRqqt1/detection_loss‚0.6030925738202552 /loss/FBt1/detection_loss‚0.5059829145503629 0loss/FRt1/assignment_loss‚1.1006508728201385 0loss/FRt2/assignment_loss‚0.3042807918478423 0loss/FBt2/detection_loss‚0.29653783136294104 %loss/total_loss‚5.667097026206033 3loss/SRqqt1/assignment_loss‚0.19365008227235406 3loss/FBt2/assignment_loss‚4.103779963109615e-05 /loss/FRt2/detection_loss‚1.4293420656612117 epoch‚30 trainer/global_step‚12299 % _timestamp‚1.7843101970495071e+09 3loss/SRqqt2/assignment_loss‚0.24728571638527455ióï¼1’ % _timestamp‚1.7843101970495071e+09 3loss/SRqqt2/assignment_loss‚0.24728571638527455 3loss/FBt1/assignment_loss‚7.918079401138455e-06 1loss/SRqqt2/detection_loss‚0.5417471991818517 _runtime‚20381.266440443 _step‚1013 0loss/FRt1/detection_loss‚0.44447802242476825 1loss/SRqqt1/detection_loss‚0.6030925738202552 /loss/FBt1/detection_loss‚0.5059829145503629 0loss/FRt1/assignment_loss‚1.1006508728201385 0loss/FRt2/assignment_loss‚0.3042807918478423 0loss/FBt2/detection_loss‚0.29653783136294104 %loss/total_loss‚5.667097026206033 3loss/SRqqt1/assignment_loss‚0.19365008227235406 3loss/FBt2/assignment_loss‚4.103779963109615e-05 /loss/FRt2/detection_loss‚1.4293420656612117 epoch‚30 trainer/global_step‚12299õš‚Õž½1‚(:“ ¸ÓéÒÖŽûö gpu.0.gpu‚99.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚339.941' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚67.98819999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0š/÷YN¾1‚(:à ¸ÓéÒäÈù disk.sda1.in‚0$ memory_percent‚82.78830051422119 proc.cpu.threads‚17 disk.sda.out‚ 1296.52734375 disk.dm-0.in‚48504.60546875" disk.nvme0n1.in‚48504.60546875 disk.sda2.out‚0 network.recv‚ 5.066815e+06 disk.sda.in‚ 108.921875 disk.dm-1.out‚ 1296.52734375$ disk.nvme0c0n1.in‚48504.60546875 cpu‚25.584394109207487 disk.dm-1.in‚ 108.921875 disk.drbd2165.in‚0 disk.sda2.in‚0* proc.memory.percent‚17.119598388671875$ disk./.usageGB‚565.0247611999512 disk.rbd0.out‚0 disk.sda3.out‚ 1296.52734375) disk.nvme0c0n1.out‚146603.03173828125' disk.nvme0n1.out‚146603.03173828125) proc.memory.availableMB‚ 2819.96484375 disk.sda1.out‚0 proc.memory.rssMB‚2804.875* disk./.usagePercent‚37.919285811480805 disk.rbd1.out‚ 53.1748046875& disk.rbd0.in‚1.759218604441423e+13 disk.sda3.in‚ 108.921875 disk.rbd3.in‚0 disk.rbd3.out‚ 0.4951171875 network.sent‚ 2.3696262e+07 disk.drbd2165.out‚0 disk.rbd1.in‚ 0.52392578125$ disk.dm-0.out‚146603.03173828125òCMeŸ¿1‚(:” ÇÓé҈Ƙ÷ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚269.062' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚53.812400000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0½\‹IÀ1‚(:¾ ÇÓéÒäÀªú disk.sda3.out‚ 1297.12109375 disk.drbd2165.in‚0 disk.rbd3.out‚ 0.4951171875$ disk.nvme0c0n1.in‚48504.82421875 disk.sda2.out‚0 disk.rbd1.in‚ 0.52392578125 disk.rbd0.out‚0 disk.sda1.out‚0 disk.rbd1.out‚ 53.1748046875 network.sent‚ 2.375956e+07 disk.sda.in‚ 108.921875 disk.dm-1.in‚ 108.921875& disk.rbd0.in‚1.759218604441423e+13 disk.sda1.in‚0 disk.drbd2165.out‚0) disk.nvme0c0n1.out‚146611.76318359375 disk.sda2.in‚0 disk.sda3.in‚ 108.921875 disk.rbd3.in‚0& proc.memory.availableMB‚ 2820.21875 proc.memory.rssMB‚2804.875 cpu‚25.584232225801706 disk.dm-0.in‚48504.82421875$ disk.dm-0.out‚146611.76318359375* proc.memory.percent‚17.119598388671875 proc.cpu.threads‚17 network.recv‚ 5.070881e+06" disk.nvme0n1.in‚48504.82421875' disk.nvme0n1.out‚146611.76318359375) disk./.usagePercent‚37.91975328123833 disk.sda.out‚ 1297.12109375$ memory_percent‚82.78675079345703$ disk./.usageGB‚565.0317268371582 disk.dm-1.out‚ 1297.12109375»¼w<”Á1‚(:‰ ÖÓéÒóÔàø gpu.0.gpu‚90.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚257.222' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚51.4444 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‚ü½?Â1‚(:´ ÖÓéÒ‰èéù disk.sda1.out‚0 disk.rbd3.in‚0 proc.memory.rssMB‚2804.875* disk./.usagePercent‚37.920198222332836& disk.rbd0.in‚1.759218604441423e+13 disk.drbd2165.out‚0 network.sent‚ 2.3764761e+07 disk.sda.in‚ 108.921875 disk.sda.out‚ 1297.8125 disk.dm-1.out‚ 1297.8125 disk.dm-0.in‚48505.24609375 disk.sda2.in‚0 disk.sda3.out‚ 1297.8125 disk.drbd2165.in‚0) disk.nvme0c0n1.out‚146628.52880859375$ disk./.usageGB‚565.0383567810059$ disk.dm-0.out‚146628.52880859375 disk.rbd3.out‚ 0.4951171875" disk.nvme0n1.in‚48505.24609375 disk.rbd1.in‚ 0.52392578125& proc.memory.availableMB‚ 2820.46875 proc.cpu.threads‚17 cpu‚25.584098322540154 disk.rbd0.out‚0 disk.sda3.in‚ 108.921875$ disk.nvme0c0n1.in‚48505.24609375' disk.nvme0n1.out‚146628.52880859375 disk.sda2.out‚0 disk.rbd1.out‚ 53.2529296875 network.recv‚ 5.074467e+06 disk.dm-1.in‚ 108.921875 disk.sda1.in‚0$ memory_percent‚82.78522491455078* proc.memory.percent‚17.119598388671875âM…ô”Ã1‚(:‰ åÓéÒù×÷ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚39.0 gpu.0.powerWatts‚135.619' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚27.1238 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ï=7@Ä1‚(:µ åÓéÒÇ«¯ù$ memory_percent‚82.78672695159912* proc.memory.percent‚17.119598388671875 network.recv‚ 5.077696e+06( disk.nvme0c0n1.out‚146637.6884765625& disk.nvme0n1.out‚146637.6884765625 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda.in‚ 108.921875 proc.cpu.threads‚17# disk./.usageGB‚565.044506072998 disk.drbd2165.out‚0" disk.nvme0n1.in‚48505.44921875) proc.memory.availableMB‚ 2820.22265625 disk.rbd3.in‚0 disk.sda2.in‚0 disk.rbd1.in‚ 0.52392578125 network.sent‚ 2.3769355e+07 disk.sda3.in‚ 108.921875 disk.rbd3.out‚ 0.4951171875 disk.sda2.out‚0 disk.dm-1.in‚ 108.921875& disk.rbd0.in‚1.759218604441423e+13 proc.memory.rssMB‚2804.875 disk.drbd2165.in‚0 disk.rbd1.out‚ 53.2529296875 disk.dm-1.out‚ 1298.15625 disk.dm-0.in‚48505.44921875 cpu‚25.583940041173395# disk.dm-0.out‚146637.6884765625) disk./.usagePercent‚37.92061090647803 disk.sda3.out‚ 1298.15625$ disk.nvme0c0n1.in‚48505.44921875 disk.sda.out‚ 1298.15625 disk.rbd0.out‚0û²í’Å1‚(:‡ ôÓéÒë߉÷ gpu.0.gpu‚90.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚269.59' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.918 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0”¯—=Æ1‚(:² ôÓéÒáÅæù disk.sda.in‚ 108.921875 disk.sda.out‚ 1298.53125" disk.dm-0.out‚146648.111328125 cpu‚25.583816355916035 disk.sda1.in‚0 disk.sda3.in‚ 108.921875 disk.sda3.out‚ 1298.53125 disk.rbd3.in‚0 disk.rbd3.out‚ 0.4951171875$ memory_percent‚82.78577327728271 disk.sda2.out‚0 disk.dm-1.in‚ 108.921875 disk.rbd0.out‚0) proc.memory.availableMB‚ 2820.37890625 proc.cpu.threads‚17$ disk./.usageGB‚565.0513648986816 disk.sda2.in‚0 disk.rbd1.in‚ 0.52392578125 disk.rbd1.out‚ 53.3154296875$ disk.nvme0c0n1.in‚48610.58984375) disk./.usagePercent‚37.92107120802459& disk.rbd0.in‚1.759218604441423e+13 disk.drbd2165.out‚0" disk.nvme0n1.in‚48610.58984375% disk.nvme0n1.out‚146648.111328125 proc.memory.rssMB‚2804.875 network.recv‚ 5.081289e+06 disk.drbd2165.in‚0' disk.nvme0c0n1.out‚146648.111328125 network.sent‚ 2.377496e+07 disk.dm-1.out‚ 1298.53125* proc.memory.percent‚17.119598388671875 disk.dm-0.in‚48610.58984375 disk.sda1.out‚0[“ü4Ç1/ _wandbruntime‚20458  _runtime‚20458£_=€ÉÈ1à % _timestamp‚1.7843102740401673e+09 _runtime‚20458.25734237 _step‚1014 & lr-AdamW/pg1‚0.0002291118271227853 & lr-AdamW/pg2‚0.0002291118271227853 trainer/global_step‚12349õ‡a]ÎÉ1È & lr-AdamW/pg1‚0.0002291118271227853 & lr-AdamW/pg2‚0.0002291118271227853 trainer/global_step‚12349 % _timestamp‚1.7843102740401673e+09 _runtime‚20458.25734237 _step‚1014öÛ¢öO4Ê1/ _wandbruntime‚20458  _runtime‚20458:»°²(Ë1P O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 105308645 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11614608366080 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚31677794 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚31677789 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚3548235962171392 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚34986442 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚242 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚973569449060864 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 105308645 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3548235962171392 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11594735660338 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚958592253612032 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14977227888128 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14977195448832 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚3558298452516146 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11614608366080 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 105308651 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 106926544 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚973569849591296 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚3546703742283776 _step‚1015 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 106936517 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3559850544364032 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3559851915927040 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 % _timestamp‚1.7843102740412805e+09 _runtime‚20458.258579215 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚34986200 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚32439296 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚3546703716855808 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 trainer/global_step‚12349 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3559851915927040 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚3559850544364032 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11615953755648 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚66663989 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 105308651 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 212235189 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3559850544364032 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3559851915927040 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 % _timestamp‚1.7843102740412805e+09 _runtime‚20458.258579215 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚34986200 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚32439296 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚3546703716855808 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 trainer/global_step‚12349 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3559851915927040 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚3559850544364032 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11615953755648 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚66663989 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 105308651 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 212235189 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚242 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚973569449060864 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 105308645 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3548235962171392 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11594735660338 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚958592253612032 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14977227888128 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14977195448832 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚3558298452516146 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11614608366080 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 105308651 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 106926544 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚973569849591296÷éÁ¼4Í1/ _wandbruntime‚20459  _runtime‚20459®¾¦]'Î1™N UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3548513936728064 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 106935022 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 trainer/global_step‚12349 _runtime‚20459.643250795 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3548513962901504 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚31680337 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚973648142176768 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚3558577257760568 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11597003727018 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚230 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3560130841699840 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 106945009 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 212251873 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚34988948 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚66669515 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11595659126584 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 105316851 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11616878798336 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3546981624061952 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11616878798336 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 212261866 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14978402157056 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 105316857 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 106935022 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 _step‚1016 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚3548513936728064 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3560129470129664 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚958670108110848 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚34989178 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 105316851 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚3548513962901504 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚31680332 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚66669280 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11615533401600 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 106945009 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11615533401600Ì@Âh$'Ï1žN OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 106935022 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 _step‚1016 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚3548513936728064 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3560129470129664 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚958670108110848 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚34989178 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 105316851 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚3548513962901504 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚31680332 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚66669280 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11615533401600 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 106945009 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11615533401600 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3548513936728064 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 106935022 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 trainer/global_step‚12349 _runtime‚20459.643250795 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3548513962901504 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚31680337 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚973648142176768 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚3558577257760568 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11597003727018 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚230 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3560130841699840 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 106945009 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 212251873 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚34988948 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚66669515 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11595659126584 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 105316851 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11616878798336 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3546981624061952 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11616878798336 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 212261866 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14978402157056 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 105316857 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5ø’r‘4Ð1/ _wandbruntime‚20459  _runtime‚20459z›Ñ1 3loss/FBt2/assignment_loss‚5.576058716875199e-05 /loss/FRt2/detection_loss‚1.2977460793877558 1loss/SRqqt2/detection_loss‚0.5352910639072797 trainer/global_step‚12349 3loss/SRqqt1/assignment_loss‚0.23486075171706752 % _timestamp‚1.7843102755119839e+09 _step‚1017 0loss/FRt2/assignment_loss‚0.2845857470266281 3loss/SRqqt2/assignment_loss‚0.20426100972090028 4loss/FBt1/assignment_loss‚5.2933940193473785e-05 0loss/FRt1/detection_loss‚0.47442689127369725 /loss/FBt2/detection_loss‚0.3685709389883965 %loss/total_loss‚5.451128708258052 epoch‚30 _runtime‚20459.728928685 0loss/FRt1/assignment_loss‚1.1636865600370443 1loss/SRqqt1/detection_loss‚0.5730514326412873 /loss/FBt1/detection_loss‚0.3145395390306329 •×Ò1’ % _timestamp‚1.7843102755119839e+09 _step‚1017 0loss/FRt2/assignment_loss‚0.2845857470266281 3loss/SRqqt2/assignment_loss‚0.20426100972090028 4loss/FBt1/assignment_loss‚5.2933940193473785e-05 0loss/FRt1/detection_loss‚0.47442689127369725 /loss/FBt2/detection_loss‚0.3685709389883965 %loss/total_loss‚5.451128708258052 epoch‚30 _runtime‚20459.728928685 0loss/FRt1/assignment_loss‚1.1636865600370443 1loss/SRqqt1/detection_loss‚0.5730514326412873 /loss/FBt1/detection_loss‚0.3145395390306329 3loss/FBt2/assignment_loss‚5.576058716875199e-05 /loss/FRt2/detection_loss‚1.2977460793877558 1loss/SRqqt2/detection_loss‚0.5352910639072797 trainer/global_step‚12349 3loss/SRqqt1/assignment_loss‚0.23486075171706752ù•”Ó1‚(:‰ ƒÔéÒ˜€è÷ gpu.0.gpu‚70.0 gpu.0.memory‚21, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚38.0 gpu.0.powerWatts‚110.527' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.1054 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Â| "IÔ1‚(:¾ ƒÔéÒõĹù' disk.nvme0n1.out‚146660.97705078125 disk.sda.out‚ 1298.9609375 disk.dm-1.out‚ 1298.9609375 disk.sda3.out‚ 1298.9609375) proc.memory.availableMB‚ 2819.89453125 proc.memory.rssMB‚2804.875 proc.cpu.threads‚17) disk.nvme0c0n1.out‚146660.97705078125 disk.rbd0.out‚0 disk.drbd2165.out‚0$ disk.nvme0c0n1.in‚48610.80859375" disk.nvme0n1.in‚48610.80859375 disk.sda2.in‚0 disk.sda.in‚ 108.921875 disk.drbd2165.in‚0* proc.memory.percent‚17.119598388671875 cpu‚25.58364604317636 disk.rbd3.in‚0 disk.rbd1.in‚ 0.52392578125& disk.rbd0.in‚1.759218604441423e+13 network.recv‚ 5.084594e+06 disk.rbd1.out‚ 53.3154296875 network.sent‚ 2.3780029e+07$ disk./.usageGB‚565.0557899475098$ disk.dm-0.out‚146660.97705078125 disk.sda1.out‚0 disk.sda3.in‚ 108.921875) disk./.usagePercent‚37.92136817676431 disk.dm-1.in‚ 108.921875 disk.dm-0.in‚48610.80859375 disk.rbd3.out‚ 0.4951171875 disk.sda2.out‚0 disk.sda1.in‚0$ memory_percent‚82.78872966766357®¬4B”Õ1‚(:‰ ’ÔéÒ¹½Ëö gpu.0.gpu‚90.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚268.238' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.6476 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÞK_AÖ1‚(:¶ ’ÔéÒž­ú disk.sda1.in‚0 disk.rbd1.out‚ 53.3935546875 network.recv‚ 5.089137e+06 disk./.usageGB‚565.0625 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.rbd3.out‚ 0.4951171875$ disk.nvme0c0n1.in‚48611.01171875" disk.nvme0n1.in‚48611.01171875' disk.nvme0n1.out‚146671.07177734375 disk.sda2.out‚0 disk.dm-1.in‚ 108.921875& disk.rbd0.in‚1.759218604441423e+13$ disk.dm-0.out‚146671.07177734375 disk.sda2.in‚0 disk.sda.in‚ 108.921875# memory_percent‚82.7873706817627 network.sent‚ 2.384283e+07 disk.dm-0.in‚48611.01171875 disk.sda3.in‚ 108.921875 disk.dm-1.out‚ 1299.11328125 proc.memory.rssMB‚2804.875 disk.sda3.out‚ 1299.11328125 disk.rbd3.in‚0( proc.memory.availableMB‚ 2820.1171875 proc.cpu.threads‚17) disk./.usagePercent‚37.92181849401704 disk.drbd2165.out‚0) disk.nvme0c0n1.out‚146671.07177734375 disk.rbd1.in‚ 0.52392578125 disk.sda.out‚ 1299.11328125* proc.memory.percent‚17.119598388671875 cpu‚25.583509291003026 disk.rbd0.out‚0­’m”×1‚(:‰ ¡ÔéÒ¸ÎÏö gpu.0.gpu‚99.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚214.949' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚42.9898 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0{À ŠIØ1‚(:¾ ¡ÔéÒþÌÜø disk.rbd0.out‚0* proc.memory.percent‚17.119598388671875 disk.sda1.in‚0 disk.rbd3.out‚ 0.4951171875 disk.sda2.in‚0 disk.rbd1.in‚ 0.52392578125 disk.dm-1.in‚ 108.921875 network.recv‚ 5.091368e+06$ disk./.usageGB‚565.0694808959961 disk.sda3.out‚ 1299.27734375 disk.sda2.out‚0 disk.rbd1.out‚ 53.3935546875& disk.rbd0.in‚1.759218604441423e+13 cpu‚25.583379342250822 network.sent‚ 2.3847522e+07 disk.sda.in‚ 108.921875 proc.cpu.threads‚17( disk./.usagePercent‚37.9222869878047 disk.sda3.in‚ 108.921875 disk.drbd2165.in‚0 disk.drbd2165.out‚0$ disk.nvme0c0n1.in‚48611.43359375 disk.dm-1.out‚ 1299.27734375 proc.memory.rssMB‚2804.875 disk.dm-0.in‚48611.43359375$ disk.dm-0.out‚146679.54345703125 disk.sda1.out‚0" disk.nvme0n1.in‚48611.43359375 disk.rbd3.in‚0) disk.nvme0c0n1.out‚146679.54345703125$ memory_percent‚82.78446197509766& proc.memory.availableMB‚ 2820.59375' disk.nvme0n1.out‚146679.54345703125 disk.sda.out‚ 1299.27734375ƒšèžÙ1‚(:“ °ÔéÒ‚æ¦÷ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚268.527' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚53.70539999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0oÞïr7Ú1‚(:¬ °ÔéÒ½ËÏù( disk./.usagePercent‚37.9226054611773$ disk./.usageGB‚565.0742263793945 disk.sda.out‚ 1299.72265625 disk.dm-0.out‚146694.9453125 disk.sda3.out‚ 1299.72265625 disk.dm-1.out‚ 1299.72265625& disk.rbd0.in‚1.759218604441423e+13 disk.drbd2165.out‚0! disk.nvme0c0n1.in‚ 48714.28125) proc.memory.availableMB‚ 2820.55859375 proc.memory.rssMB‚2804.875* proc.memory.percent‚17.119598388671875 cpu‚25.583218653543774 network.recv‚ 5.094779e+06 disk.sda2.in‚0 disk.rbd1.in‚ 0.52392578125 disk.sda.in‚ 108.921875 disk.rbd3.in‚0 network.sent‚ 2.3852594e+07 disk.sda2.out‚0 disk.sda1.in‚0 disk.sda3.in‚ 108.921875$ memory_percent‚82.78467655181885% disk.nvme0c0n1.out‚146694.9453125 disk.nvme0n1.in‚ 48714.28125# disk.nvme0n1.out‚146694.9453125 disk.rbd0.out‚0 disk.rbd1.out‚ 53.4560546875 disk.dm-1.in‚ 108.921875 disk.rbd3.out‚ 0.4951171875 proc.cpu.threads‚17 disk.dm-0.in‚ 48714.28125 disk.sda1.out‚0 disk.drbd2165.in‚0’‰&O”Û1‚(:‰ ¿ÔéÒþ¾èö gpu.0.gpu‚92.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚46.0 gpu.0.powerWatts‚337.816' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚67.5632 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¼߶¦Ü1‚(:¬ ¿ÔéÒÂàù disk.rbd3.in‚0" disk.nvme0c0n1.in‚ 48714.484375& disk.rbd0.in‚1.759218604441423e+13 disk.dm-0.in‚ 48714.484375 proc.cpu.threads‚17 network.sent‚ 2.3857683e+07 network.recv‚ 5.097901e+06$ disk.nvme0c0n1.out‚ 146703.953125 disk.dm-0.out‚ 146703.953125) disk./.usagePercent‚37.92304528212112 disk.rbd3.out‚ 0.4951171875 disk.dm-1.out‚ 1300.05078125$ memory_percent‚82.78472423553467$ disk./.usageGB‚565.0807800292969 disk.sda3.out‚ 1300.05078125 disk.drbd2165.out‚0 disk.rbd1.out‚ 53.4560546875 disk.sda.out‚ 1300.05078125 proc.memory.rssMB‚2804.875* proc.memory.percent‚17ÜŽÜÖ‘.119598388671875 disk.sda3.in‚ 108.921875 disk.drbd2165.in‚0 disk.sda2.out‚0 disk.dm-1.in‚ 108.921875 disk.rbd1.in‚ 0.52392578125 disk.rbd0.out‚0) proc.memory.availableMB‚ 2820.55078125 disk.nvme0n1.in‚ 48714.484375" disk.nvme0n1.out‚ 146703.953125 disk.sda.in‚ 108.921875 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.in‚0 cpu‚25.58308453349459ÿ‹ ŸÝ1‚(:” ÎÔéÒ”¶Öö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚273.071' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚54.614200000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚01,óë>Þ1‚(:³ ÎÔéÒÛ¿šú proc.cpu.threads‚17 disk.sda1.in‚0 network.recv‚ 5.101624e+06 disk.sda3.out‚ 1300.27734375 disk.drbd2165.out‚0) proc.memory.availableMB‚ 2820.27734375 disk.dm-1.in‚ 108.921875 proc.memory.rssMB‚2804.875 disk.rbd3.in‚0 disk.sda1.out‚0$ disk./.usageGB‚565.0877342224121 disk.sda.in‚ 108.921875 disk.dm-1.out‚ 1300.27734375 disk.dm-0.in‚ 48714.90625" disk.dm-0.out‚146720.830078125 cpu‚25.582937629502425 network.sent‚ 2.3862892e+07) disk./.usagePercent‚37.92351198385604 disk.drbd2165.in‚0% disk.nvme0n1.out‚146720.830078125 disk.rbd1.out‚ 53.5341796875$ memory_percent‚82.78639316558838 disk.sda3.in‚ 108.921875! disk.nvme0c0n1.in‚ 48714.90625 disk.rbd1.in‚ 0.52392578125 disk.rbd3.out‚ 0.4951171875' disk.nvme0c0n1.out‚146720.830078125 disk.nvme0n1.in‚ 48714.90625 disk.sda2.out‚0 disk.sda.out‚ 1300.27734375& disk.rbd0.in‚1.759218604441423e+13* proc.memory.percent‚17.119598388671875 disk.rbd0.out‚0 disk.sda2.in‚0/ÎH„4ß1/ _wandbruntime‚20536  _runtime‚20536¶ŒÝLÊà1Ä & lr-AdamW/pg1‚0.0002265061232250612 & lr-AdamW/pg2‚0.0002265061232250612 trainer/global_step‚12399 % _timestamp‚1.7843103527154963e+09 _runtime‚20536.932656527 _step‚10183gZbÏá1É & lr-AdamW/pg1‚0.0002265061232250612 & lr-AdamW/pg2‚0.0002265061232250612 trainer/global_step‚12399 % _timestamp‚1.7843103527154963e+09 _runtime‚20536.932656527 _step‚1018ú¢~1 4â1/ _wandbruntime‚20536  _runtime‚20536¿í<*(ã1P O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 213071241 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 105718951 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚3573796896800256 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11660924296704 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚31804944 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚962466573996032 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11660924296704 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3573796896800256 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 107362269 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚3560597805764608 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 213081220 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚977503938702848 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11640972773446 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 105718951 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚66925648¥3 “(ä1”P H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 105718951 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚66925648 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 213071241 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 105718951 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚3573796896800256 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11660924296704 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚31804944 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚962466573996032 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11660924296704 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3573796896800256 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 107362269 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚3560597805764608 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 213081220 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚977503938702848 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11640972773446 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640û/m&4å1/ _wandbruntime‚20538  _runtime‚20538ÙT÷Pá%æ1šN K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 107360774 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 213097918 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 107360774 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3562413973233664 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚241 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 213097918 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3574077194136064 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚977583026812416 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3574077194136064 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 105727157 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11661849332224 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚35123524 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3562413999407104 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 400523264 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _runtime‚20538.317551704 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7843103541002076e+09 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 213087925 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚31807487 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 trainer/global_step‚12399 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3560875687542784 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚66930765 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚66931011 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3574075822565888 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚1020 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3572518953810878 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 105727151 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚3562413973233664 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11663194728960 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3562413999407104 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 400523264 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _runtime‚20538.317551704 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7843103541002076e+09 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 213087925 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚31807487 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 trainer/global_step‚12399 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3560875687542784 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚66930765 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚66931011 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3574075822565888 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚1020 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3572518953810878 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 105727151 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚3562413973233664 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11663194728960 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 213097918 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 107360774 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3562413973233664 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚241 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 213097918 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3574077194136064 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚977583026812416 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3574077194136064 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 105727157 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0ü¯=•4è1/ _wandbruntime‚20538  _runtime‚20538Ä/‚mé1‹ 0loss/SRqqt1/detection_loss‚0.565903337402429 0loss/FRt1/assignment_loss‚1.1443883790059035 2loss/SRqqt2/assignment_loss‚0.2220249546741984 4loss/FBt2/assignment_loss‚3.9013927474487986e-08 epoch‚30 _runtime‚20538.404063263 /loss/FRt1/detection_loss‚0.4467756475564375 1loss/SRqqt2/detection_loss‚0.5195775187008663 trainer/global_step‚12399 % _timestamp‚1.7843103541870794e+09 3loss/SRqqt1/assignment_loss‚0.17151187253839378 /loss/FBt1/detection_loss‚0.3874542053346522 /loss/FBt2/detection_loss‚0.3011164588541512 %loss/total_loss‚5.411561432916438 _step‚1021 1loss/FRt2/assignment_loss‚0.25234013353681706 3loss/FBt1/assignment_loss‚0.0031066417949469384 /loss/FRt2/detection_loss‚1.3973622445037135®Øšê1 3loss/SRqqt1/assignment_loss‚0.17151187253839378 /loss/FBt1/detection_loss‚0.3874542053346522 /loss/FBt2/detection_loss‚0.3011164588541512 %loss/total_loss‚5.411561432916438 _step‚1021 1loss/FRt2/assignment_loss‚0.25234013353681706 3loss/FBt1/assignment_loss‚0.0031066417949469384 /loss/FRt2/detection_loss‚1.3973622445037135 0loss/SRqqt1/detection_loss‚0.565903337402429 0loss/FRt1/assignment_loss‚1.1443883790059035 2loss/SRqqt2/assignment_loss‚0.2220249546741984 4loss/FBt2/assignment_loss‚3.9013927474487986e-08 epoch‚30 _runtime‚20538.404063263 /loss/FRt1/detection_loss‚0.4467756475564375 1loss/SRqqt2/detection_loss‚0.5195775187008663 trainer/global_step‚12399 % _timestamp‚1.7843103541870794e+09ý^úוë1‚(:Š ÝÔéÒè÷äö gpu.0.gpu‚100.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚41.0 gpu.0.powerWatts‚111.642' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.3284 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0©Qïù9ì1‚(:® ÝÔéÒ¨¾Óù network.recv‚ 5.105124e+06 disk.drbd2165.out‚0 disk.rbd1.out‚ 53.5341796875 proc.cpu.threads‚17$ disk./.usageGB‚565.0921859741211 disk.sda1.in‚0 disk.sda.out‚ 1300.5390625 disk.dm-1.out‚ 1300.5390625 network.sent‚ 2.386807e+07 disk.rbd0.out‚0 disk.sda3.in‚ 108.921875 disk.sda3.out‚ 1300.5390625 disk.rbd3.in‚0 disk.sda.in‚ 108.921875) proc.memory.percent‚17.45932102203369 disk.dm-0.in‚ 48715.125 disk.rbd3.out‚ 0.4951171875& disk.nvme0n1.out‚146730.3935546875# disk.dm-0.out‚146730.3935546875 disk.drbd2165.in‚0 disk.nvme0n1.in‚ 48715.125& disk.rbd0.in‚1.759218604441423e+13 disk.sda2.in‚0( disk./.usagePercent‚37.9238107446485 disk.sda1.out‚0( disk.nvme0c0n1.out‚146730.3935546875 disk.sda2.out‚0 disk.rbd1.in‚ 0.52392578125 disk.dm-1.in‚ 108.921875) proc.memory.availableMB‚ 2764.61328125# proc.memory.rssMB‚ 2860.53515625 disk.nvme0c0n1.in‚ 48715.125# memory_percent‚83.1261396408081 cpu‚25.584312473246936ùѼ”í1‚(:‰ ìÔéÒ¬½Ê÷ gpu.0.gpu‚99.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚327.052' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚65.4104 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÿìøGî1‚(:¼ ìÔéÒòÛ‚ú$ disk./.usageGB‚565.0834350585938# disk.nvme0c0n1.in‚ 48848.8046875) proc.memory.availableMB‚ 2820.26171875 network.sent‚ 2.393136e+07 disk.sda.in‚ 108.921875 disk.drbd2165.in‚0 proc.memory.rssMB‚2804.875 cpu‚25.586120092956367* disk./.usagePercent‚37.923223463364955' disk.nvme0n1.out‚146738.43505859375 disk.sda2.in‚0 disk.rbd1.in‚ 0.52392578125 disk.sda3.out‚ 1300.6015625 disk.rbd3.in‚0) disk.nvme0c0n1.out‚146738.43505859375! disk.nvme0n1.in‚ 48848.8046875 disk.dm-1.in‚ 108.921875 disk.dm-1.out‚ 1300.6015625 disk.dm-0.in‚ 48848.8046875$ disk.dm-0.out‚146738.43505859375* proc.memory.percent‚17.119598388671875 network.recv‚ 5.109671e+06 disk.sda2.out‚0& disk.rbd0.in‚1.759218604441423e+13 disk.rbd3.out‚ 0.4951171875 disk.rbd1.out‚ 53.5966796875 disk.rbd0.out‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda3.in‚ 108.921875 disk.drbd2165.out‚0 proc.cpu.threads‚17 disk.sda.out‚ 1300.6015625$ memory_percent‚82.78648853302002þ½n4ï1/ _wandbruntime‚20565  _runtime‚20565BVË>ð1Ž| ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.6822024583816528 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.7862469553947449 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5781930088996887 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.012067905627191067 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.38293829560279846 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.13045786321163177 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.04898689314723015 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.22993911802768707 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.05000000074505806 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.1628623753786087 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.04907483235001564 +jet/accuracy_2_of_3‚0.46805015206336975 epoch‚30 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6285915970802307 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9820137023925781 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.12470284104347229 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7059388756752014 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.6288412809371948 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.20455582439899445 , Purity/0FRt2SRqqt*FBt/FBt_purity‚0.90625 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.2903188169002533 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.1588941365480423 +particle/sensitivity‚0.5953745245933533 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.18020455539226532 *jet/accuracy_1_of_3‚0.8984122276306152 /particle/accuracy_4_of_4‚0.9534059166908264 (particle/accuracy‚0.8351818919181824 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.7765683531761169 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7221112251281738 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.1872425228357315 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5760305523872375 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6614834070205688 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 jet/accuracy_1_of_5‚1 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.1505020409822464 +jet/accuracy_3_of_4‚0.05117548257112503 particle/accuracy_1_of_2‚1 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.2239583283662796 6 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.991265058517456 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9877339601516724 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 8"Purity/0FRt1SRqqt1FBt/event_purity‚0.319769024848938 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7546296119689941 particle/accuracy_2_of_3‚1 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 )jet/accuracy_1_of_1‚0.282317191362381 particle/accuracy_2_of_2‚1 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 particle/accuracy_1_of_5‚1 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7756613492965698 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9166666865348816 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9970017671585083 particle/accuracy_1_of_3‚1 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9038461446762085 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.1666666716337204 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7328761219978333 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.02614707127213478 jet/accuracy_2_of_5‚1 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 *jet/accuracy_1_of_2‚0.7255961894989014 particle/accuracy_1_of_4‚1 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6278411149978638 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.2232426255941391 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 )loss/val_total_loss‚9.164915354424314 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5570452809333801 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9988662004470825 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.013552890159189701 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 particle/accuracy_3_of_5‚1 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6485661268234253 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.1703970581293106 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6592270731925964 +jet/accuracy_2_of_2‚0.11627529561519623 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.17218506336212158 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4656342566013336 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.6867148280143738 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9954294562339783 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.15677452087402344 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 , Purity/0FRt2SRqqt1FBt/FBt_purity‚0.90625 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6606248021125793 *jet/accuracy_2_of_4‚0.6956015825271606 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5531973242759705 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9856237769126892 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6393315196037292 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6329103112220764 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.7862469553947449 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4609375 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.14014756679534912 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7131425142288208 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.16799411177635193 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8572776317596436 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6116797924041748 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6050034761428833 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6731327772140503 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6137813329696655 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7611300349235535 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.46941930055618286 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9176920652389526 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.15394604206085205 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3499999940395355 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.15107253193855286 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9934056401252747 6 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.998511552810669 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9398148655891418 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.03068261221051216 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9937976002693176 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.026113858446478844 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.19230769574642181 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.19230769574642181 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.17540580034255981 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 /particle/accuracy_3_of_4‚0.9974937438964844 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9929051399230957 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.24329575896263123 8"Purity/0FRt0SRqqt1FBt/event_purity‚0.998511552810669 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7721278071403503 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 ,jet/accuracy_4_of_4‚0.013899499550461769 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 particle/accuracy_1_of_1‚1 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.8125 particle/accuracy_4_of_5‚1 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.5862090587615967 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6285645365715027 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.05041275545954704 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9898549318313599 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3375891149044037 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6188209056854248 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.5612345933914185 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9875718951225281 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.11953040212392807 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2630346715450287 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6734656095504761 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6633542776107788 particle/accuracy_2_of_4‚1 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.42129626870155334 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.05041275545954704 jet/accuracy_5_of_5‚0 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.22034505009651184 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9140046238899231 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9800359606742859 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 . Purity/2FRt0SRqqt1FBt/FRt_purity‚ 0.5703125 % _timestamp‚1.7843103813853855e+09 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9954496026039124 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.04953377693891525 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.05000000074505806 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8184767365455627 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.049319829791784286 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.3940850794315338 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 particle/accuracy_5_of_5‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.7670823931694031 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.10499999672174454 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 +particle/specificity‚0.9007679224014282 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7142857313156128 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5671795606613159 6validation_average_jet_accuracy‚0.3244267404079437 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9989176988601685 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.1666666716337204 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.44212964177131653 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.12134471535682678 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 +jet/accuracy_3_of_3‚0.04895462468266487 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.10455316305160522 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 _step‚1022 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.006220660172402859 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5016666650772095 *jet/accuracy_1_of_4‚0.9596239924430847 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6720088720321655 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5869879722595215 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9712474346160889 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7181078195571899 'particle/f_score‚0.6080596446990967 6 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.996728777885437 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.15547089278697968 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9960039854049683 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 jet/accuracy_3_of_5‚1 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.14129796624183655 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6283711791038513 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 particle/accuracy_3_of_3‚1 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.33225375413894653 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.15677452087402344 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8005693554878235 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9964826703071594 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.13333430886268616 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.1164589524269104 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.6923092007637024 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6413664817810059 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 jet/accuracy_4_of_5‚0 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5760305523872375 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 6 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.994455873966217 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.8571428656578064 particle/accuracy_2_of_5‚1 trainer/global_step‚12399 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.17117774486541748 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9398148655891418 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9971989393234253 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7216700315475464 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.625 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.1568988561630249 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7546296119689941 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 _runtime‚20565.60316314ÎþÏüñ1“| ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.013552890159189701 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 particle/accuracy_3_of_5‚1 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6485661268234253 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.1703970581293106 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6592270731925964 +jet/accuracy_2_of_2‚0.11627529561519623 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.17218506336212158 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4656342566013336 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.6867148280143738 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9954294562339783 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.15677452087402344 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 , Purity/0FRt2SRqqt1FBt/FBt_purity‚0.90625 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6606248021125793 *jet/accuracy_2_of_4‚0.6956015825271606 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5531973242759705 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9856237769126892 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6393315196037292 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6329103112220764 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.7862469553947449 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 0"P­ÃêÄ7urity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4609375 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.14014756679534912 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7131425142288208 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.16799411177635193 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8572776317596436 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6116797924041748 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6050034761428833 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6731327772140503 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6137813329696655 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7611300349235535 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.46941930055618286 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9176920652389526 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.15394604206085205 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3499999940395355 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.15107253193855286 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9934056401252747 6 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.998511552810669 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9398148655891418 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.03068261221051216 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9937976002693176 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.026113858446478844 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.19230769574642181 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.19230769574642181 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.17540580034255981 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 /particle/accuracy_3_of_4‚0.9974937438964844 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9929051399230957 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.24329575896263123 8"Purity/0FRt0SRqqt1FBt/event_purity‚0.998511552810669 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7721278071403503 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 ,jet/accuracy_4_of_4‚0.013899499550461769 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 particle/accuracy_1_of_1‚1 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.8125 particle/accuracy_4_of_5‚1 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.5862090587615967 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6285645365715027 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.05041275545954704 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9898549318313599 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3375891149044037 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6188209056854248 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.5612345933914185 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9875718951225281 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.11953040212392807 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2630346715450287 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6734656095504761 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6633542776107788 particle/accuracy_2_of_4‚1 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.42129626870155334 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.05041275545954704 jet/accuracy_5_of_5‚0 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.22034505009651184 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9140046238899231 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9800359606742859 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 . Purity/2FRt0SRqqt1FBt/FRt_purity‚ 0.5703125 % _timestamp‚1.7843103813853855e+09 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9954496026039124 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.04953377693891525 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.05000000074505806 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8184767365455627 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.049319829791784286 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.3940850794315338 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 particle/accuracy_5_of_5‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.7670823931694031 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.10499999672174454 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 +particle/specificity‚0.9007679224014282 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7142857313156128 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5671795606613159 6validation_average_jet_accuracy‚0.3244267404079437 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9989176988601685 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.1666666716337204 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.44212964177131653 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.12134471535682678 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 +jet/accuracy_3_of_3‚0.04895462468266487 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.10455316305160522 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 _step‚1022 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.006220660172402859 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5016666650772095 *jet/accuracy_1_of_4‚0.9596239924430847 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6720088720321655 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5869879722595215 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9712474346160889 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7181078195571899 'particle/f_score‚0.6080596446990967 6 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.996728777885437 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.15547089278697968 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9960039854049683 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 jet/accuracy_3_of_5‚1 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.14129796624183655 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6283711791038513 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 particle/accuracy_3_of_3‚1 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.33225375413894653 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.15677452087402344 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8005693554878235 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9964826703071594 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.13333430886268616 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.1164589524269104 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.6923092007637024 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6413664817810059 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 jet/accuracy_4_of_5‚0 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5760305523872375 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 6 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.994455873966217 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.8571428656578064 particle/accuracy_2_of_5‚1 trainer/global_step‚12399 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.17117774486541748 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9398148655891418 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9971989393234253 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7216700315475464 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.625 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.1568988561630249 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7546296119689941 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 _runtime‚20565.60316314 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.6822024583816528 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.7862469553947449 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5781930088996887 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.012067905627191067 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.38293829560279846 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.13045786321163177 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.04898689314723015 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.22993911802768707 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.05000000074505806 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.1628623753786087 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.04907483235001564 +jet/accuracy_2_of_3‚0.46805015206336975 epoch‚30 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6285915970802307 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9820137023925781 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.12470284104347229 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7059388756752014 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.6288412809371948 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.20455582439899445 , Purity/0FRt2SRqqt*FBt/FBt_purity‚0.90625 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.2903188169002533 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.1588941365480423 +particle/sensitivity‚0.5953745245933533 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.18020455539226532 *jet/accuracy_1_of_3‚0.8984122276306152 /particle/accuracy_4_of_4‚0.9534059166908264 (particle/accuracy‚0.8351818919181824 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.7765683531761169 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7221112251281738 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.1872425228357315 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5760305523872375 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6614834070205688 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 jet/accuracy_1_of_5‚1 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.1505020409822464 +jet/accuracy_3_of_4‚0.05117548257112503 particle/accuracy_1_of_2‚1 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.2239583283662796 6 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.991265058517456 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9877339601516724 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 8"Purity/0FRt1SRqqt1FBt/event_purity‚0.319769024848938 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7546296119689941 particle/accuracy_2_of_3‚1 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 )jet/accuracy_1_of_1‚0.282317191362381 particle/accuracy_2_of_2‚1 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 particle/accuracy_1_of_5‚1 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7756613492965698 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9166666865348816 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9970017671585083 particle/accuracy_1_of_3‚1 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9038461446762085 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.1666666716337204 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7328761219978333 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.02614707127213478 jet/accuracy_2_of_5‚1 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 *jet/accuracy_1_of_2‚0.7255961894989014 particle/accuracy_1_of_4‚1 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6278411149978638 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.2232426255941391 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 )loss/val_total_loss‚9.164915354424314 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5570452809333801 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9988662004470825 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969þÈŸÝžò1‚(:“ ûÔéÒ²üåö gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚295.687' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚59.13740000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0B?UEó1‚(:º ûÔéҙ͖ú disk.sda2.in‚0 disk.sda.out‚ 1300.8828125 disk.dm-1.out‚ 1300.8828125 disk.rbd0.out‚0* proc.memory.percent‚17.119598388671875 disk.dm-0.in‚ 48849.0078125 disk.rbd3.in‚0 disk.rbd3.out‚ 0.4951171875! disk.nvme0n1.in‚ 48849.0078125$ memory_percent‚82.78851509094238 cpu‚25.586372522976284 disk.sda3.out‚ 1300.8828125 disk.drbd2165.out‚0( proc.memory.availableMB‚ 2819.9296875 network.recv‚ 5.112662e+06# disk./.usageGB‚565.101921081543 disk.sda.in‚ 108.921875 disk.sda3.in‚ 108.921875 disk.drbd2165.in‚0 disk.sda2.out‚0 disk.sda1.in‚0 network.sent‚ 2.3935937e+07 disk.dm-1.in‚ 108.921875$ disk.dm-0.out‚146754.29443359375# disk.nvme0c0n1.in‚ 48849.0078125) disk.nvme0c0n1.out‚146754.29443359375 disk.rbd1.in‚ 0.52392578125 disk.rbd1.out‚ 53.5966796875 proc.cpu.threads‚17) disk./.usagePercent‚37.92446407587588& disk.rbd0.in‚1.759218604441423e+13 proc.memory.rssMB‚2804.875 disk.sda1.out‚0' disk.nvme0n1.out‚146754.29443359375ÌO¥ñ”ô1‚(:‰ ŠÕéÒàçàø gpu.0.gpu‚99.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚346.941' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚69.3882 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0t :}Gõ1‚(:¼ ŠÕéÒýžú) disk.nvme0c0n1.out‚146762.19287109375 disk.rbd0.out‚0* proc.memory.percent‚17.119598388671875' disk.nvme0n1.out‚146762.19287109375 cpu‚25.586527773702628 disk.sda.out‚ 1301.09765625* disk./.usagePercent‚37.924921305332035 disk.rbd3.in‚0# disk.nvme0c0n1.in‚ 48849.2265625 disk.sda2.in‚0 disk.sda.in‚ 108.921875 disk.dm-1.out‚ 1301.09765625 network.sent‚ 2.3994335e+07 disk.drbd2165.out‚0 disk.sda2.out‚0 disk.rbd1.in‚ 0.52392578125 disk.dm-1.in‚ 108.921875& disk.rbd0.in‚1.759218604441423e+13 disk.dm-0.in‚ 48849.2265625 proc.memory.rssMB‚2804.875 network.recv‚ 5.117136e+06$ disk./.usageGB‚565.1087341308594 disk.sda3.in‚ 108.921875 disk.sda3.out‚ 1301.09765625$ disk.dm-0.out‚146762.19287109375# memory_percent‚82.7901840209961 proc.cpu.threads‚17 disk.rbd1.out‚ 53.6748046875 disk.sda1.in‚0 disk.sda1.out‚0& proc.memory.availableMB‚ 2819.65625! disk.nvme0n1.in‚ 48849.2265625 disk.drbd2165.in‚0 disk.rbd3.out‚ 0.4951171875ðιyžö1‚(:“ ™ÕéÒ´¶óö gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚262.054' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚52.41079999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0íDÕ:÷1‚(:¯ ™ÕéÒñƾù proc.cpu.threads‚17 disk.rbd0.out‚0 disk.sda1.in‚0 disk.drbd2165.out‚0 cpu‚25.586612184909978$ disk.nvme0n1.out‚146772.51953125 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-0.in‚ 48849.6640625 disk.sda3.in‚ 108.921875 disk.rbd3.out‚ 0.4951171875 network.recv‚ 5.120162e+06 disk.rbd1.in‚ 0.52392578125 disk.dm-1.out‚ 1301.515625 disk.sda3.out‚ 1301.515625# disk.nvme0c0n1.in‚ 48849.6640625& disk.nvme0c0n1.out‚146772.51953125& disk.rbd0.in‚1.759218604441423e+13 network.sent‚ 2.3998883e+07 disk.sda.out‚ 1301.515625 disk.dm-1.in‚ 108.921875! disk.dm-0.out‚146772.51953125 disk.drbd2165.in‚0 disk.rbd1.out‚ 53.6748046875 disk.sda1.out‚0 disk.rbd3.in‚0$ memory_percent‚82.79023170471191( proc.memory.availableMB‚ 2819.6484375$ disk./.usageGB‚565.1160469055176 disk.sda.in‚ 108.921875! disk.nvme0n1.in‚ 48849.6640625* proc.memory.percent‚17.119598388671875) disk./.usagePercent‚37.92541207177517 proc.memory.rssMB‚2804.875õÃhº•ø1‚(:Š ¨ÕéÒ͸È÷ gpu.0.gpu‚100.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚341.052' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚68.2104 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0™ú¯;ù1‚(:° ¨ÕéÒŒèù) disk./.usagePercent‚37.92569009595736# disk.dm-0.out‚146785.4892578125 disk.rbd3.in‚0 cpu‚25.58676634112867 network.recv‚ 5.123824e+06 disk.sda.in‚ 108.921875 disk.dm-0.in‚ 49014.265625 disk.rbd1.out‚ 53.7998046875( proc.memory.availableMB‚ 2819.6484375 network.sent‚ 2.4004648e+07 disk.sda1.in‚0 disk.sda2.in‚0 proc.cpu.threads‚17 disk.sda.out‚ 1301.640625 disk.drbd2165.in‚0 disk.nvme0n1.in‚ 49014.265625& disk.nvme0n1.out‚146785.4892578125 disk.dm-1.out‚ 1301.640625 disk.rbd0.out‚0 disk.rbd3.out‚ 0.4951171875 disk.rbd1.in‚ 0.52392578125 disk.sda3.out‚ 1301.640625( disk.nvme0c0n1.out‚146785.4892578125& disk.rbd0.in‚1.759218604441423e+13 disk.sda1.out‚0 disk.drbd2165.out‚0" disk.nvme0c0n1.in‚ 49014.265625 disk.sda2.out‚0$ memory_percent‚82.79023170471191 proc.memory.rssMB‚2804.875# disk./.usageGB‚565.120189666748 disk.dm-1.in‚ 108.921875 disk.sda3.in‚ 108.921875* proc.memory.percent‚17.119598388671875Á쇒ú1‚(:‡ ·ÕéÒÛêœ÷ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚269.59' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.918 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚07‹@û1‚(:µ ·ÕéÒóÜú proc.cpu.threads‚17& disk.rbd0.in‚1.759218604441423e+13) disk./.usagePercent‚37.92615014149639$ disk.dm-0.out‚146796.88330078125 disk.sda3.out‚ 1301.9453125& proc.memory.availableMB‚ 2819.65625 network.recv‚ 5.126795e+06 disk.dm-0.in‚ 49014.484375 disk.sda2.out‚0 disk.rbd1.out‚ 53.7998046875# memory_percent‚82.7901840209961$ disk./.usageGB‚565.1270446777344 disk.drbd2165.out‚0' disk.nvme0n1.out‚146796.88330078125 disk.sda.in‚ 108.921875 disk.rbd3.in‚0 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.nvme0n1.in‚ 49014.484375 disk.sda2.in‚0" disk.nvme0c0n1.in‚ 49014.484375) disk.nvme0c0n1.out‚146796.88330078125 disk.dm-1.in‚ 108.921875 proc.memory.rssMB‚2804.875* proc.memory.percent‚17.119598388671875 disk.rbd3.out‚ 0.4951171875 network.sent‚ 2.4009718e+07 disk.rbd1.in‚ 0.52392578125 disk.rbd0.out‚0 cpu‚25.586835936699316 disk.sda1.in‚0 disk.sda3.in‚ 108.921875 disk.sda.out‚ 1301.9453125 disk.dm-1.out‚ 1301.9453125dƒà‰4ü1/ _wandbruntime‚20643  _runtime‚20643 Šu¼Ìý1Æ ' lr-AdamW/pg1‚0.00022390808932892364 ' lr-AdamW/pg2‚0.00022390808932892364 trainer/global_step‚12449 % _timestamp‚1.7843104593567646e+09 _runtime‚20643.573963981 _step‚1023¤JÚ-Ñþ1Ë trainer/global_step‚12449 % _timestamp‚1.7843104593567646e+09 _runtime‚20643.573963981 _step‚1023 ' lr-AdamW/pg1‚0.00022390808932892364 ' lr-AdamW/pg2‚0.00022390808932892364ÿ‘Ò 4ÿ1/ _wandbruntime‚20643  _runtime‚20643pƼ(€2P M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 106261251 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚3579864016617472 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _step‚1024 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚966389006626816 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3591578444846080 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚254 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚3578317803606016 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11713082839040 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚981494119119872 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3591578444846080 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚31941012 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚35293838 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚3578317829033984 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚981493716492288 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚3590010842624007 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚3591577073283072 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11714428228608 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚966389374717952 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _runtime‚20643.575208601 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3579863990444032 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚3579863990444032 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚35294087 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11693039017991 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 107847003 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15104709865472 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3591577073283072 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3579864016617472 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚19950õ2•P A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚3579863990444032 VHDeviceStatsMonitor.on_train_ba90Óþ'tch_start/inactive_split_bytes.all.current‚ 402627584 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚35294087 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11693039017991 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 107847003 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15104709865472 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3591577073283072 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3579864016617472 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 106261251 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚3579864016617472 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _step‚1024 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚966389006626816 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3591578444846080 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚254 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚3578317803606016 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11713082839040 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚981494119119872 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3591578444846080 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚31941012 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚35293838 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚3578317829033984 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚981493716492288 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚3590010842624007 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚3591577073283072 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11714428228608 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚966389374717952 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _runtime‚20643.575208601 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3579863990444032 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504€XÚv4‚2/ _wandbruntime‚20644  _runtime‚20644dßjl'ƒ2˜N RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 107845508 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11695307084671 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 106269463 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 106269463 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11714007874560 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 214124958 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚67240370 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15105907144704 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3578595685384192 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11693962484237 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 107855495 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 214114965 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 214124958 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15105941673984 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3580142017347584 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3578595710812160 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚3580142017347584 _runtime‚ 20644.9604492 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚243 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚35296572 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚966466861125632 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3580141991174144 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11714007874560 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚981572400179200 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3590291017896831 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 106269457 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚67240122 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚981572802799616 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 107845508 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚31943555 trainer/global_step‚12449 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11715353271296 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3591855999048704 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 214114965 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 107855495 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7843104607430382e+09 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3591857370618880 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3591857370618880 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚254279689˯†#'„2N F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚981572400179200 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3590291017896831 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 106269457 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚67240122 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚981572802799616 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 107845508 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚31943555 trainer/global_step‚12449 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11715353271296 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3591855999048704 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 214114965 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 107855495 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7843104607430382e+09 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3591857370618880 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3591857370618880 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 107845508 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11695307084671 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 106269463 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 106269463 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11714007874560 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 214124958 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚67240370 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15105907144704 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3578595685384192 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11693962484237 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 107855495 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 214114965 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 214124958 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15105941673984 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3580142017347584 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3578595710812160 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚3580142017347584 _runtime‚ 20644.9604492 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚243 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚35296572 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚966466861125632 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3580141991174144 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11714007874560 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820jz,…4…2/ _wandbruntime‚20645  _runtime‚20645?È? †2‡ /loss/FBt2/detection_loss‚0.2966052804742289 trainer/global_step‚12449 _runtime‚20645.046389174 epoch‚31 0loss/FRt1/assignment_loss‚1.0633438302386076 1loss/FRt2/assignment_loss‚0.23896388599821622 3loss/FBt1/assignment_loss‚7.892044373891383e-05 2loss/FBt2/assignment_loss‚5.34407841327385e-05 .loss/FRt2/detection_loss‚1.190946279126529 0loss/SRqqt2/detection_loss‚0.504207130315623 _step‚1026 3loss/SRqqt1/assignment_loss‚0.15252202007426718 1loss/SRqqt2/assignment_loss‚0.178330402411532 /loss/FRt1/detection_loss‚0.4325786647926374 /loss/FBt1/detection_loss‚0.5236298724587353 %loss/total_loss‚5.139800183059629 % _timestamp‚1.7843104608294523e+09 1loss/SRqqt1/detection_loss‚0.5585404559413811Þ¿‡2Œ 2loss/FBt2/assignment_loss‚5.34407841327385e-05 .loss/FRt2/detection_loss‚1.190946279126529 0loss/SRqqt2/detection_loss‚0.504207130315623 _step‚1026 3loss/SRqqt1/assignment_loss‚0.15252202007426718 1loss/SRqqt2/assignment_loss‚0.178330402411532 /loss/FRt1/detection_loss‚0.4325786647926374 /loss/FBt1/detection_loss‚0.5236298724587353 %loss/total_loss‚5.139800183059629 % _timestamp‚1.7843104608294523e+09 1loss/SRqqt1/detection_loss‚0.5585404559413811 /loss/FBt2/detection_loss‚0.2966052804742289 trainer/global_step‚12449 _runtime‚20645.046389174 epoch‚31 0loss/FRt1/assignment_loss‚1.0633438302386076 1loss/FRt2/assignment_loss‚0.23896388599821622 3loss/FBt1/assignment_loss‚7.892044373891383e-05‚¯äྞˆ2‚(:“ ÆÕéÒö¶Îö gpu.0.gpu‚87.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚333.618' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚66.72359999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0QLAY‰2‚(:° ÆÕéҥ݃ú) disk./.usagePercent‚37.92666318059501 disk.dm-0.in‚ 49014.90625 disk.sda3.in‚ 108.921875' disk.nvme0n1.out‚146809.59326171875 disk.sda2.in‚0 disk.sda2.out‚0' proc.memory.availableMB‚ 2819.671875 disk.rbd0.out‚0 disk.sda3.out‚ 1302.234375 disk.drbd2165.out‚0 disk.rbd1.in‚ 0.52392578125 disk.rbd1.out‚ 53.8779296875 network.recv‚ 5.130453e+06 disk.sda.out‚ 1302.234375 disk.rbd3.in‚0$ disk./.usageGB‚565.1346893310547* proc.memory.percent‚17.119598388671875 disk.nvme0n1.in‚ 49014.90625 disk.sda.HÙ¼âin‚ 108.921875 disk.dm-1.out‚ 1302.234375& disk.rbd0.in‚1.759218604441423e+13 disk.sda1.out‚0 disk.drbd2165.in‚0) disk.nvme0c0n1.out‚146809.59326171875 cpu‚25.58694216238442 network.sent‚ 2.4014782e+07 disk.dm-1.in‚ 108.921875$ disk.dm-0.out‚146809.59326171875 disk.rbd3.out‚ 0.4951171875! disk.nvme0c0n1.in‚ 49014.90625 proc.memory.rssMB‚2804.875 proc.cpu.threads‚17 disk.sda1.in‚0$ memory_percent‚82.79008865356445­Í7w”Š2‚(:‰ ÕÕéÒ¶ É÷ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚265.822' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.1644 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0p‡=‹2‚(:² ÕÕéÒ¥¿¢ù disk.sda2.out‚0 disk.rbd1.out‚ 53.8779296875& disk.rbd0.in‚1.759218604441423e+13 disk.sda1.out‚0 disk.drbd2165.out‚0 disk.rbd3.out‚ 0.4951171875 network.sent‚ 2.4078084e+07 disk.sda3.out‚ 1302.453125 disk.rbd3.in‚0# memory_percent‚82.7901840209961 cpu‚25.587038162704104 proc.cpu.threads‚17 disk.sda.out‚ 1302.453125 disk.dm-1.out‚ 1302.453125 disk.rbd0.out‚0 disk.nvme0n1.in‚ 49015.109375) disk./.usagePercent‚37.92696040534226$ disk./.usageGB‚565.1391181945801 disk.dm-1.in‚ 108.921875 disk.dm-0.in‚ 49015.109375 disk.sda3.in‚ 108.921875 disk.sda2.in‚0 proc.memory.rssMB‚2804.875 disk.sda1.in‚0" disk.nvme0c0n1.in‚ 49015.109375& proc.memory.availableMB‚ 2819.65625 disk.rbd1.in‚ 0.52392578125 network.recv‚ 5.134683e+06 disk.sda.in‚ 108.921875$ disk.dm-0.out‚146820.43603515625 disk.drbd2165.in‚0) disk.nvme0c0n1.out‚146820.43603515625* proc.memory.percent‚17.119598388671875' disk.nvme0n1.out‚146820.43603515625×4ä–žŒ2‚(:“ äÕéÒž—®÷ gpu.0.gpu‚87.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚325.701' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚65.14020000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ìö”\F2‚(:» äÕéÒòªù% disk.nvme0n1.out‚146828.337890625* disk./.usagePercent‚37.927578151522376 disk.sda.in‚ 108.921875 disk.sda.out‚ 1302.66015625" disk.dm-0.out‚146828.337890625 disk.sda1.in‚0 disk.drbd2165.in‚0$ disk.nvme0c0n1.in‚49106.10546875 proc.memory.rssMB‚2804.875 network.recv‚ 5.138291e+06# disk./.usageGB‚565.148323059082 disk.dm-1.in‚ 108.921875 disk.rbd3.in‚0 disk.rbd3.out‚ 0.4951171875' disk.nvme0c0n1.out‚146828.337890625 network.sent‚ 2.4082664e+07 disk.sda3.in‚ 108.921875 disk.sda2.in‚0 disk.sda2.out‚0 proc.cpu.threads‚17& disk.rbd0.in‚1.759218604441423e+13 disk.rbd0.out‚0 disk.drbd2165.out‚0 disk.rbd1.in‚ 0.52392578125* proc.memory.percent‚17.119598388671875 disk.dm-0.in‚49106.10546875" disk.nvme0n1.in‚49106.10546875 disk.rbd1.out‚ 53.9404296875) proc.memory.availableMB‚ 2819.87890625 cpu‚25.58713187320088 disk.dm-1.out‚ 1302.66015625 disk.sda1.out‚0 disk.sda3.out‚ 1302.66015625$ memory_percent‚82.78882503509521Ç+½åŸŽ2‚(:” óÕéÒ¨÷“÷ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚276.599' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚55.319799999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÂÕCG2‚(:¼ óÕéÒΚÇù disk.rbd0.out‚0# disk.dm-0.out‚146843.6630859375 disk.sda3.out‚ 1302.9140625 disk.drbd2165.out‚0) disk./.usagePercent‚37.92787076813402 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.sda.in‚ 108.921875 proc.cpu.threads‚17 disk.rbd1.in‚ 0.52392578125 disk.sda.out‚ 1302.9140625& disk.nvme0n1.out‚146843.6630859375 disk.sda3.in‚ 108.921875 network.sent‚ 2.4087732e+07 network.recv‚ 5.141909e+06$ memory_percent‚82.79030323028564 disk.rbd3.out‚ 0.4951171875 disk.rbd1.out‚ 53.9404296875 proc.memory.rssMB‚2804.875 disk.dm-1.in‚ 108.921875" disk.nvme0n1.in‚49106.32421875) proc.memory.availableMB‚ 2819.63671875 disk.dm-1.out‚ 1302.9140625& disk.rbd0.in‚1.759218604441423e+13( disk.nvme0c0n1.out‚146843.6630859375 disk.sda2.out‚0* proc.memory.percent‚17.119598388671875 cpu‚25.587225065036193 disk.sda2.in‚0$ disk./.usageGB‚565.1526832580566 disk.dm-0.in‚49106.32421875 disk.sda1.in‚0 disk.rbd3.in‚0$ disk.nvme0c0n1.in‚49106.32421875SIà”2‚(:‰ ‚Öéҟߪ÷ gpu.0.gpu‚87.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚346.282' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚69.2564 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0CŸþGK‘2‚(:À ‚ÖéÒÂØþù disk.drbd2165.in‚0 disk.rbd3.in‚0$ memory_percent‚82.79023170471191 disk.sda1.out‚0( proc.memory.availableMB‚ 2819.6484375 disk.sda1.in‚0 disk.sda3.out‚ 1303.13671875 cpu‚25.58737787942621 proc.cpu.threads‚17 network.sent‚ 2.4093365e+07 disk.sda.in‚ 108.921875 disk.sda.out‚ 1303.13671875 disk.sda3.in‚ 108.921875 disk.sda2.in‚0 proc.memory.rssMB‚2804.875 network.recv‚ 5.145538e+06& disk.rbd0.in‚1.759218604441423e+13 disk.rbd0.out‚0 disk.rbd3.out‚ 0.4951171875) disk.nvme0c0n1.out‚146851.16748046875' disk.nvme0n1.out‚146851.16748046875 disk.rbd1.out‚ 54.0029296875) disk./.usagePercent‚37.92818488937853 disk.dm-1.out‚ 1303.13671875 disk.drbd2165.out‚0" disk.nvme0n1.in‚49106.52734375 disk.sda2.out‚0 disk.rbd1.in‚ 0.52392578125 disk.dm-1.in‚ 108.921875$ disk.dm-0.out‚146851.16748046875$ disk.nvme0c0n1.in‚49106.52734375* proc.memory.percent‚17.119598388671875$ disk./.usageGB‚565.1573638916016 disk.dm-0.in‚49106.52734375µÅjÂ4’2/ _wandbruntime‚20722  _runtime‚20722zDŠÌ“2Æ ' lr-AdamW/pg1‚0.00022131789230123505 ' lr-AdamW/pg2‚0.00022131789230123505 trainer/global_step‚12499 % _timestamp‚1.7843105387278194e+09 _runtime‚20722.945005206 _step‚1027FNïRÑ”2Ë ' lr-AdamW/pg2‚0.00022131789230123505 trainer/global_step‚12499 % _timestamp‚1.7843105387278194e+09 _runtime‚20722.945005206 _step‚1027 ' lr-AdamW/pg1‚0.00022131789230123505ƒAÙ4•2/ _wandbruntime‚20722  _runtime‚20722£ÿ$[(–2P [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11759398769664 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 106671557 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚985428208718848 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚3603951168645915 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚3592211892514816 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚250 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚970263695101952 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚35432028 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15164916244480 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚255 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15164881708032 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 108262782 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3605524797282304 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 _runtime‚20722.946267028 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11760744159232 % _timestamp‚1.7843105387289357e+09 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3603952538674205 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 214944312 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3605523425719296 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11760744159232 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚67499935 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11740620731421 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3593764026949632 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚985428611346432 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 108272755 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚32068157 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚35431778 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚3592211917942784 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008£9°(—2•P NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 108272755 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚32068157 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚35431778 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚3592211917942784 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11759398769664 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 106671557 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚985428208718848 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚3603951168645915 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚3592211892514816 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚250 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚970263695101952 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚35432028 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15164916244480 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚255 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15164881708032 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 108262782 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3605524797282304 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 _runtime‚20722.946267028 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11760744159232 % _timestamp‚1.7843105387289357e+09 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3603952538674205 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 214944312 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3605523425719296 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11760744159232 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚67499935 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11740620731421 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3593764026949632 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚985428611346432 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627„üzu4˜2/ _wandbruntime‚20724  _runtime‚20724ÐðqÙ2™N T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚3604229973890337 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚3594042027679744 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚32070700 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 214951017 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11761669201920 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11760323805184 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11760323805184 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚20724.33190722 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 214961010 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11740199597345 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 108271260 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 106679763 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 108271260 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚250 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3592489799720960 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚12499 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3605802351484928 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚245 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3592489774292992 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3605803723055104 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3594042053853184 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3594042027679744 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15166077824000 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11741544197779 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 214961010 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 % _timestamp‚1.7843105401144092e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚985507293862912 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3605803723055104 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚35434764 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚970341181509632 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚35434519 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚970340813418496 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚67505219 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚1029 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 214951017 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11761669201920 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11760323805184 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11760323805184 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚20724.33190722 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 214961010 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11740199597345 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 108271260 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 106679763 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 108271260 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚250 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3592489799720960 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚12499 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3605802351484928 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚245 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3592489774292992 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3605803723055104 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3594042053853184 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3594042027679744 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15166077824000 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11741544197779 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 214961010 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 % _timestamp‚1.7843105401144092e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚985507293862912 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3605803723055104 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚35434764 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚970341181509632 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚35434519 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚970340813418496 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚67505219 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚1029 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B¡2‚(:³  ÖéÒóŒŠù disk.sda3.in‚ 108.921875 disk.rbd3.in‚0) proc.memory.availableMB‚ 2819.66015625 network.sent‚ 2.4162022e+07# disk./.usageGB‚565.171070098877( disk.nvme0c0n1.out‚146876.9736328125 disk.rbd1.out‚ 54.0654296875 disk.rbd0.out‚0* proc.memory.percent‚17.119598388671875! disk.nvme0n1.in‚ 49178.8203125 disk.sda1.in‚0 disk.sda1.out‚0 proc.cpu.threads‚17 disk.dm-1.out‚ 1303.71875 disk.rbd3.out‚ 0.4951171875 disk.sda2.out‚0 disk.drbd2165.in‚0* disk./.usagePercent‚37.929104724449054# disk.nvme0c0n1.in‚ 49178.8203125 disk.sda2.in‚0 disk.sda.in‚ 108.921875& disk.rbd0.in‚1.759218604441423e+13 disk.sda3.out‚ 1303.71875 disk.drbd2165.out‚0$ memory_percent‚82.79016017913818 disk.rbd1.in‚ 0.52392578125 disk.sda.out‚ 1303.71875 disk.dm-1.in‚ 108.921875 disk.dm-0.in‚ 49178.8203125# disk.dm-0.out‚146876.9736328125 proc.memory.rssMB‚2804.875 network.recv‚ 5.153607e+06& disk.nvme0n1.out‚146876.9736328125 cpu‚25.587530843178293Ó–4”¢2‚(:‰ ¯ÖéÒšÖæö gpu.0.gpu‚99.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚333.229' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚66.6458 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0g}vàC£2‚(:¸ ¯ÖéÒàäªù) disk.nvme0c0n1.out‚146888.45068359375 disk.dm-1.out‚ 1303.95703125 disk.sda1.out‚0$ memory_percent‚82.79187679290771 disk.sda3.in‚ 108.921875 disk.rbd3.in‚0 disk.rbd1.in‚ 0.52392578125$ disk.dm-0.out‚146888.45068359375) proc.memory.availableMB‚ 2819.37890625 disk.sda3.out‚ 1303.95703125 disk.sda2.in‚0 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.rbd1.out‚ 54.0654296875 disk.dm-1.in‚ 108.921875& disk.rbd0.in‚1.759218604441423e+13* proc.memory.percent‚17.119598388671875 disk.sda2.out‚0 proc.cpu.threads‚17 disk.nvme0n1.in‚ 49205.71875' disk.nvme0n1.out‚146888.45068359375 network.sent‚ 2.4166617e+07$ disk./.usageGB‚565.1759033203125! disk.nvme0c0n1.in‚ 49205.71875 disk.rbd0.out‚0 network.recv‚ 5.156756e+06 disk.sda.in‚ 108.921875 disk.sda.out‚ 1303.95703125 disk.sda1.in‚0 proc.memory.rssMB‚2804.875 disk.dm-0.in‚ 49205.71875) disk./.usagePercent‚37.92942908599494 disk.rbd3.out‚ 0.4951171875 cpu‚25.58767149502817® ┤2‚(:‰ ¾ÖéÒ¶–èö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚277.986' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.5972 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0l=G¥2‚(:¼ ¾ÖéÒúœ ù network.recv‚ 5.160405e+06) disk.nvme0c0n1.out‚146896.47607421875 disk.sda2.in‚0* proc.memory.percent‚17.119598388671875 disk.dm-0.in‚ 49206.140625$ disk.dm-0.out‚146896.47607421875 disk.sda3.out‚ 1304.24609375 disk.rbd3.in‚0$ memory_percent‚82.79035091400146& disk.rbd0.in‚1.759218604441423e+13' disk.nvme0n1.out‚146896.47607421875 proc.memory.rssMB‚2804.875 network.sent‚ 2.4172262e+07 disk.dm-1.in‚ 108.921875 disk.rbd1.in‚ 0.52392578125 disk.rbd1.out‚ 54.1904296875 disk.sda1.out‚0 disk.sda3.in‚ 108.921875 disk.drbd2165.in‚0 disk.rbd3.out‚ 0.4951171875) proc.memory.availableMB‚ 2819.62890625 proc.cpu.threads‚17) disk./.usagePercent‚37.93006116859698 disk.sda.in‚ 108.921875" disk.nvme0c0n1.in‚ 49206.140625 disk.sda2.out‚0$ disk./.usageGB‚565.1853218078613 disk.sda.out‚ 1304.24609375 disk.rbd0.out‚0 disk.sda1.in‚0 disk.nvme0n1.in‚ 49206.140625 cpu‚25.587728695012704 disk.dm-1.out‚ 1304.24609375 disk.drbd2165.out‚0±õf•¦2‚(:Š ÍÖéÒì¾üö gpu.0.gpu‚100.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚342.642' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚68.5284 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0§Wæ0§2‚(:¥ ÍÖéÒñµºù disk.sda3.out‚ 1304.484375" disk.nvme0c0n1.in‚ 49206.359375 disk.nvme0n1.in‚ 49206.359375# disk./.usageGB‚565.189567565918 disk.dm-1.in‚ 108.921875 disk.dm-0.in‚ 49206.359375 disk.rbd3.out‚ 0.4951171875 disk.rbd1.out‚ 54.1904296875 proc.memory.rssMB‚2804.875 disk.sda.out‚ 1304.484375 disk.rbd0.out‚0 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.rbd3.in‚0 disk.sda2.in‚0 disk.rbd1.in‚ 0.52392578125 network.sent‚ 2.4177407e+07 disk.dm-1.out‚ 1304.484375) proc.memory.availableMB‚ 2819.62109375 network.recv‚ 5.163492e+06$ disk.nvme0c0n1.out‚ 146911.703125* proc.memory.percent‚17.119598388671875 disk.sda2.out‚0" disk.nvme0n1.out‚ 146911.703125 proc.cpu.threads‚17) disk./.usagePercent‚37.93034610498259& disk.rbd0.in‚1.759218604441423e+13 disk.dm-0.out‚ 146911.703125 disk.sda1.in‚0$ memory_percent‚82.79039859771729 cpu‚25.58786912886302 disk.sda.in‚ 108.921875 disk.sda1.out‚0 disk.sda3.in‚ 108.921875µ’54¨2/ _wandbruntime‚20802  _runtime‚20802ôÄùvË©2Å $ _timestamp‚1.784310617894719e+09 _runtime‚20802.111873311 _step‚1031 ' lr-AdamW/pg1‚0.00021873569850551036 ' lr-AdamW/pg2‚0.00021873569850551036 trainer/global_step‚12549 fC{Ъ2Ê ' lr-AdamW/pg1‚0.00021873569850551036 ' lr-AdamW/pg2‚0.00021873569850551036 trainer/global_step‚12549 $ _timestamp‚1.784310617894719e+09 _runtime‚20802.111873311 _step‚1031‡A€0°4«2/ _wandbruntime‚20802  _runtime‚20802¥ù\¬2P =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3607664089628672 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 107081857 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 108688534 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11786857844529 $ _timestamp‚1.784310617895735e+09 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 107081857 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚67765163 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3619471149718528 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 trainer/global_step‚12549 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚3607664089628672 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚32195307 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚974138015485952 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15224866020352 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11785513244207 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 108698507 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 108688534 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚32195312 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11807060089856 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚67765417 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11805714700288 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 107081851 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚3617891494667823 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚35569856 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3617892864696113 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 107081851 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 215770385 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3619471149718528 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚989362513415168 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 108698507 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚3606106006851584 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚35570105 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚249 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚974137647394816 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3619469778155520 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3607664063455232 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 F@DeviceStatsMonitor.on_train_batch_startólj¹/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 _step‚1032 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚254 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15224900556800 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 _runtime‚20802.113054671 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚989362916042752 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504½Av(­2”P ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚989362916042752 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3607664089628672 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 107081857 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 108688534 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11786857844529 $ _timestamp‚1.784310617895735e+09 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 107081857 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚67765163 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3619471149718528 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 trainer/global_step‚12549 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚3607664089628672 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚32195307 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚974138015485952 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15224866020352 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11785513244207 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 108698507 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 108688534 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚32195312 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11807060089856 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚67765417 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11805714700288 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 107081851 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚3617891494667823 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚35569856 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3617892864696113 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 107081851 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 215770385 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3619471149718528 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚989362513415168 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 108698507 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚3606106006851584 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚35570105 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚249 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚974137647394816 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3619469778155520 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3607664063455232 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 _step‚1032 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚254 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15224900556800 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 _runtime‚20802.113054671 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0ˆÊ 4®2/ _wandbruntime‚20803  _runtime‚20803Yo '¯2šN ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11807985132544 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 % _timestamp‚1.7843106192913537e+09 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚32197850 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 215797062 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11786436710453 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3606383863201792 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 107090063 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3619750075491328 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 _step‚1033 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3607942090358784 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚35572846 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3618171669940647 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3606383888629760 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 108706999 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚253 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚35572598 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 _runtime‚20803.508764958 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚989441593701376 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 215787069 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚67770448 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11787781310887 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 108697012 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154’¸t%'°2ŸN BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚35572846 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3618171669940647 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3606383888629760 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 108706999 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚253 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚35572598 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 _runtime‚20803.508764958 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚989441593701376 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 215787069 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚67770448 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11787781310887 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 108697012 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11807985132544 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 % _timestamp‚1.7843106192913537e+09 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚32197850 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 215797062 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11786436710453 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3606383863201792 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 107090063 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3619750075491328 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 _step‚1033 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3607942090358784‰PWå4±2/ _wandbruntime‚20803  _runtime‚20803ЊEpO²2 0loss/FRt1/assignment_loss‚1.0133793497985264 /loss/FBt1/detectionŽ2¤ºÄ_loss‚0.6951431860094696 %loss/total_loss‚5.498278563551127 epoch‚31 % _timestamp‚1.7843106193728595e+09 _step‚1034 /loss/FRt1/detection_loss‚0.4855758730389095 /loss/FRt2/detection_loss‚1.2969324873674872 1loss/SRqqt1/detection_loss‚0.5540094145803273 1loss/SRqqt2/detection_loss‚0.5562646351090377 3loss/SRqqt1/assignment_loss‚0.21697658286210486 2loss/FBt1/assignment_loss‚0.000212914740773081 4loss/FBt2/assignment_loss‚1.4773175337998195e-05 0loss/FBt2/detection_loss‚0.25838160672779625 trainer/global_step‚12549 _runtime‚20803.589992824 1loss/FRt2/assignment_loss‚0.20899515740479094 3loss/SRqqt2/assignment_loss‚0.21239258273656583Î-z±³2’ /loss/FRt2/detection_loss‚1.2969324873674872 1loss/SRqqt1/detection_loss‚0.5540094145803273 1loss/SRqqt2/detection_loss‚0.5562646351090377 3loss/SRqqt1/assignment_loss‚0.21697658286210486 2loss/FBt1/assignment_loss‚0.000212914740773081 4loss/FBt2/assignment_loss‚1.4773175337998195e-05 0loss/FBt2/detection_loss‚0.25838160672779625 trainer/global_step‚12549 _runtime‚20803.589992824 1loss/FRt2/assignment_loss‚0.20899515740479094 3loss/SRqqt2/assignment_loss‚0.21239258273656583 0loss/FRt1/assignment_loss‚1.0133793497985264 /loss/FBt1/detection_loss‚0.6951431860094696 %loss/total_loss‚5.498278563551127 epoch‚31 % _timestamp‚1.7843106193728595e+09 _step‚1034 /loss/FRt1/detection_loss‚0.4855758730389095Š¸ &…”´2‚(:‰ ÜÖéÒú “÷ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚263.646' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.7292 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ö½©å?µ2‚(:´ ÜÖéÒ€µÂù disk.rbd3.in‚0 disk.sda2.out‚0$ disk.dm-0.out‚146929.49853515625$ memory_percent‚82.78665542602539' proc.memory.availableMB‚ 2820.234375* proc.memory.percent‚17.119598388671875) disk./.usagePercent‚37.93111310355517) disk.nvme0c0n1.out‚146929.49853515625 disk.sda.out‚ 1305.77734375 disk.sda1.out‚0 disk.sda3.in‚ 108.921875 proc.memory.rssMB‚2804.875' disk.nvme0n1.out‚146929.49853515625 disk.dm-1.out‚ 1305.77734375& disk.rbd0.in‚1.759218604441423e+13 disk.dm-0.in‚ 49206.5625 network.recv‚ 5.166992e+06 disk.drbd2165.out‚0 disk.nvme0n1.in‚ 49206.5625 disk.rbd1.in‚ 0.52392578125 disk.sda1.in‚0$ disk./.usageGB‚565.2009963989258 disk.rbd1.out‚ 54.2529296875 disk.sda.in‚ 108.921875 disk.dm-1.in‚ 108.921875 proc.cpu.threads‚17 network.sent‚ 2.4182528e+07 disk.drbd2165.in‚0 disk.sda2.in‚0 disk.sda3.out‚ 1305.77734375 cpu‚25.587961722277537 disk.rbd3.out‚ 0.4951171875 disk.nvme0c0n1.in‚ 49206.5625 disk.rbd0.out‚0åÔx§ž¶2‚(:“ ëÖéÒØ®èö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚267.11' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚53.422000000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0·ð ÑE·2‚(:º ëÖéÒ¸Êåø disk.sda3.in‚ 108.921875! disk.nvme0n1.in‚ 49386.1015625 disk.rbd0.out‚0 network.recv‚ 5.170906e+06 disk.rbd3.in‚0 disk.sda.in‚ 108.921875& disk.rbd0.in‚1.759218604441423e+13 proc.cpu.threads‚17 disk.dm-0.in‚ 49386.1015625( disk.nvme0c0n1.out‚146948.9853515625 disk.drbd2165.out‚0# disk.nvme0c0n1.in‚ 49386.1015625& disk.nvme0n1.out‚146948.9853515625$ memory_percent‚82.78701305389404 network.sent‚ 2.4245817e+07$ disk./.usageGB‚565.2078056335449# disk.dm-0.out‚146948.9853515625 disk.rbd3.out‚ 0.4951171875* proc.memory.percent‚17.119598388671875) disk./.usagePercent‚37.93157007700379 disk.sda3.out‚ 1306.10546875 disk.sda2.out‚0 disk.rbd1.out‚ 54.3935546875 disk.dm-1.in‚ 108.921875 disk.dm-1.out‚ 1306.10546875 cpu‚25.5880562168412 disk.sda1.in‚0 disk.drbd2165.in‚0 disk.sda2.in‚0 disk.rbd1.in‚ 0.52392578125) proc.memory.availableMB‚ 2820.17578125 disk.sda1.out‚0 disk.sda.out‚ 1306.10546875 proc.memory.rssMB‚2804.875=Q›ž¸2‚(:“ úÖéÒ·ÍÝö gpu.0.gpu‚99.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚282.255' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚56.45099999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0öx ÀJ¹2‚(:¿ úÖéÒº¢Äú& proc.memory.availableMB‚ 2820.40625* proc.memory.percent‚17.119598388671875 network.sent‚ 2.4250021e+07$ disk./.usageGB‚565.2079582214355 disk.sda.out‚ 1309.12890625$ disk.dm-0.out‚146963.10986328125$ disk.nvme0c0n1.in‚49386.32421875 disk.sda2.out‚0 disk.rbd0.out‚0 disk.sda3.in‚ 108.921875 disk.dm-1.out‚ 1309.12890625 disk.dm-0.in‚49386.32421875$ memory_percent‚82.78560638427734 proc.cpu.threads‚17 disk.rbd1.out‚ 54.4560546875& disk.rbd0.in‚1.759218604441423e+13 disk.rbd3.in‚0 proc.memory.rssMB‚2804.875 disk.rbd1.in‚ 0.52392578125 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.rbd3.out‚ 0.4951171875 network.recv‚ 5.17357e+06 disk.sda2.in‚0 disk.sda3.out‚ 1309.12890625 disk.drbd2165.out‚0) disk.nvme0c0n1.out‚146963.10986328125" disk.nvme0n1.in‚49386.32421875' disk.nvme0n1.out‚146963.10986328125 cpu‚25.588204265151123* disk./.usagePercent‚37.931580317305155 disk.sda.in‚ 108.921875 disk.dm-1.in‚ 108.921875 disk.sda1.in‚0ÐZ~ ”º2‚(:‰ ‰×éÒŽìö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚304.221' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚60.8442 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚05§ÁJ»2‚(:¿ ‰×éÒ¥ÇÑù disk.sda.out‚ 1309.77734375& disk.rbd0.in‚1.759218604441423e+13 disk.dm-0.in‚49386.52734375 disk.sda3.in‚ 108.921875 disk.rbd3.out‚ 0.4951171875) proc.memory.availableMB‚ 2820.41015625 disk.sda3.out‚ 1309.77734375) disk./.usagePercent‚37.93204420295721* proc.memory.percent‚17.119598388671875 disk.sda1.in‚0 disk.rbd3.in‚0$ disk.nvme0c0n1.in‚49386.52734375 network.recv‚ 5.176916e+06 disk.drbd2165.in‚0" disk.nvme0n1.in‚49386.52734375& disk.nvme0n1.out‚146975.4833984375 disk.sda2.out‚0 proc.memory.rssMB‚2804.875$ disk./.usageGB‚565.2148704528809 disk.rbd0.out‚0# disk.dm-0.out‚146975.4833984375 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda.in‚ 108.921875$ memory_percent‚82.78558254241943 network.sent‚ 2.4255209e+07 disk.dm-1.in‚ 108.921875 disk.drbd2165.out‚0 disk.rbd1.in‚ 0.52392578125 disk.rbd1.out‚ 54.4560546875 disk.dm-1.out‚ 1309.77734375( disk.nvme0c0n1.out‚146975.4833984375 cpu‚25.588274819009737 proc.cpu.threads‚17 ò”¼2‚(:‰ ˜×éÒÐÎæö gpu.0.gpu‚90.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚247.797' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚49.5594 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‚¶ñ@½2‚(:µ ˜×éÒîˆù network.recv‚ 5.180563e+06" disk.nvme0n1.in‚49386.94921875) disk./.usagePercent‚37.93249324017226 disk.sda3.in‚ 108.921875 proc.cpu.threads‚17$ disk./.usageGB‚565.2215614318848 disk.sda.out‚ 1309.99609375 disk.sda1.in‚0 disk.sda3.out‚ 1309.99609375 proc.memory.rssMB‚2804.875* proc.memory.percent‚17.119598388671875 disk.dm-1.in‚ 108.921875 disk.rbd0.out‚0 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.rbd3.out‚ 0.4951171875$ disk.nvme0c0n1.in‚49386.94921875% disk.nvme0c0n1.out‚146988.9140625# disk.nvme0n1.out‚146988.9140625& disk.rbd0.in‚1.759218604441423e+13 disk.sda2.in‚0 disk.sda2.out‚0 cpu‚25.588380303626533 network.sent‚ 2.426033e+07 disk.rbd1.out‚ 54.5185546875 disk.sda.in‚ 108.921875 disk.dm-0.in‚49386.94921875 disk.dm-0.out‚146988.9140625) proc.memory.availableMB‚ 2820.10546875 disk.rbd1.in‚ 0.52392578125 disk.dm-1.out‚ 1309.99609375 disk.sda1.out‚0 disk.rbd3.in‚0$ memory_percent‚82.78744220733643q’éË”¾2‚(:‰ §×éÒ¤ö¨÷ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚277.374' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.4748 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0OñØ¢G¿2‚(:¼ §×éÒæÔáú disk.dm-1.out‚ 1310.27734375 disk.dm-0.in‚49589.65234375 disk.sda3.in‚ 108.921875 disk.drbd2165.out‚0& disk.rbd0.in‚1.759218604441423e+13 disk.sda3.out‚ 1310.27734375$ disk.nvme0c0n1.in‚49589.65234375$ memory_percent‚82.78450965881348 network.sent‚ 2.4265451e+07 disk.rbd0.out‚0 disk.drbd2165.in‚0 disk.rbd3.in‚0 disk.sda2.in‚0 disk.rbd1.in‚ 0.52392578125 disk.dm-1.in‚ 108.921875( disk.nvme0c0n1.out‚147000.6259765625 disk.sda1.out‚0 disk.sda.in‚ 108.921875 disk.sda.out‚ 1310.27734375 disk.sda1.in‚0 cpu‚25.588456245537493 network.recv‚ 5.1839e+06) disk./.usagePercent‚37.93172291350173$ disk./.usageGB‚565.2100830078125( proc.memory.availableMB‚ 2820.5859375* proc.memory.percent‚17.119598388671875# disk.dm-0.out‚147000.6259765625 disk.rbd3.out‚ 0.4951171875" disk.nvme0n1.in‚49589.65234375 disk.sda2.out‚0 disk.rbd1.out‚ 54.5966796875 proc.memory.rssMB‚2804.875 proc.cpu.threads‚17& disk.nvme0n1.out‚147000.6259765625#jõ=4À2/ _wandbruntime‚20881  _runtime‚208817q—ùÌÁ2Æ ' lr-AdamW/pg1‚0.00021616167379123183 ' lr-AdamW/pg2‚0.00021616167379123183 trainer/global_step‚12599 % _timestamp‚1.7843106976087844e+09 _runtime‚20881.825924499 _step‚1035áq¶àÑÂ2Ë ' lr-AdamW/pg1‚0.00021616167379123183 ' lr-AdamW/pg2‚0.00021616167379123183 trainer/global_step‚12599 % _timestamp‚1.7843106976087844e+09 _runtime‚20881.825924499 _step‚1035‹×xǸ4Ã2/ _wandbruntime‚20881  _runtime‚20881) òª(Ä2P O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 216606437 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚3621564099960832 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 107492151 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚3621564126134272 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3633416130591744 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚3631831820689731 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11853376020480 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 $ _timestamp‚1.784310697609788e+09 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚32322462 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15284789409792 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 107492157 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚993297125279744 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 _step‚1036 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11833094957637 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3633417502154752 _runtime‚20881.827105417 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚68030237 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11852030630912 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 107492151 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚68030491 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚3633416130591744 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11852030630912 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚3620000095760384 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚3620000070332416 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 107492157 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11853376020480 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚35708029 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚978012335869952 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968­_ƒ(Å2”P GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 216606437 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚3621564099960832 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 107492151 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚3621564126134272 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3633416130591744 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚3631831820689731 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11853376020480 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 $ _timestamp‚1.784310697609788e+09 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚32322462 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15284789409792 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 107492157 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚993297125279744 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 _step‚1036 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11833094957637 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3633417502154752 _runtime‚20881.827105417 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚68030237 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11852030630912 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 107492151 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚68030491 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚3633416130591744 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11852030630912 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚3620000095760384 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚3620000070332416 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 107492157 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11853376020480 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚35708029 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚978012335869952 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0ŒÛ)*æ4Æ2/ _wandbruntime‚20883  _runtime‚20883-•©¤÷Ç2™N >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚32325005 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 trainer/global_step‚12599 _runtime‚20883.212812807 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11852955666432 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 $ _timestamp‚1.784310698995425e+09 TEDeviceStatsMšB?(#onitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚68035746 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚993375806157824 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15285983880192 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚68035499 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15285949350912 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 216633114 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 107500363 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11854301063168 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 109122764 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3632111995962555 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 109132751 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3633695056357376 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚35710499 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚3632110625934153 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 216623121 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 216623121 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 109132751 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11852955666432 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 216633114 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3620277952110592 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚3621842126864384 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚68035746 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚993375806157824 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15285983880192 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚68035499 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15285949350912 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 216633114 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 107500363 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11854301063168 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 109122764 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3632111995962555 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 109132751 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3633695056357376 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚35710499 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚3632110625934153 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 216623121 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 216623121 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 109132751 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11852955666432 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 216633114 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3620277952110592 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚3621842126864384 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚32325005DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 107902457 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11899691951104 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 217452468 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3645773516739929 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚3635464136466432 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚997231608843264 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 400530432 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 109550011 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3647363854590976 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3647362483027968 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15344952589312 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 trainer/global_step‚12649 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚981886288162816 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚236 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3635464136466432 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11879332070745 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11898346561536 =7DeviceStatsMonitor.on_train_batch_ýW:î­start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 107902457 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11899691951104 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 217452468 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3645773516739929 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚3635464136466432 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚997231608843264 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 400530432 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 109550011 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3647363854590976 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3647362483027968 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15344952589312 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 trainer/global_step‚12649 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚981886288162816 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚236 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3635464136466432 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11879332070745 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11898346561536 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚226 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚3635742137196544 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚32432128 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 109558503 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3646052321984463 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11878910936669 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚981963774570496 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15346129643520 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 107910657 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11899271597056 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 109548516 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚12649 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 217459173 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚35848905 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚981964142661632 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 217469166 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚68300829 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚35848679 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11899271597056 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3647642780363776 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3635742137196544 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 217469166ƒ”ÃqÁÞ2ŸN PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚3635742163369984 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11900616993792 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚12649 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 217459173 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚35848905 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚981964142661632 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 217469166 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚68300829 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚35848679 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11899271597056 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3647642780363776 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3635742137196544 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 H@DeviceStatsMonitor.on_tr*B "dain_batch_end/segment.small_pool.current‚820 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 217469166 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 107910663 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 107910657 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚32452155 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 107910663 QDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚226 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚3635742137196544 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚32432128 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 109558503 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3646052321984463 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11878910936669 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚981963774570496 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15346129643520 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 107910657 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11899271597056 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 109548516 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BŸä2‚(:” ØéÒ°§ñö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚41.0 gpu.0.powerWatts‚299.276' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚59.855199999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0“ð‡PFå2‚(:» ØéÒäãÜù disk.rbd3.out‚ 0.4951171875$ disk.nvme0c0n1.in‚49788.58984375$ memory_percent‚82.79151916503906 disk.sda2.in‚0$ disk.dm-0.out‚148236.93408203125) disk.nvme0c0n1.out‚148236.93408203125 cpu‚25.589183006068595 disk.dm-1.in‚ 108.921875 disk.dm-1.out‚ 1312.3359375 disk.sda3.out‚ 1312.3359375 disk.rbd3.in‚0 disk.dm-0.in‚49788.58984375 disk.sda1.out‚0 proc.memory.rssMB‚2804.875 proc.cpu.threads‚17 network.sent‚ 2.4417322e+07 network.recv‚ 5.209907e+06) disk./.usagePercent‚37.94114603882205" disk.nvme0n1.in‚49788.58984375 disk.rbd1.out‚ 54.8466796875& disk.rbd0.in‚1.759218604441423e+13 disk.drbd2165.out‚0% proc.memory.availableMB‚ 2819.4375' disk.nvme0n1.out‚148236.93408203125 disk.sda2.out‚0 disk.sda.in‚ 108.921875 disk.sda.out‚ 1312.3359375 disk.sda1.in‚0 disk.sda3.in‚ 108.921875* proc.memory.percent‚17.119598388671875$ disk./.usageGB‚565.3504943847656 disk.rbd1.in‚ 0.52392578125 disk.drbd2165.in‚0 disk.rbd0.out‚07õm”æ2‚(:‰ ŸØéÒ±Õæö gpu.0.gpu‚90.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚281.638' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚56.3276 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0’Å•.ç2‚(:£ ŸØéÒú„”ù" disk.nvme0c0n1.in‚ 50019.140625# disk.nvme0n1.out‚148252.3984375 disk.sda2.in‚0 disk.rbd1.out‚ 54.8466796875 disk.dm-0.in‚ 50019.140625 network.recv‚ 5.212867e+06 disk.rbd0.out‚0$ memory_percent‚82.79128074645996( proc.memory.availableMB‚ 2819.4765625 cpu‚25.58927535776228 disk.sda3.in‚ 108.921875 disk.sda3.out‚ 1312.8125 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd2165.in‚0 proc.memory.rssMB‚2804.875* disk./.usagePercent‚37.941517249746695% disk.nvme0c0n1.out‚148252.3984375 disk.dm-1.in‚ 108.921875& disk.rbd0.in‚1.759218604441423e+13* proc.memory.percent‚17.119598388671875 disk.nvme0n1.in‚ 50019.140625 disk.rbd1.in‚ 0.52392578125 disk.dm-0.out‚148252.3984375 disk.drbd2165.out‚0 network.sent‚ 2.4422949e+07$ disk./.usageGB‚565.3560256958008 disk.sda.in‚ 108.921875 disk.sda.out‚ 1312.8125 disk.dm-1.out‚ 1312.8125 disk.rbd3.in‚0 disk.rbd3.out‚ 0.4951171875 proc.cpu.threads‚17 disk.sda2.out‚0fÖhkŸè2‚(:” ®ØéÒüÑàö gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚298.104' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚59.620799999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0& ¹–Aé2‚(:¶ ®ØéÒèщù network.recv‚ 5.216436e+06 disk.dm-1.out‚ 1315.81640625 disk.dm-0.in‚ 50019.359375" disk.nvme0c0n1.in‚ 50019.359375 disk.rbd1.in‚ 0.52392578125& disk.rbd0.in‚1.759218604441423e+13 disk.sda1.out‚0' disk.nvme0c0n1.out‚148271.458984375$ memory_percent‚82.79225826263428 cpu‚25.589330763870866" disk.dm-0.out‚148271.458984375 disk.drbd2165.in‚0% disk.nvme0n1.out‚148271.458984375 disk.sda2.out‚0 network.sent‚ 2.4428029e+07 disk.sda.out‚ 1315.81640625 disk.rbd0.out‚0 disk.sda1.in‚0 disk.sda3.out‚ 1315.81640625) disk./.usagePercent‚37.94185338763915 disk.sda.in‚ 108.921875 disk.sda3.in‚ 108.921875 disk.nvme0n1.in‚ 50019.359375 disk.sda2.in‚0) proc.memory.availableMB‚ 2819.31640625 proc.memory.rssMB‚2804.875 proc.cpu.threads‚17 disk.rbd3.in‚0* proc.memory.percent‚17.119598388671875 disk.rbd3.out‚ 0.4951171875 disk.rbd1.out‚ 54.9091796875$ disk./.usageGB‚565.3610343933105 disk.dm-1.in‚ 108.921875 disk.drbd2165.out‚0!Òj”ê2‚(:‰ ½ØéÒ×õúö gpu.0.gpu‚90.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚290.327' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚58.0654 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0`Õ™@ë2‚(:µ ½ØéÒÆÖàù disk.sda3.out‚ 1318.1796875 disk.drbd2165.out‚0) disk.nvme0c0n1.out‚148289.84521484375 disk.dm-1.in‚ 108.921875) proc.memory.availableMB‚ 2819.31640625 disk.sda1.out‚0$ disk./.usageGB‚565.3702507019043 disk.rbd1.out‚ 54.9091796875* proc.memory.percent‚17.119598388671875 proc.cpu.threads‚17 network.sent‚ 2.4433135e+07 proc.memory.rssMB‚2804.875 disk.sda2.out‚0 disk.sda.out‚ 1318.1796875& disk.rbd0.in‚1.759218604441423e+13 disk.sda3.in‚ 108.921875$ memory_percent‚82.79225826263428 cpu‚25.589443547974387 disk.sda.in‚ 108.921875$ disk.dm-0.out‚148289.84521484375 disk.drbd2165.in‚0 disk.rbd3.in‚0! disk.nvme0c0n1.in‚ 50019.78125 disk.rbd1.in‚ 0.52392578125 disk.dm-0.in‚ 50019.78125 disk.rbd3.out‚ 0.4951171875 disk.nvme0n1.in‚ 50019.78125 disk.sda1.in‚0 disk.sda2.in‚0' disk.nvme0n1.out‚148289.84521484375 network.recv‚ 5.21972e+06) disk./.usagePercent‚37.94247190184188 disk.dm-1.out‚ 1318.1796875 disk.rbd0.out‚0qÿù4ì2/ _wandbruntime‚21040  _runtime‚21040u¤¢lËí2Å $ _timestamp‚1.784310856150749e+09 _runtime‚21040.367964531 _step‚1043 ' lr-AdamW/pg1‚0.00021103879237089961 ' lr-AdamW/pg2‚0.00021103879237089961 trainer/global_step‚126994BÁÐî2Ê _runtime‚21040.367964531 _step‚1043 ' lr-AdamW/pg1‚0.00021103879237089961 ' lr-AdamW/pg2‚0.00021103879237089961 trainer/global_step‚12699 $ _timestamp‚1.784310856150749e+09“êcÍ|4ï2/ _wandbruntime‚21040  _runtime‚21040Èk Ó(ð2’P S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3661308835464192 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚3649364199145472 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11944662492160 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11925569183853 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1001165971187712 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚985760608546816 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚68561478 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚35984721 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3649364172972032 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 108312751 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11924224583531 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3661310207027200 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 218278541 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1001165568560128 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15404994549760 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3649364199145472 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 108312757 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚35984970 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3659713842761837 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 108312751 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚249 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15404960013312 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 trainer/global_step‚12699 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3661308835464192 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚3649364199145472 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11944662492160 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11925569183853 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1001165971187712 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚985760608546816 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚68561478 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚35984721 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3649364172972032 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 108312751 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11924224583531 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3661310207027200 E;DeviceStatsMonitor.on_tr…6c9Àain_batch_start/allocation.all.peak‚12154 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 218278541 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1001165568560128 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15404994549760 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3649364199145472 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 108312757 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚35984970 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0”™c24ò2/ _wandbruntime‚21041  _runtime‚21041r TÉ"'ó2œN OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 108320963 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11945587527680 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚68566774 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚3649642173702144 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚35987717 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚3659991277977969 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 109974268 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1001244254196736 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 % _timestamp‚1.7843108575381734e+09 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3661589132800000 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 108320957 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 109984255 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11946932924416 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 108320963 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3648066155356160 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 218295225 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3661589132800000 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15406159242240 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚985838463045632 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11925148049777 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚248 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 _step‚1045 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 109984255 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚12699 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 218305218 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11946932924416 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚243 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3649642173702144 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3649642199875584 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 109974268 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11926492650211 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚3649642199875584 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15406193771520 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚32579305 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 108320957 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3661587761229824 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚68567022 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚985838094954496 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3659992648006371 _runtime‚21041.755537322 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11945587527680 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 218305218 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0ü¥''ô2¡N OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 108320957 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 109984255 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11946932924416 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 108320963 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3648066155356160 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 218295225 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3661589132800000 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15406159242240 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚985838463045632 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11925148049777 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚248 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 _step‚1045 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 109984255 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚12699 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 218305218 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11946932924416 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚243 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3649642173702144 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3649642199875584 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 109974268 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11926492650211 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚3649642199875584 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15406193771520 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚32579305 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 108320957 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3661587761229824 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚68567022 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚985838094954496 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3659992648006371 _runtime‚21041.755537322 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11945587527680 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 218305218 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 108320963 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11945587527680 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚68566774 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚3649642173702144 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚35987717 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚3659991277977969 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 109974268 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1001244254196736 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 % _timestamp‚1.7843108575381734e+09 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3661589132800000•£³ª)4õ2/ _wandbruntime‚21041  _runtime‚2104179S ö2Œ 0loss/FRt1/assignment_loss‚1.0420727460468464 /loss/FRt2/assignment_loss‚0.201952503008001 3loss/SRqqt1/assignment_loss‚0.12140316799059106 1loss/SRqqt1/detection_loss‚0.5715841321343159 _runtime‚21041.84084301 _step‚1046 2loss/FBt2/assignment_loss‚0.002762320451395403 0loss/FRt1/detection_loss‚0.46425661170557386 2loss/SRqqt2/detection_loss‚0.44156835995761684 0loss/FBt2/detection_loss‚0.23705642228255389 %loss/total_loss‚5.008465285625141 0loss/FBt1/detection_loss‚0.40770185393034253 epoch‚31 3loss/SRqqt2/assignment_loss‚0.25329243936143003 3loss/FBt1/assignment_loss‚0.0017634926964335618 /loss/FRt2/detection_loss‚1.2630512360600394 trainer/global_step‚12699 % _timestamp‚1.7843108576238935e+09R ÷2‘ 3loss/SRqqt1/assignment_loss‚0.12140316799059106 1loss/SRqqt1/detection_loss‚0.5715841321343159 _runtime‚21041.84084301 _step‚1046 2loss/FBt2/assignment_loss‚0.002762320451395403 0loss/FRt1/detection_loss‚0.46425661170557386 2loss/SRqqt2/detection_loss‚0.44156835995761684 0loss/FBt2/detection_loss‚0.23705642228255389 %loss/total_loss‚5.008465285625141 0loss/FBt1/detection_loss‚0.40770185393034253 epoch‚31 3loss/SRqqt2/assignment_loss‚0.25329243936143003 3loss/FBt1/assignment_loss‚0.0017634926964335618 /loss/FRt2/detection_loss‚1.2630512360600394 trainer/global_step‚12699 % _timestamp‚1.7843108576238935e+09 0loss/FRt1/assignment_loss‚1.0420727460468464 /loss/FRt2/assignment_loss‚0.201952503008001–”Ûû”ø2‚(:‰ ÌØéÒíþ£÷ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚286.551' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚57.3102 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0;¯˜¿Fù2‚(:» ÌØéÒÐÚèù disk.rbd1.in‚ 0.52392578125 network.recv‚ 5.223232e+06 disk.dm-0.in‚ 50019.984375 disk.sda1.in‚0 disk.sda1.out‚0' disk.nvme0n1.out‚148302.73095703125 cpu‚25.589498310115793 disk.sda2.in‚0 disk.sda.in‚ 108.921875 disk.dm-1.in‚ 108.921875 disk.rbd0.out‚0$ disk.dm-0.out‚148302.73095703125 disk.sda3.in‚ 108.921875 disk.sda3.out‚ 1318.58984375' proc.memory.availableMB‚ 2819.796875 disk.drbd2165.out‚0 disk.rbd3.out‚ 0.4951171875* proc.memory.percent‚17.119598388671875 disk.sda2.out‚0$ memory_percent‚82.78932571411133 disk.rbd1.out‚ 54.9716796875 network.sent‚ 2.4438308e+07* disk./.usagePercent‚37.942758630280224& disk.rbd0.in‚1.759218604441423e+13 proc.cpu.threads‚17" disk.nvme0c0n1.in‚ 50019.984375 disk.nvme0n1.in‚ 50019.984375 disk.sda.out‚ 1318.58984375 disk.rbd3.in‚0$ disk./.usageGB‚565.3745231628418 disk.drbd2165.in‚0) disk.nvme0c0n1.out‚148302.73095703125 proc.memory.rssMB‚2804.875 disk.dm-1.out‚ 1318.58984375ÑÚÂÉ“ú2‚(:ˆ ÛØéÒ߀÷ gpu.0.gpu‚99.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚289.215' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚57.843 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0?Þ/DFû2‚(:» ÛØéÒ‡µ·ù network.sent‚ 2.4501813e+07 disk.rbd3.out‚ 0.4951171875" disk.nvme0n1.in‚50110.96484375 disk.sda2.out‚0 disk.sda1.in‚0 disk.sda3.out‚ 1318.91796875$ disk./.usageGB‚565.3791580200195$ disk.nvme0c0n1.in‚50110.96484375 disk.sda2.in‚0 disk.rbd1.in‚ 0.52392578125 disk.dm-1.out‚ 1318.91796875) proc.memory.availableMB‚ 2820.28515625* proc.memory.percent‚17.119598388671875 disk.sda.in‚ 108.921875 disk.dm-1.in‚ 108.921875 disk.sda1.out‚0 cpu‚25.58950676064913% disk.nvme0n1.out‚148314.775390625 disk.drbd2165.out‚0 proc.cpu.threads‚17) disk./.usagePercent‚37.94306967943433 disk.rbd0.out‚0$ memory_percent‚82.78634548187256 disk.drbd2165.in‚0' disk.nvme0c0n1.out‚148314.775390625 disk.sda.out‚ 1318.91796875 disk.sda3.in‚ 108.921875 network.recv‚ 5.227443e+06 disk.dm-0.in‚50110.96484375" disk.dm-0.out‚148314.775390625 disk.rbd3.in‚0 disk.rbd1.out‚ 54.9716796875& disk.rbd0.in‚1.759218604441423e+13 proc.memory.rssMB‚2804.875—ËbF”ü2‚(:‰ êØéÒæ¼ù gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚295.976' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚59.1952 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¤–öGý2‚(:¼ êØéÒˆ„‡ú proc.cpu.threads‚17 disk.rbd3.in‚0 disk.nvme0n1.in‚ 50194.109375 proc.memory.rssMB‚2804.875 disk.rbd3.out‚ 0.4951171875 disk.sda2.in‚0 disk.sda.out‚ 1320.21484375 disk.dm-1.out‚ 1320.21484375* proc.memory.percent‚17.119598388671875 network.recv‚ 5.230705e+06& disk.rbd0.in‚1.759218604441423e+13 disk.sda1.out‚0 disk.drbd2165.out‚0 disk.rbd1.in‚ 0.52392578125 network.sent‚ 2.4506888e+07 disk.dm-0.in‚ 50194.109375 disk.sda1.in‚0 disk.sda3.in‚ 108.921875) disk.nvme0c0n1.out‚148323.85791015625 disk.sda2.out‚0 disk.rbd0.out‚0 disk.drbd2165.in‚0" disk.nvme0c0n1.in‚ 50194.109375) disk./.usagePercent‚37.94344447446446 disk.sda.in‚ 108.921875 disk.dm-1.in‚ 108.921875$ disk.dm-0.out‚148323.85791015625 disk.sda3.out‚ 1320.21484375' disk.nvme0n1.out‚148323.85791015625 disk.rbd1.out‚ 55.0341796875$ memory_percent‚82.78787136077881) proc.memory.availableMB‚ 2820.03515625$ disk./.usageGB‚565.3847427368164 cpu‚25.589346762557142ç|¢6”þ2‚(:‰ ùØéÒÀ…‚÷ gpu.0.gpu‚90.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚276.276' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.2552 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0yàU¨Hÿ2‚(:½ ùØéÒ§ñŸù) proc.memory.availableMB‚ 2819.79296875* proc.memory.percent‚17.119598388671875 network.recv‚ 5.234219e+06 disk.drbd2165.in‚0" disk.nvme0n1.in‚50194.51953125 disk.sda3.in‚ 108.921875' disk.nvme0n1.out‚148339.52099609375 disk.sda2.in‚0 disk.sda2.out‚0 proc.memory.rssMB‚2804.875 proc.cpu.threads‚17) disk./.usagePercent‚37.94387379909938$ disk./.usageGB‚565.3911399841309 disk.dm-1.out‚ 1321.0703125 disk.rbd1.in‚ 0.52392578125$ disk.dm-0.out‚148339.52099609375 disk.sda1.in‚0 disk.sda3.out‚ 1321.0703125) disk.nvme0c0n1.out‚148339.52099609375 disk.drbd2165.out‚0 disk.sda.in‚ 108.921875 disk.dm-1.in‚ 108.921875& disk.rbd0.in‚1.759218604441423e+13 disk.rbd0.out‚0$ disk.nvme0c0n1.in‚50194.51953125 cpu‚25.58921428114474 disk.sda.out‚ 1321.0703125 disk.dm-0.in‚50194.51953125 disk.rbd3.in‚0 network.sent‚ 2.451158e+07 disk.sda1.out‚0 disk.rbd3.out‚ 0.4951171875 disk.rbd1.out‚ 55.0341796875$ memory_percent‚82.78934955596924éZH`”€3‚(:‰ ˆÙéÒË‚×÷ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚39.0 gpu.0.powerWatts‚110.656' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.1312 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¤CeˆI3‚(:¾ ˆÙéÒ™óÑú disk.sda1.out‚0 disk.drbd2165.in‚0) proc.memory.availableMB‚ 2820.03515625 proc.memory.rssMB‚2804.875 proc.cpu.threads‚17 network.sent‚ 2.4517193e+07" disk.nvme0n1.in‚50194.73828125& disk.rbd0.in‚1.759218604441423e+13 disk.drbd2165.out‚0 disk.rbd3.in‚0 disk.rbd3.out‚ 0.4951171875 disk.sda2.out‚0) disk.nvme0c0n1.out‚148374.64501953125 disk.sda3.in‚ 108.921875 disk.sda3.out‚ 1322.0703125 disk.rbd1.in‚ 0.52392578125 disk.rbd1.out‚ 55.0966796875 disk.sda.in‚ 108.921875 disk.dm-1.out‚ 1322.0703125 disk.rbd0.out‚0$ disk.dm-0.out‚148374.64501953125 network.recv‚ 5.237777e+06) disk./.usagePercent‚37.99591782674262' disk.nvme0n1.out‚148374.64501953125 disk.sda.out‚ 1322.0703125 disk.sda1.in‚0$ memory_percent‚82.78787136077881# disk./.usageGB‚566.166633605957 disk.dm-0.in‚50194.73828125* proc.memory.percent‚17.119598388671875$ disk.nvme0c0n1.in‚50194.73828125 disk.dm-1.in‚ 108.921875 cpu‚25.589052781297713 disk.sda2.in‚0«}X4‚3/ _wandbruntime‚21119  _runtime‚21119˜|kÑʃ3Ä % _timestamp‚1.7843109351633236e+09 _runtime‚21119.380525509 _step‚1047 & lr-AdamW/pg1‚0.0002084902646979465 & lr-AdamW/pg2‚0.0002084902646979465 trainer/global_step‚12749B ëCÏ„3É % _timestamp‚1.7843109351633236e+09 _runtime‚21119.380525509 _step‚1047 & lr-AdamW/pg1‚0.0002084902646979465 & lr-AdamW/pg2‚0.0002084902646979465 trainer/global_step‚12749—‘­1C4…3/ _wandbruntime‚21119  _runtime‚21119ȵe„†3’P M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚68826079 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚3661682337058816 _step‚1048 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11990978422784 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚3661682362486784 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3675255187900416 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 108723057 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚36122172 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3673654168783745 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 110391542 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1005100122486784 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11992323812352 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11992323812352 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 108723051 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15464825464832 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚3673652798755455 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚36122424 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11990978422784 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 108723057 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 108723051 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚32703912 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15464790928384 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 110401515 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚3675255187900416 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11971806296961 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚3663264209477632 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚12749 % _timestamp‚1.7843109351644757e+09 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚989635297021952 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚257 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚68826336 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 110391542 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚252 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚989634928930816 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚3663264235651072 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 219124572 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 219114593 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3663264235651072 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚32703907 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1005099719859200 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 _runtime‚21119.384944516 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3675256559463424 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3675256559463424@-š‘(‡3—P JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚36122172 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3673654168783745 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 110391542 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1005100122486784 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11992323812352 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11992323812352 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 108723051 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15464825464832 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚3673652798755455 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚36122424 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11990978422784 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 108723057 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 108723051 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚32703912 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15464790928384 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 110401515 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚3675255187900416 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11971806296961 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚3663264209477632 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚12749 % _timestamp‚1.7843109351644757e+09 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚989635297021952 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚257 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚68826336 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 110391542 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚252 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚989634928930816 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚3663264235651072 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 219124572 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 219114593 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3663264235651072 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚32703907 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1005099719859200 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 _runtime‚21119.384944516 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3675256559463424 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3675256559463424 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚68826079 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚3661682337058816 _step‚1048 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11990978422784 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚3661682362486784 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3675255187900416 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 108723057 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820˜µw¨`”ˆ3‚(:‰ —ÙéÒˆØþö gpu.0.gpu‚99.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚288.968' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚57.7936 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‚AH‹K‰3‚(:À —ÙéÒâÍ“ú) disk.nvme0c0n1.out‚148397.48876953125 disk.rbd0.out‚0 disk.sda3.in‚ 108.921875 disk.drbd2165.out‚0' disk.nvme0n1.out‚148397.48876953125 disk.sda.in‚ 108.921875 disk.dm-1.out‚ 1325.03515625 disk.sda1.in‚0$ memory_percent‚82.78937339782715 proc.cpu.threads‚17$ disk./.usageGB‚566.1511497497559" disk.nvme0n1.in‚50194.94140625 disk.sda2.in‚0$ disk.dm-0.out‚148397.48876953125 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.rbd3.in‚0( proc.memory.availableMB‚ 2819.7890625 network.recv‚ 5.240883e+06) disk./.usagePercent‚37.99487869216114 disk.sda2.out‚0 disk.rbd1.in‚ 0.52392578125& disk.rbd0.in‚1.759218604441423e+13 disk.dm-0.in‚50194.94140625 disk.rbd3.out‚ 0.4951171875 proc.memory.rssMB‚2804.875$ disk.nvme0c0n1.in‚50194.94140625 disk.sda3.out‚ 1325.03515625* proc.memory.percent‚17.119598388671875 network.sent‚ 2.4522274e+07 disk.rbd1.out‚ 55.0966796875 disk.dm-1.in‚ 108.921875 cpu‚25.58893169990987 disk.sda.out‚ 1325.03515625¹2ùn4Š3/ _wandbruntime‚21120  _runtime‚21120Ét¸!'‹3›N LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 108731263 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚32706455 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11972729763319 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚68831362 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 _step‚1049 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15466019853312 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3663542210207744 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 108731263 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 110400020 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3661960218836992 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3675534113666048 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 _runtime‚21120.773317248 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 110410007 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 108731257 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚3663542236381184 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 219141270 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3675535485236224 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11991903458304 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 110410007 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚3663542210207744 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚989712783429632 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚32706450 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11971385162885 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 219131277 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15465985324032 trainer/global_step‚12749 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11993248855040 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚3673931603999877 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11993248855040 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚68831619 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚257 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 108731257 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820ÈÇç·êŒ3 N QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚68831362 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 _step‚1049 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15466019853312 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3663542210207744 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 108731263 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 110400020 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 F=DeviceStatsMonitor.on_train_Ïi0<batch_end/segment.large_pool.peak‚1175 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3661960218836992 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3675534113666048 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 _runtime‚21120.773317248 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 110410007 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 108731257 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚3663542236381184 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 219141270 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3675535485236224 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11991903458304 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 110410007 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚3663542210207744 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚989712783429632 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚32706450 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11971385162885 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 219131277 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15465985324032 trainer/global_step‚12749 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11993248855040 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚3673931603999877 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11993248855040 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚68831619 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚257 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 108731257 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 108731263 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚32706455 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11972729763319 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0™ƒâÒu43/ _wandbruntime‚21120  _runtime‚21120!ÈÀŽ3‰ 1loss/FRt2/assignment_loss‚0.24766704332914283 3loss/FBt1/assignment_loss‚0.0006321838787522745 0loss/FBt2/detection_loss‚0.22199276056246667 % _timestamp‚1.7843109366386733e+09 0loss/FRt1/assignment_loss‚1.0848747408627784 2loss/SRqqt1/assignment_loss‚0.1381762275501556 2loss/FBt2/assignment_loss‚0.017338649267793087 /loss/FRt1/detection_loss‚0.4637593434410807 /loss/FRt2/detection_loss‚1.3379066140718683 /loss/FBt1/detection_loss‚0.3160994947955604 epoch‚31 _runtime‚21120.85564676 3loss/SRqqt2/assignment_loss‚0.23094935264338862 1loss/SRqqt1/detection_loss‚0.5748014740343701 %loss/total_loss‚5.119554283785964 _step‚1050 0loss/SRqqt2/detection_loss‚0.485356399348608 trainer/global_step‚127498àÍ3Ž 1loss/FRt2/assignment_loss‚0.24766704332914283 3loss/FBt1/assignment_loss‚0.0006321838787522745 0loss/FBt2/detection_loss‚0.22199276056246667 % _timestamp‚1.7843109366386733e+09 0loss/FRt1/assignment_loss‚1.0848747408627784 2loss/SRqqt1/assignment_loss‚0.1381762275501556 2loss/FBt2/assignment_loss‚0.017338649267793087 /loss/FRt1/detection_loss‚0.4637593434410807 /loss/FRt2/detection_loss‚1.3379066140718683 /loss/FBt1/detection_loss‚0.3160994947955604 epoch‚31 _runtime‚21120.85564676 3loss/SRqqt2/assignment_loss‚0.23094935264338862 1loss/SRqqt1/detection_loss‚0.5748014740343701 %loss/total_loss‚5.119554283785964 _step‚1050 0loss/SRqqt2/detection_loss‚0.485356399348608 trainer/global_step‚12749šÇÜr ž3‚(:“ ¦ÙéÒ¬œ“÷ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚289.405' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚57.88099999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ɨG‘3‚(:¼ ¦ÙéÒ¬Åú disk.rbd3.out‚ 0.4951171875 proc.memory.rssMB‚2804.875 cpu‚25.588761346670974) disk./.usagePercent‚37.94541368441784$ disk./.usageGB‚565.4140853881836 disk.drbd2165.in‚0" disk.nvme0n1.in‚50368.94140625 disk.rbd1.out‚ 55.1591796875 disk.sda1.in‚0 disk.sda3.out‚ 1325.78515625 disk.sda.in‚ 108.921875 disk.sda.out‚ 1325.78515625 disk.rbd0.out‚0" disk.dm-0.out‚148862.123046875% disk.nvme0n1.out‚148862.123046875 disk.sda3.in‚ 108.921875 disk.drbd2165.out‚0' disk.nvme0c0n1.out‚148862.123046875 disk.rbd1.in‚ 0.52392578125* proc.memory.percent‚17.119598388671875$ disk.nvme0c0n1.in‚50368.94140625 disk.dm-1.out‚ 1325.78515625 disk.sda1.out‚0 disk.rbd3.in‚0 disk.sda2.in‚0 disk.sda2.out‚0& disk.rbd0.in‚1.759218604441423e+13$ memory_percent‚82.78982639312744 network.sent‚ 2.4574331e+07 disk.dm-1.in‚ 108.921875 disk.dm-0.in‚50368.94140625) proc.memory.availableMB‚ 2819.71484375 proc.cpu.threads‚17 network.recv‚ 5.245627e+06„Í;”’3‚(:‰ µÙéÒஈ÷ gpu.0.gpu‚99.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚46.0 gpu.0.powerWatts‚333.089' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚66.6178 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÉAF“3‚(:» µÙéÒرöù& disk.rbd0.in‚1.759218604441423e+13 disk.dm-0.in‚50369.16015625 disk.sda3.out‚ 1326.83984375$ disk.nvme0c0n1.in‚50369.16015625 proc.memory.rssMB‚2804.875 proc.cpu.threads‚17* proc.memory.percent‚17.119598388671875 disk.sda.in‚ 108.921875 disk.rbd3.in‚0' disk.nvme0c0n1.out‚148875.216796875 disk.sda2.out‚0 disk.rbd1.out‚ 55.1591796875 disk.rbd1.in‚ 0.52392578125 disk.sda.out‚ 1326.83984375 disk.dm-1.in‚ 108.921875 disk.drbd2165.out‚0" disk.nvme0n1.in‚50369.16015625% disk.nvme0n1.out‚148875.216796875 disk.sda2.in‚0 network.recv‚ 5.249487e+06 disk.sda1.out‚0 disk.drbd2165.in‚0$ memory_percent‚82.78820514678955) proc.memory.availableMB‚ 2819.98046875) disk./.usagePercent‚37.94389632776239 disk.sda1.in‚0 disk.rbd3.out‚ 0.4951171875 disk.rbd0.out‚0" disk.dm-0.out‚148875.216796875 disk.sda3.in‚ 108.921875 cpu‚25.588640962466297$ disk./.usageGB‚565.3914756774902 disk.dm-1.out‚ 1326.83984375 network.sent‚ 2.462763e+07üv >ž”3‚(:“ ÄÙéÒ™¿øø gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚262.648' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚52.52960000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÉCz«5•3‚(:ª ÄÙéÒ²›³ú network.recv‚ 5.252245e+06 disk.dm-1.out‚ 1327.2890625& disk.rbd0.in‚1.759218604441423e+13 disk.sda.out‚ 1327.2890625 disk.dm-1.in‚ 108.921875 disk.dm-0.in‚ 50369.375 disk.rbd3.out‚ 0.4951171875* proc.memory.percent‚17.119598388671875 proc.cpu.threads‚17) disk./.usagePercent‚37.94436763763292# disk.dm-0.out‚148883.0380859375 disk.rbd1.out‚ 55.2216796875 proc.memory.rssMB‚2804.875 network.sent‚ 2.4631793e+07 disk.rbd0.out‚0 disk.sda1.out‚0 disk.sda3.in‚ 108.921875$ disk./.usageGB‚565.3984985351562 disk.sda3.out‚ 1327.2890625( disk.nvme0c0n1.out‚148883.0380859375& disk.nvme0n1.out‚148883.0380859375 disk.rbd1.in‚ 0.52392578125" memory_percent‚82.790207862854 disk.rbd3.in‚0 disk.nvme0n1.in‚ 50369.375 disk.sda2.out‚0 cpu‚25.58848263457187 disk.sda.in‚ 108.921875 disk.sda1.in‚0 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.nvme0c0n1.in‚ 50369.375 disk.sda2.in‚0) proc.memory.availableMB‚ 2819.65234375Ímaÿ”–3‚(:‰ ÓÙéÒàâ‚ø gpu.0.gpu‚93.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚46.0 gpu.0.powerWatts‚333.229' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚66.6458 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ñg1æH—3‚(:½ ÓÙéÒîî³ù disk.sda.in‚ 108.921875 disk.drbd2165.out‚0$ disk./.usageGB‚565.4050445556641& disk.rbd0.in‚1.759218604441423e+13) disk.nvme0c0n1.out‚148899.60107421875& proc.memory.availableMB‚ 2819.40625* proc.memory.percent‚17.119598388671875 cpu‚25.588351783431655 proc.cpu.threads‚17 disk.dm-0.in‚50369.78515625 disk.sda3.in‚ 108.921875" disk.nvme0n1.in‚50369.78515625 disk.sda.out‚ 1327.9609375 disk.rbd0.out‚0$ memory_percent‚82.79170989990234 disk.dm-1.in‚ 108.921875 disk.sda3.out‚ 1327.9609375 disk.sda2.in‚0* disk./.usagePercent‚37.944806946561684 disk.dm-1.out‚ 1327.9609375$ disk.dm-0.out‚148899.60107421875 disk.drbd2165.in‚0 disk.rbd3.out‚ 0.4951171875 proc.memory.rssMB‚2804.875 network.sent‚ 2.4636341e+07 disk.rbd1.in‚ 0.52392578125 disk.sda1.in‚0 disk.sda1.out‚0$ disk.nvme0c0n1.in‚50369.78515625' disk.nvme0n1.out‚148899.60107421875 disk.sda2.out‚0 network.recv‚ 5.255585e+06 disk.rbd1.out‚ 55.2216796875 disk.rbd3.in‚0q™ëÕ”˜3‚(:‰ âÙéÒÎÃëö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚266.646' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.3292 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚06b‹M™3‚(: âÙéÒºÙ•ù$ disk./.usageGB‚565.4040870666504 disk.rbd3.in‚0' disk.nvme0n1.out‚148913.23388671875 disk.rbd1.in‚ 0.52392578125 proc.memory.rssMB‚2804.875 disk.rbd3.out‚ 0.4951171875 disk.rbd0.out‚0 disk.dm-0.in‚50531.40234375$ disk.nvme0c0n1.in‚50531.40234375 disk.dm-1.out‚ 1330.73828125& disk.rbd0.in‚1.759218604441423e+13 disk.drbd2165.out‚0* disk./.usagePercent‚37.944742688670594 disk.sda1.out‚0 disk.drbd2165.in‚0) disk.nvme0c0n1.out‚148913.23388671875 disk.sda2.in‚0 disk.rbd1.out‚ 55.2841796875 disk.sda.in‚ 108.921875 disk.dm-1.in‚ 108.921875* proc.memory.percent‚17.119598388671875 cpu‚25.588207439101183 proc.cpu.threads‚17 network.sent‚ 2.4641461e+07 disk.sda1.in‚0 disk.sda3.out‚ 1330.73828125) proc.memory.availableMB‚ 2819.42578125 disk.sda3.in‚ 108.921875 disk.sda2.out‚0 disk.sda.out‚ 1330.73828125 network.recv‚ 5.259212e+06$ disk.dm-0.out‚148913.23388671875" disk.nvme0n1.in‚50531.40234375# memory_percent‚82.7915906906128àë34š3/ _wandbruntime‚21198  _runtime‚21198ûýªÊ›3Ä ' lr-AdamW/pg1‚0.00020595056415150803 ' lr-AdamW/pg2‚0.00020595056415150803 trainer/global_step‚12799 # _timestamp‚1.78431101429223e+09 _runtime‚21198.509429917 _step‚1051q£¸ÑÏœ3É ' lr-AdamW/pg2‚0.00020595056415150803 trainer/global_step‚12799 # _timestamp‚1.78431101429223e+09 _runtime‚21198.509429917 _step‚1051 ' lr-AdamW/pg1‚0.00020595056415150803›·0À(43/ _wandbruntime‚21198  _runtime‚21198Çö3±(ž3’P G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚993509617405952 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3677164245983232 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚3689201540336640 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚3675576451395584 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚69091699 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12037294353408 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12037294353408 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _step‚1052 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3689201540336640 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1009033999686656 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚32831062 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 219960624 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚254 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3689202911899648 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 110827267 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚249 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15524750371840 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3677164272156672 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 109133351 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _runtime‚21198.510706759 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12038639742976 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚3675576425967616 % _timestamp‚1.7843110142933958e+09 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚3677164245983232 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926:I»¹yŸ3—P XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12037294353408 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12037294353408 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _step‚1052 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3689201540336640 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1009033999686656 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDev%i·¾¤iceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚32831062 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 219960624 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚254 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3689202911899648 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 110827267 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚249 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15524750371840 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3677164272156672 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 109133351 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _runtime‚21198.510706759 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12038639742976 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚3675576425967616 % _timestamp‚1.7843110142933958e+09 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚3677164245983232 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚993509617405952 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3677164245983232 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚3689201540336640 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚3675576451395584 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚69091699 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648œ23dM4 3/ _wandbruntime‚21199  _runtime‚21199ØÚ®ö"'¡3œN H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 219977322 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 % _timestamp‚1.7843110156783497e+09 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚69096725 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3675854333173760 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12039564785664 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚69096975 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1009113078129664 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 _step‚1053 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 219967329 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15525974316032 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12018966876427 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1009112675509248 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 219977322 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 110835759 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3687873300050187 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 trainer/global_step‚12799 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚993586735722496 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15525939786752 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚993587103813632 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3677442272886784 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚3677442272886784 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚245 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚32833600 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 109141563 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 110825772 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 110825772 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12038219388928 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚32833605 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0%m>|''¢3¡N D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 219967329 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15525974316032 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12018966876427 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1009112675509248 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 219977322 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 110835759 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3687873300050187 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 trainer/global_step‚12799 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚993586735722496 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15525939786752 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚993587103813632 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3677442272886784 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚3677442272886784 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚245 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚32833600 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 109141563 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 110825772 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 110825772 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12038219388928 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚32833605 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 219977322 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 % _timestamp‚1.7843110156783497e+09 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚69096725 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3675854333173760 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12039564785664 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚69096975 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1009113078129664 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 _step‚1053Æ!VÈ4£3/ _wandbruntime‚21199  _runtime‚21199;BQ;¤3Œ % _timestamp‚1.7843110157645915e+09 0loss/FBt1/detection_loss‚0.31784478202462196 0loss/FBt2/detection_loss‚0.26170404531062974 0loss/FRt1/assignment_loss‚1.0805254430138975 0loss/FRt2/assignment_loss‚0.2199826283114297 3loss/FBt1/assignment_loss‚4.162312886067428e-05 3loss/FBt2/assignment_loss‚0.0001005675409911982 1loss/SRqqt1/detection_loss‚0.6379928947667635 trainer/global_step‚12799 2loss/SRqqt2/assignment_loss‚0.2530405478213182 /loss/FRt1/detection_loss‚0.4423638091412652 %loss/total_loss‚5.224403151649841 _runtime‚21199.981522206 _step‚1054 epoch‚31 3loss/SRqqt1/assignment_loss‚0.14882172805386693 /loss/FRt2/detection_loss‚1.3452854918669563 1loss/SRqqt2/detection_loss‚0.5166995906692399ˆŠ9¥3‘ 1loss/SRqqt1/detection_loss‚0.6379928947667635 trainer/global_step‚12799 2loss/SRqqt2/assignment_loss‚0.2530405478213182 /loss/FRt1/detection_loss‚0.4423638091412652 %loss/total_loss‚5.224403151649841 _runtime‚21199.981522206 _step‚1054 epoch‚31 3loss/SRqqt1/assignment_loss‚0.14882172805386693 /loss/FRt2/detection_loss‚1.3452854918669563 1loss/SRqqt2/detection_loss‚0.5166995906692399 % _timestamp‚1.7843110157645915e+09 0loss/FBt1/detection_loss‚0.31784478202462196 0loss/FBt2/detection_loss‚0.26170404531062974 0loss/FRt1/assignment_loss‚1.0805254430138975 0loss/FRt2/assignment_loss‚0.2199826283114297 3loss/FBt1/assignment_loss‚4.162312886067428e-05 3loss/FBt2/assignment_loss‚0.0001005675409911982žv’h. ¦3‚(:• ñÙéÒ½ä÷ gpu.0.gpu‚100.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚115.956' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.191200000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0cÒAzF§3‚(:» ñÙéÒé½úü disk.rbd0.out‚0 disk.rbd1.in‚ 0.52392578125 disk.dm-1.in‚ 108.921875" proc.memory.rssMB‚ 2856.2421875 cpu‚25.58926588569136 disk.dm-0.in‚ 50531.6171875 disk.drbd2165.in‚0 disk.rbd3.in‚0 disk.sda.out‚ 1332.1953125 network.recv‚ 5.262682e+06$ disk./.usageGB‚565.3956680297852( disk.nvme0c0n1.out‚148925.7470703125 disk.sda2.out‚0$ memory_percent‚83.10821056365967 disk.rbd3.out‚ 0.4951171875# disk.dm-0.out‚148925.7470703125 disk.sda1.in‚0 disk.sda1.out‚0& disk.nvme0n1.out‚148925.7470703125 disk.sda2.in‚0 disk.sda.in‚ 108.921875 disk.rbd1.out‚ 55.2841796875 disk.dm-1.out‚ 1332.1953125) proc.memory.availableMB‚ 2767.55078125 disk.sda3.in‚ 108.921875 disk.sda3.out‚ 1332.1953125 disk.drbd2165.out‚0! disk.nvme0n1.in‚ 50531.6171875 network.sent‚ 2.4647074e+07# disk.nvme0c0n1.in‚ 50531.6171875) proc.memory.percent‚17.43311882019043 proc.cpu.threads‚17) disk./.usagePercent‚37.94417768004252& disk.rbd0.in‚1.759218604441423e+13 ¡å꟨3‚(:” €ÚéÒÔô¾ö gpu.0.gpu‚100.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚46.0 gpu.0.powerWatts‚339.413' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚67.88260000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ë°éùP©3‚(:Å €ÚéÒœéÏù disk.dm-1.in‚ 108.921875$ disk.dm-0.out‚148939.49267578125( proc.memory.availableMB‚ 2767.8515625* disk./.usagePercent‚37.944644637784975 disk.drbd2165.out‚0 disk.sda.out‚ 1332.92578125 disk.dm-1.out‚ 1332.92578125 proc.cpu.threads‚17" disk.nvme0n1.in‚50532.02734375 disk.rbd0.out‚0 cpu‚25.590882009929704$ disk.nvme0c0n1.in‚50532.02734375 disk.rbd1.in‚ 0.52392578125& disk.rbd0.in‚1.759218604441423e+13$ memory_percent‚83.10637474060059 disk.sda3.out‚ 1332.92578125 disk.rbd3.out‚ 0.4951171875 disk.sda2.in‚0 disk.rbd1.out‚ 55.3466796875 disk.drbd2165.in‚0' disk.nvme0n1.out‚148939.49267578125 disk.sda1.in‚0 disk.sda1.out‚0 network.sent‚ 2.4709871e+07 network.recv‚ 5.267153e+06 disk.sda3.in‚ 108.921875 disk.rbd3.in‚0 disk.sda2.out‚0 disk.sda.in‚ 108.921875" proc.memory.rssMB‚ 2856.2421875) proc.memory.percent‚17.43311882019043$ disk./.usageGB‚565.4026260375977 disk.dm-0.in‚50532.02734375) disk.nvme0c0n1.out‚148939.49267578125Øì.4ª3/ _wandbruntime‚21226  _runtime‚21226y[ú† «3ˆ| =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.12155276536941528 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.15294209122657776 particle/accuracy_2_of_3‚1 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6308631896972656 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 - Purity/0FRt2SRqqt1FBt/FBt_purity‚0.921875 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.1419702023267746 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6070085763931274 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.7821767926216125 +jet/accuracy_2_of_2‚0.11351753771305084 particle/accuracy_1_of_1‚1 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.14197362959384918 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.04776725918054581 particle/accuracy_1_of_3‚1 /particle/accuracy_4_of_4‚0.9362371563911438 jet/accuracy_2_of_5‚1 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.1669590324163437 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9897521734237671 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9960039854049683 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 'particle/accuracy‚0.830946147441864 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.15508432686328888 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.6821038126945496 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.5417256355285645 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6342129707336426 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.04776725918054581 ,jet/accuracy_3_of_4‚0.053976599127054214 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.15379130840301514 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 jet/accuracy_5_of_5‚0 particle/accuracy_1_of_5‚1 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6079894304275513 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6109256148338318 9"Purity/2FRt*9ÿ€ôð4SRqqt1FBt/SRqqt_purity‚0.7083333134651184 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.7821767926216125 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5699836015701294 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9978214502334595 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.7428761124610901 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.37387651205062866 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6472724080085754 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 particle/accuracy_3_of_3‚1 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6394903659820557 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.5826151371002197 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 7validation_average_jet_accuracy‚0.31650224328041077 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9930043816566467 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 - Purity/0FRt2SRqqt*FBt/FBt_purity‚0.921875 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5197677612304688 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.21296295523643494 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.02385476976633072 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6460807919502258 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.21668021380901337 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.6776612401008606 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.5683504939079285 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9955589771270752 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9845107197761536 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6290109753608704 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 *jet/accuracy_2_of_4‚0.6772531270980835 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.15396741032600403 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.05000000074505806 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7053413391113281 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.26923078298568726 particle/accuracy_2_of_5‚1 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9918205738067627 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.32455185055732727 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.3302382528781891 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4609375 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7283949851989746 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7078421711921692 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.32038235664367676 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3499999940395355 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.19341298937797546 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9090664386749268 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.6800076365470886 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7128168940544128 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.4866666793823242 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.8935184478759766 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6326842308044434 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9943017363548279 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8033699989318848 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.08833333104848862 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.17696639895439148 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7069863080978394 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4120370149612427 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.05967573821544647 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9984056353569031 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.16020777821540833 particle/accuracy_2_of_2‚1 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.6237148642539978 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5547305941581726 'particle/f_score‚0.5995420217514038 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6279886960983276 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7550643086433411 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.14386551082134247 *jet/accuracy_1_of_2‚0.7103968858718872 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9965519905090332 epoch‚31 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.05967727676033974 *jet/accuracy_1_of_1‚0.2744761109352112 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.8571428656578064 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 particle/accuracy_1_of_2‚1 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9537036418914795 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5547305941581726 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.1664639413356781 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7625094056129456 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9661681056022644 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.005527078174054623 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.8653846383094788 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9988662004470825 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.625 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5885134339332581 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.25098779797554016 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.44984784722328186 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9978214502334595 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.7587679624557495 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.21296295523643494 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.44212964177131653 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.24391376972198486 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 _runtime‚21226.591935027 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9830840826034546 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.38545945286750793 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.2005365937948227 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.45199304819107056 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.011806732043623924 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.15396741032600403 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 jet/accuracy_4_of_5‚0 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 6 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.594357430934906 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7428761124610901 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7142857313156128 % _timestamp‚1.7843110423740282e+09 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.14775702357292175 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.11771820485591888 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 jet/accuracy_3_of_5‚1 trainer/global_step‚12799 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.023260122165083885 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.011053602211177349 *jet/accuracy_1_of_3‚0.8886526226997375 +particle/sensitivity‚0.5893288850784302 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.12616457045078278 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.5853005647659302 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9916365742683411 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.26923078298568726 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 jet/accuracy_1_of_5‚1 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.14176678657531738 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9537036418914795 particle/accuracy_3_of_5‚1 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9735708832740784 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 ,jet/accuracy_4_of_4‚0.013899499550461769 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9970017671585083 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.13313958048820496 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 )loss/val_total_loss‚9.441954636824548 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.25606754422187805 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.1590307503938675 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.5494791865348816 _step‚1055 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.5889885425567627 *jet/accuracy_1_of_4‚0.9616719484329224 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8457808494567871 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.056189123541116714 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 8"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.558027982711792 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5312556028366089 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 particle/accuracy_5_of_5‚0 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8069409728050232 6 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.996728777885437 7 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.0563272163271904 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.11698544770479202 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6619164347648621 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5372449159622192 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9971989393234253 particle/accuracy_2_of_4‚1 +particle/specificity‚0.8970231413841248 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7001782059669495 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6254368424415588 particle/accuracy_1_of_4‚1 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9878317713737488 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7083333134651184 particle/accuracy_4_of_5‚1 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.16799390316009521 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.028946561738848686 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.17323794960975647 +jet/accuracy_3_of_3‚0.04800165444612503 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.15448449552059174 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9937976002693176 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 -"Purity/2FRt0SRqqt1FBt/event_purity‚0.1875 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.22288791835308075 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9049807786941528 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9856430888175964 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_3_of_4‚1 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9773991107940674 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.14179480075836182 *jet/accuracy_2_of_3‚0.4621640145778656 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.8125 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.05000000074505806 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 8"Purity/*FRt2SRqqt0FBt/event_purity‚0.171228289604187 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6234015226364136 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094üÜÐe>¬3| 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.6237148642539978 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5547305941581726 'particle/f_score‚0.5995420217514038 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6279886960983276 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7550643086433411 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.14386551082134247 *jet/accuracy_1_of_2‚0.7103968858718872 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9965519905090332 epoch‚31 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.05967727676033974 *jet/accuracy_1_of_1‚0.2744761109352112 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.8571428656578064 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 particle/accuracy_1_of_2‚1 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9537036418914795 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5547305941581726 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.1664639413356781 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7625094056129456 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9661681056022644 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.005527078174054623 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.8653846383094788 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9988662004470825 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.625 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5885134339332581 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.25098779797554016 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.44984784722328186 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9978214502334595 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.7587679624557495 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.21296295523643494 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.44212964177131653 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.24391376972198486 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 _runtime‚21226.591935027 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9830840826034546 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.38545945286750793 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.2005365937948227 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.45199304819107056 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.011806732043623924 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.15396741032600403 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 jet/accuracy_4_of_5‚0 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 6 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.594357430934906 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7428761124610901 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7142857313156128 % _timestamp‚1.7843110423740282e+09 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.14775702357292175 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.11771820485591888 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 jet/accuracy_3_of_5‚1 trainer/global_step‚12799 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.023260122165083885 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.011053602211177349 *jet/accuracy_1_of_3‚0.8886526226997375 +particle/sensitivity‚0.5893288850784302 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.12616457045078278 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.5853005647659302 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9916365742683411 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.26923078298568726 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 jet/accuracy_1_of_5‚1 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.14176678657531738 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9537036418914795 particle/accuracy_3_of_5‚1 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9735708832740784 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 ,jet/accuracy_4_of_4‚0.013899499550461769 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9970017671585083 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.13313958048820496 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 )loss/val_total_loss‚9.441954636824548 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.25606754422187805 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.1590307503938675 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.5494791865348816 _step‚1055 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.5889885425567627 *jet/accuracy_1_of_4‚0.9616719484329224 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8457808494567871 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.056189123541116714 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 8"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.558027982711792 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5312556028366089 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 particle/accuracy_5_of_5‚0 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8069409728050232 6 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.996728777885437 7 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.0563272163271904 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.11698544770479202 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6619164347648621 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5372449159622192 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9971989393234253 particle/accuracy_2_of_4‚1 +particle/specificity‚0.8970231413841248 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7001782059669495 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6254368424415588 particle/accuracy_1_of_4‚1 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9878317713737488 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7083333134651184 particle/accuracy_4_of_5‚1 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.16799390316009521 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.028946561738848686 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.17323794960975647 +jet/accuracy_3_of_3‚0.04800165444612503 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.15448449552059174 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9937976002693176 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 -"Purity/2FRt0SRqqt1FBt/event_purity‚0.1875 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.22288791835308075 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9049807786941528 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9856430888175964 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_3_of_4‚1 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9773991107940674 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.14179480075836182 *jet/accuracy_2_of_3‚0.4621640145778656 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.8125 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.05000000074505806 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 8"Purity/*FRt2SRqqt0FBt/event_purity‚0.171228289604187 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6234015226364136 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.12155276536941528 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.15294209122657776 particle/accuracy_2_of_3‚1 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6308631896972656 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 - Purity/0FRt2SRqqt1FBt/FBt_purity‚0.921875 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.1419702023267746 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6070085763931274 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.7821767926216125 +jet/accuracy_2_of_2‚0.11351753771305084 particle/accuracy_1_of_1‚1 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.14197362959384918 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.04776725918054581 particle/accuracy_1_of_3‚1 /particle/accuracy_4_of_4‚0.9362371563911438 jet/accuracy_2_of_5‚1 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.1669590324163437 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9897521734237671 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9960039854049683 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 'particle/accuracy‚0.830946147441864 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.15508432686328888 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.6821038126945496 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.5417256355285645 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6342129707336426 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.04776725918054581 ,jet/accuracy_3_of_4‚0.053976599127054214 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.15379130840301514 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 jet/accuracy_5_of_5‚0 particle/accuracy_1_of_5‚1 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6079894304275513 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6109256148338318 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7083333134651184 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.7821767926216125 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5699836015701294 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9978214502334595 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.7428761124610901 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.37387651205062866 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6472724080085754 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 particle/accuracy_3_of_3‚1 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6394903659820557 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.5826151371002197 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 7validation_average_jet_accuracy‚0.31650224328041077 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9930043816566467 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 - Purity/0FRt2SRqqt*FBt/FBt_purity‚0.921875 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5197677612304688 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.21296295523643494 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.02385476976633072 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6460807919502258 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.21668021380901337 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.6776612401008606 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.5683504939079285 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9955589771270752 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9845107197761536 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6290109753608704 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 *jet/accuracy_2_of_4‚0.6772531270980835 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.15396741032600403 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.05000000074505806 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7053413391113281 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.26923078298568726 particle/accuracy_2_of_5‚1 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9918205738067627 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.32455185055732727 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.3302382528781891 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4609375 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7283949851989746 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7078421711921692 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.32038235664367676 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3499999940395355 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.19341298937797546 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9090664386749268 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.6800076365470886 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7128168940544128 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.4866666793823242 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.8935184478759766 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6326842308044434 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9943017363548279 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8033699989318848 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.08833333104848862 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.17696639895439148 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7069863080978394 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4120370149612427 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.05967573821544647 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9984056353569031 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.16020777821540833 particle/accuracy_2_of_2‚1ŸàówM­3‚(: ÚéÒæŒÎø disk.dm-0.in‚50532.24609375 disk.rbd3.out‚ 0.4951171875 disk.rbd1.in‚ 0.52392578125 disk.rbd0.out‚0' disk.nvme0n1.out‚148951.04833984375 proc.memory.rssMB‚2804.875 disk.rbd1.out‚ 55.3466796875 disk.sda1.out‚0 disk.sda3.in‚ 108.921875) proc.memory.availableMB‚ 2819.39453125 network.recv‚ 5.270642e+06$ disk./.usageGB‚565.4222183227539 disk.sda3.out‚ 1333.45703125 disk.drbd2165.out‚0 network.sent‚ 2.4715614e+07 disk.sda.out‚ 1333.45703125 disk.dm-1.out‚ 1333.45703125& disk.rbd0.in‚1.759218604441423e+13$ disk.dm-0.out‚148951.04833984375 disk.drbd2165.in‚0 disk.rbd3.in‚0) disk.nvme0c0n1.out‚148951.04833984375 disk.sda1.in‚0" disk.nvme0n1.in‚50532.24609375 disk.sda2.in‚0$ memory_percent‚82.79178142547607* proc.memory.percent‚17.119598388671875 cpu‚25.591067368437997 proc.cpu.threads‚17) disk./.usagePercent‚37.94595949248084 disk.sda.in‚ 108.921875$ disk.nvme0c0n1.in‚50532.24609375 disk.sda2.out‚0 disk.dm-1.in‚ 108.921875•hñr’®3‚(:‡ ÚéÒã¤õ÷ gpu.0.gpu‚98.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚276.31' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.262 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ª²'Á”¯3‚(:‰ žÚéÒ‡¥øö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚40.0 gpu.0.powerWatts‚112.013' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.4026 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0´*¨ãA°3‚(:¶ žÚéÒºžÖù) disk./.usagePercent‚37.94642440216302 disk.rbd0.out‚0* proc.memory.percent‚17.119598388671875$ disk./.usageGB‚565.4291458129883 disk.rbd3.in‚0$ memory_percent‚82.79480934143066 network.recv‚ 5.275263e+06 disk.sda2.in‚0 network.sent‚ 2.477299e+07 disk.sda3.in‚ 108.921875 disk.drbd2165.in‚0" disk.nvme0c0n1.in‚ 50752.046875 proc.memory.rssMB‚2804.875 cpu‚25.59091821445545 disk.drbd2165.out‚0 disk.nvme0n1.in‚ 50752.046875 disk.dm-1.in‚ 108.921875 proc.cpu.threads‚17) disk.nvme0c0n1.out‚148959.42626953125' disk.nvme0n1.out‚148959.42626953125 disk.sda2.out‚0 disk.rbd1.out‚ 55.4716796875 disk.dm-1.out‚ 1334.2109375 disk.dm-0.in‚ 50752.046875$ disk.dm-0.out‚148959.42626953125 disk.sda1.in‚0 disk.rbd1.in‚ 0.52392578125 disk.sda.in‚ 108.921875( proc.memory.availableMB‚ 2818.8984375& disk.rbd0.in‚1.759218604441423e+13 disk.sda1.out‚0 disk.sda3.out‚ 1334.2109375 disk.rbd3.out‚ 0.4951171875 disk.sda.out‚ 1334.2109375F:º¡±3‚(:“ ­ÚéÒˆÞ€÷ gpu.0.gpu‚94.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚281.143' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚56.2285V†vx†9999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ¾]D²3‚(:¹ ­ÚéÒëÖù disk.sda1.out‚0 disk.sda2.out‚0$ disk.nvme0c0n1.in‚50752.45703125" disk.nvme0n1.in‚50752.45703125& disk.nvme0n1.out‚148974.6435546875 disk.sda1.in‚0 disk.drbd2165.in‚0 disk.sda.in‚ 108.921875# disk.dm-0.out‚148974.6435546875$ memory_percent‚82.79480934143066 disk.rbd3.in‚0 disk.rbd3.out‚ 0.4951171875 disk.dm-1.in‚ 108.921875 cpu‚25.590783709684374 disk.sda3.out‚ 1334.8828125 disk.drbd2165.out‚0 disk.rbd1.in‚ 0.52392578125 disk.rbd0.out‚0( proc.memory.availableMB‚ 2818.8984375 proc.cpu.threads‚17 network.sent‚ 2.4778715e+07 disk.sda3.in‚ 108.921875 disk.dm-1.out‚ 1334.8828125& disk.rbd0.in‚1.759218604441423e+13 disk.dm-0.in‚50752.45703125 proc.memory.rssMB‚2804.875* proc.memory.percent‚17.119598388671875) disk./.usagePercent‚37.94685884694862 network.recv‚ 5.27848e+06# disk./.usageGB‚565.435619354248( disk.nvme0c0n1.out‚148974.6435546875 disk.sda2.in‚0 disk.rbd1.out‚ 55.4716796875 disk.sda.out‚ 1334.8828125d¬g攳3‚(:‰ ¼ÚéҜ̃÷ gpu.0.gpu‚93.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚337.816' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚67.5632 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¡àÚ7E´3‚(:º ¼ÚéÒÙÀÛù disk.rbd0.out‚0# memory_percent‚82.7951192855835$ disk.nvme0c0n1.in‚50752.67578125 disk.rbd1.in‚ 0.52392578125 disk.rbd1.out‚ 55.5341796875 disk.sda3.in‚ 108.921875 disk.sda3.out‚ 1335.4609375 disk.drbd2165.out‚0 disk.sda.out‚ 1335.4609375& disk.rbd0.in‚1.759218604441423e+13 disk.dm-0.in‚50752.67578125" disk.nvme0n1.in‚50752.67578125 disk.dm-1.in‚ 108.921875 disk.dm-1.out‚ 1335.4609375) proc.memory.availableMB‚ 2818.84765625 proc.cpu.threads‚17 disk.rbd3.in‚0 disk.rbd3.out‚ 0.4951171875( disk.nvme0c0n1.out‚148982.3251953125 disk.sda2.out‚0 disk.sda1.in‚0) disk./.usagePercent‚37.94723953015203# disk./.usageGB‚565.441291809082 network.sent‚ 2.4783801e+07 network.recv‚ 5.282016e+06& disk.nvme0n1.out‚148982.3251953125# disk.dm-0.out‚148982.3251953125 disk.drbd2165.in‚0* proc.memory.percent‚17.119598388671875 disk.sda2.in‚0 disk.sda.in‚ 108.921875 disk.sda1.out‚0 proc.memory.rssMB‚2804.875 cpu‚25.590649326139015¦#6”µ3‚(:‰ ËÚéÒÂóö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚265.534' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.1068 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0³­Õ C¶3‚(:¸ ËÚéҥĊù disk.rbd3.out‚ 0.4951171875 cpu‚25.59049334512526* disk./.usagePercent‚37.947716984203396 disk.dm-1.in‚ 108.921875" disk.nvme0c0n1.in‚ 50752.890625 disk.nvme0n1.in‚ 50752.890625' disk.nvme0n1.out‚148992.26416015625 disk.sda2.out‚0$ disk./.usageGB‚565.4484062194824 disk.rbd0.out‚0 disk.drbd2165.out‚0 disk.sda2.in‚0 disk.sda.in‚ 108.921875 disk.sda1.out‚0 network.recv‚ 5.285451e+06 disk.dm-1.out‚ 1336.02734375 disk.rbd3.in‚0 network.sent‚ 2.4789125e+07 disk.sda1.in‚0 proc.memory.rssMB‚2804.875 proc.cpu.threads‚17 disk.sda.out‚ 1336.02734375 disk.dm-0.in‚ 50752.890625) disk.nvme0c0n1.out‚148992.26416015625# memory_percent‚82.7951431274414& disk.rbd0.in‚1.759218604441423e+13$ disk.dm-0.out‚148992.26416015625 disk.sda3.in‚ 108.921875 disk.sda3.out‚ 1336.02734375 disk.rbd1.in‚ 0.52392578125 disk.rbd1.out‚ 55.5341796875& proc.memory.availableMB‚ 2818.84375* proc.memory.percent‚17.119598388671875 disk.drbd2165.in‚0èýãó4·3/ _wandbruntime‚21303  _runtime‚21303¾•Þɸ3à trainer/global_step‚12849 % _timestamp‚1.7843111193893387e+09 _runtime‚21303.60650473 _step‚1056 & lr-AdamW/pg1‚0.0002034198538518043 & lr-AdamW/pg2‚0.0002034198538518043ßyÖι3È & lr-AdamW/pg1‚0.0002034198538518043 & lr-AdamW/pg2‚0.0002034198538518043 trainer/global_step‚12849 % _timestamp‚1.7843111193893387e+09 _runtime‚21303.60650473 _step‚1056 hO†@4º3/ _wandbruntime‚21303  _runtime‚21303«p>\(»3’P O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚69395667 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12070109654614 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3706981716819456 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚3706981716819456 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 111312001 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 111302028 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚32967125 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 _step‚1057 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 111302028 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚268 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚3694892263923712 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 109675663 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3706983088382464 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚997431681945600 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚12849 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚69395935 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 111312001 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3694892263923712 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12089452895744 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12068765054292 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15591867476480 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12090798285312 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 109675657 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚263 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995M*Ór(¼3—P H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3706983088382464 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚997431681945600 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚12849 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚69395935 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 111312001 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3694892263923712 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12089452895744 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12068765054292 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15591867476480 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12090798285312 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 109675657 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚263 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚69395667 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12070109654614 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3706981716819456 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚3706981716819456 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 111312001 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 111302028 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚32967125 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 _step‚1057 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 111302028 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚268 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚3694892263923712 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 109675663 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290¡©Œ.¬4½3/ _wandbruntime‚21304  _runtime‚21304ñ°+­¼¾3œN ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12071033120972 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 109683869 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚997509168353280 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 _runtime‚21304.992076798 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 221004362 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚69400951 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 109683863 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚32969668 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15593094311424 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚3705643994107738 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3695170290827264 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12069688520538 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12091723328000 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 111320493 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚32969673 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚36431530 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚247 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3693574305587200 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚3695170264653824 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1013102630755840 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚997509536444416 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 _step‚1058 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 111320493 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚36431283 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 221004362 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 111310506 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3693574331015168 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15593059782144 =7DeviceStW}Nf atsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 220994369 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 220994369 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 111310506 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1013102228135424 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7843111207745707e+09 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3707262014155264 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 trainer/global_step‚12849 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12091723328000 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 109683869 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3705645364136140 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820wåÁš''¿3¡N _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3695170290827264 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12069688520538 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12091723328000 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 111320493 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚32969673 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚36431530 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚247 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3693574305587200 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚3695170264653824 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1013102630755840 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚997509536444416 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 _step‚1058 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 111320493 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚36431283 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 221004362 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 111310506 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3693574331015168 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15593059782144 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 220994369 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 220994369 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 111310506 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1013102228135424 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7843111207745707e+09 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3707262014155264 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 trainer/global_step‚12849 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12091723328000 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 109683869 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3705645364136140 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12071033120972 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 109683869 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚997509168353280 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 _runtime‚21304.992076798 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 221004362 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚69400951 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 109683863 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚32969668 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15593094311424 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚3705643994107738 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0¢ÀˆZž4À3/ _wandbruntime‚21305  _runtime‚213052¨øÁ3 _step‚1059 0loss/FRt1/assignment_loss‚1.1142806795736153 0loss/FRt2/assignment_loss‚0.1917068985037673 /loss/FRt2/detection_loss‚1.2371126164257191 1loss/SRqqt1/detection_loss‚0.4950077491288539 0loss/FBt2/detection_loss‚0.19424625879718932 _runtime‚21305.078571751 %loss/total_loss‚4.782137073012286 3loss/FBt1/assignment_loss‚7.591715081607737e-07 3loss/FBt2/assignment_loss‚0.0019124340407074907 /loss/FBt1/detection_loss‚0.3079804962318648 3loss/SRqqt1/assignment_loss‚0.12809817175575475 3loss/SRqqt2/assignment_loss‚0.20260145311642858 /loss/FRt1/detection_loss‚0.4320245631187077 2loss/SRqqt2/detection_loss‚0.47716499314816924 epoch‚32 trainer/global_step‚12849 % _timestamp‚1.7843111208615968e+09›»ïNÂ3’ 3loss/SRqqt1/assignment_loss‚0.12809817175575475 3loss/SRqqt2/assignment_loss‚0.20260145311642858 /loss/FRt1/detection_loss‚0.4320245631187077 2loss/SRqqt2/detection_loss‚0.47716499314816924 epoch‚32 trainer/global_step‚12849 % _timestamp‚1.7843111208615968e+09 _step‚1059 0loss/FRt1/assignment_loss‚1.1142806795736153 0loss/FRt2/assignment_loss‚0.1917068985037673 /loss/FRt2/detection_loss‚1.2371126164257191 1loss/SRqqt1/detection_loss‚0.4950077491288539 0loss/FBt2/detection_loss‚0.19424625879718932 _runtime‚21305.078571751 %loss/total_loss‚4.782137073012286 3loss/FBt1/assignment_loss‚7.591715081607737e-07 3loss/FBt2/assignment_loss‚0.0019124340407074907 /loss/FBt1/detection_loss‚0.3079804962318648£©Ÿî­“Ã3‚(:ˆ ÚÚéÒÒ÷éö gpu.0.gpu‚93.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚343.465' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚68.693 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0n+Ð>Ä3‚(:³ ÚÚéÒÆÕöø disk.sda3.out‚ 1336.546875 disk.rbd1.in‚ 0.52392578125* proc.memory.percent‚17.119598388671875 proc.cpu.threads‚17 disk.rbd3.out‚ 0.4951171875 disk.sda1.in‚0( proc.memory.availableMB‚ 2818.8359375 disk.sda3.in‚ 108.921875 disk.drbd2165.out‚0 disk.sda2.in‚0& disk.rbd0.in‚1.759218604441423e+13$ memory_percent‚82.79519081115723 disk.rbd0.out‚0 disk.sda1.out‚0 network.sent‚ 2.4794206e+07' disk.nvme0c0n1.out‚149006.072265625 disk.sda.in‚ 108.921875 disk.dm-1.in‚ 108.921875 disk.dm-1.out‚ 1336.546875" disk.dm-0.out‚149006.072265625$ disk./.usageGB‚565.4550514221191 disk.drbd2165.in‚0 disk.rbd3.in‚0# disk.nvme0c0n1.in‚ 50926.5546875 disk.rbd1.out‚ 55.5966796875 proc.memory.rssMB‚2804.875! disk.nvme0n1.in‚ 50926.5546875 disk.sda.out‚ 1336.546875 network.recv‚ 5.288901e+06* disk./.usagePercent‚37.948162949328044% disk.nvme0n1.out‚149006.072265625 disk.sda2.out‚0 disk.dm-0.in‚ 50926.5546875 cpu‚25.590348105411195Ú§PОÅ3‚(:“ éÚéÒÝó­÷ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚283.318' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚56.66359999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚08‘ŸÐHÆ3‚(:½ éÚéÒéüÅú& disk.rbd0.in‚1.759218604441423e+13 disk.drbd2165.out‚0 disk.sda1.in‚0 disk.rbd1.out‚ 55.5966796875) disk.nvme0c0n1.out‚149017.99560546875! disk.nvme0n1.in‚ 50926.7734375 disk.sda2.in‚0 disk.sda2.out‚0# disk.nvme0c0n1.in‚ 50926.7734375 disk.sda.in‚ 108.921875' proc.memory.availableMB‚ 2818.796875 proc.cpu.threads‚17) disk./.usagePercent‚37.94859278597803$ memory_percent‚82.79542922973633 network.sent‚ 2.4857617e+07 network.recv‚ 5.293193e+06 proc.memory.rssMB‚2804.875 cpu‚25.590198658572575$ disk.dm-0.out‚149017.99560546875 disk.rbd3.in‚0 disk.rbd3.out‚ 0.4951171875 disk.rbd1.in‚ 0.52392578125 disk.dm-1.in‚ 108.921875' disk.nvme0n1.out‚149017.99560546875 disk.drbd2165.in‚0* proc.memory.percent‚17.119598388671875 disk.sda1.out‚0 disk.sda3.in‚ 108.921875 disk.sda3.out‚ 1337.20703125$ disk./.usageGB‚565.4614562988281 disk.sda.out‚ 1337.20703125 disk.dm-1.out‚ 1337.20703125 disk.rbd0.out‚0 disk.dm-0.in‚ 50926.7734375é»TžÇ3‚(:“ øÚéÒ»£”ø gpu.0.gpu‚64.0 gpu.0.memory‚21, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚336.465' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚67.29299999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ª hIÈ3‚(:¾ øÚéÒƒ­äù disk.drbd2165.out‚0 disk.rbd3.out‚ 0.4951171875! disk.nvme0n1.in‚ 50927.1953125 disk.sda2.in‚0 disk.dm-1.in‚ 108.921875 disk.dm-1.out‚ 1337.87109375' disk.nvme0n1.out‚149029.92431640625 disk.sda1.in‚0 disk.sda3.out‚ 1337.87109375 disk.rbd1.in‚ 0.52392578125$ disk.dm-0.out‚149029.92431640625 proc.memory.rssMB‚2804.875) disk./.usagePercent‚37.94925046933349 disk.sda3.in‚ 108.921875 disk.sda2.out‚0 disk.sda.in‚ 108.921875 disk.rbd0.out‚0 proc.cpu.threads‚17$ disk./.usageGB‚565.4712562561035# disk.nvme0c0n1.in‚ 50927.1953125) disk.nvme0c0n1.out‚149029.92431640625 disk.rbd1.out‚ 55.6591796875 disk.dm-0.in‚ 50927.1953125 network.sent‚ 2.4862206e+07 disk.drbd2165.in‚0 disk.rbd3.in‚0) proc.memory.availableMB‚ 2819.26171875* proc.memory.percent‚17.119598388671875 cpu‚25.59006702523111 disk.sda1.out‚0 disk.sda.out‚ 1337.87109375$ memory_percent‚82.79259204864502 network.recv‚ 5.296731e+06& disk.rbd0.in‚1.759218604441423e+13L•’”É3‚(:‰ ‡ÛéÒÃíÿö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚283.889' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚56.7778 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0KËBÊ3‚(:· ‡ÛéÒ ”ºù disk.sda1.in‚0 disk.rbd3.out‚ 0.4951171875! disk.nvme0n1.in‚ 50927.3984375& disk.nvme0n1.out‚149046.8017578125$ memory_percent‚82.79433250427246 disk.rbd1.in‚ 0.52392578125 disk.sda.in‚ 108.921875( disk.nvme0c0n1.out‚149046.8017578125 disk.sda2.out‚0( proc.memory.availableMB‚ 2818.9765625 disk.sda1.out‚0 disk.sda3.in‚ 108.921875 disk.sda3.out‚ 1338.5390625# disk.nvme0c0n1.in‚ 50927.3984375* proc.memory.percent‚17.119598388671875 network.sent‚ 2.4867302e+07 network.recv‚ 5.300258e+06) disk./.usagePercent‚37.94952465340267& disk.rbd0.in‚1.759218604441423e+13 disk.rbd0.out‚0 disk.rbd3.in‚0 disk.sda2.in‚0 cpu‚25.589911135786156# disk./.usageGB‚565.475341796875 disk.dm-0.in‚ 50927.3984375 disk.drbd2165.in‚0 disk.drbd2165.out‚0 proc.cpu.threads‚17 disk.dm-1.in‚ 108.921875# disk.dm-0.out‚149046.8017578125 proc.memory.rssMB‚2804.875 disk.rbd1.out‚ 55.6591796875 disk.sda.out‚ 1338.5390625 disk.dm-1.out‚ 1338.5390625 ¬Kf”Ë3‚(:‰ –ÛéÒªéö gpu.0.gpu‚99.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚346.941' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚69.3882 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¬Ž•ÄEÌ3‚(:º –ÛéÒûº ù disk.dm-1.in‚ 108.921875( proc.memory.availableMB‚ 2818.9765625 proc.cpu.threads‚17 disk.rbd1.out‚ 55.7216796875" disk.dm-0.out‚149054.919921875 network.sent‚ 2.4873113e+07 network.recv‚ 5.303783e+06 disk.sda3.in‚ 108.921875 disk.sda3.out‚ 1338.83984375 disk.drbd2165.out‚0 disk.rbd3.out‚ 0.4951171875$ memory_percent‚82.79433250427246 proc.memory.rssMB‚2804.875( disk./.usagePercent‚37.9499765067006 disk.rbd3.in‚0& disk.rbd0.in‚1.759218604441423e+13 disk.sda1.in‚0 disk.drbd2165.in‚0 disk.sda2.in‚0 disk.sda.in‚ 108.921875 disk.rbd0.out‚0$ disk.nvme0c0n1.in‚50974.48828125 disk.rbd1.in‚ 0.52392578125* proc.memory.percent‚17.119598388671875 disk.sda2.out‚0 disk.sda.out‚ 1338.83984375 disk.dm-1.out‚ 1338.83984375 disk.dm-0.in‚50974.48828125 cpu‚25.589778519020182 disk.sda1.out‚0$ disk./.usageGB‚565.4820747375488' disk.nvme0c0n1.out‚149054.919921875" disk.nvme0n1.in‚50974.48828125% disk.nvme0n1.out‚149054.919921875³…&«4Í3/ _wandbruntime‚21382  _runtime‚21382Ð-ÕÉËÎ3Å ' lr-AdamW/pg1‚0.00020089829634162875 ' lr-AdamW/pg2‚0.00020089829634162875 trainer/global_step‚12899 $ _timestamp‚1.784311198203019e+09 _runtime‚21382.420146108 _step‚1060²bèÐÏ3Ê ' lr-AdamW/pg1‚0.00020089829634162875 ' lr-AdamW/pg2‚0.00020089829634162875 trainer/global_step‚12899 $ _timestamp‚1.784311198203019e+09 _runtime‚21382.420146108 _step‚1060¤È6· 4Ð3/ _wandbruntime‚21382  _runtime‚21382O7Æ(Ñ3”P VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1016957939068416 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 _runtime‚21382.421323551 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 111737753 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 110085963 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚266 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3708792300429312 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚33094280 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12135768826368 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚36565053 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 % _timestamp‚1.7843111982040222e+09 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15651532014080 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 110085957 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚69659062 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚3708792300429312 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12135768826368 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3720928069255680 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚36564787 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3719306884913514 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12116346767722 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1001306370420736 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12137114215936 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3720929440818688 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3708792326602752 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12137114215936 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 110085957 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 111737753 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚33094275 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 111727780 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚3708792326602752 trainer/global_step‚12899 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0YfUzÒ3™P WBDeviceStatsMo@Z½(nitor.on_train_batch_start/active_bytes.all.allocated‚3720929440818688 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 111727780 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36633600 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚69659333 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1016957939068416 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 _runtime‚21382.421323551 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 111737753 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 110085963 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚266 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3708792300429312 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚33094280 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12135768826368 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚36565053 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 % _timestamp‚1.7843111982040222e+09 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15651532014080 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 110085957 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚69659062 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚3708792300429312 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12135768826368 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3720928069255680 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚36564787 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3719306884913514 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12116346767722 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1001306370420736 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12137114215936 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3720929440818688 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3708792326602752 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12137114215936 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 110085957 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 111737753 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚33094275 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 111727780 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚3708792326602752 trainer/global_step‚12899 JDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15652750041600 QDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3721206995021312 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 110094169 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚3709070301159424 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 110094169 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1001383856828416 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15652713415168 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12115925633646 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12136693861888 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3707468419923968 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12136693861888 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 110094163 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚69664380 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 % _timestamp‚1.7843111995883212e+09 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3709070301159424 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404717568 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 _runtime‚21383.805665046 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚271 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚3719584320129646 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 221840414 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1017036202152448 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 111736258 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚3709070327332864 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 _step‚1062 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 111746245 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1001383488737280 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 221840414 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1017036606870016 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3721208366591488 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 111736258 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3719585690158048 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12138039258624 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚266 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 221840414 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1017036606870016 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3721208366591488 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 111736258 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3719585690158048 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12138039258624 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚266 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15652750041600 QDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3721206995021312 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 110094169 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚3709070301159424 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 110094169 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1001383856828416 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15652713415168 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12115925633646 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12136693861888 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3707468419923968 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12136693861888 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 110094163 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚69664380 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 % _timestamp‚1.7843111995883212e+09 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3709070301159424 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404717568 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 _runtime‚21383.805665046 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚271 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚3719584320129646 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 221840414 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1017036202152448 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 111736258 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚3709070327332864 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 _step‚1062 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 111746245 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1001383488737280 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175¦i¦44Ö3/ _wandbruntime‚21383  _runtime‚21383£Hw×3‰ 0loss/FBt2/detection_loss‚0.20819553648714487 $ _timestamp‚1.784311199675416e+09 0loss/FRt1/assignment_loss‚1.1200936215331372 1loss/SRqqt1/detection_loss‚0.6243319364557465 /loss/FBt1/detection_loss‚0.4893035536339127 epoch‚32 trainer/global_step‚12899 _runtime‚21383.892327286 %loss/total_loss‚5.206100127949498 0loss/FRt2/assignment_loss‚0.1430946229980682 2loss/FBt1/assignment_loss‚0.006802271856718352 2loss/FBt2/assignment_loss‚0.004269918598745965 /loss/FRt1/detection_loss‚0.4696224614978678 1loss/SRqqt2/detection_loss‚0.5151114935109042 _step‚1063 /loss/FRt2/detection_loss‚1.2521365118207237 3loss/SRqqt1/assignment_loss‚0.19191108748077912 3loss/SRqqt2/assignment_loss‚0.18122711207574954⹧fØ3Ž 0loss/FRt1/assignment_loss‚1.1200936215331372 1loss/SRqqt1/detection_loss‚0.6243319364557465 /loss/FBt1/detection_loss‚0.4893035536339127 epoch‚32 trainer/global_step‚12899 _runtime‚21383.892327286 %loss/total_loss‚5.206100127949498 0loss/FRt2/assignment_loss‚0.1430946229980682 2loss/FBt1/assignment_loss‚0.006802271856718352 2loss/FBt2/assignment_loss‚0.004269918598745965 /loss/FRt1/detection_loss‚0.4696224614978678 1loss/SRqqt2/detection_loss‚0.5151114935109042 _step‚1063 /loss/FRt2/detection_loss‚1.2521365118207237 3loss/SRqqt1/assignment_loss‚0.19191108748077912 3loss/SRqqt2/assignment_loss‚0.18122711207574954 0loss/FBt2/detection_loss‚0.20819553648714487 $ _timestamp‚1.784311199675416e+09§hŸ@‘”Ù3‚(:‰ ¥ÛéÒ¡Ñóö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚268.262' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.6524 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0?\MÚ3‚(:à ¥ÛéÒ¶¸€ù disk.sda.out‚ 1339.26953125 disk.dm-0.in‚51072.21484375 disk.sda2.out‚0 cpu‚25.589622846481276$ disk./.usageGB‚565.4893112182617 disk.sda3.in‚ 108.921875 disk.rbd3.in‚0 disk.sda2.in‚0 proc.memory.rssMB‚2804.875$ disk.dm-0.out‚149065.04736328125 disk.rbd3.outÀ\Dk‚ 0.4951171875" disk.nvme0n1.in‚51072.21484375 network.sent‚ 2.4878186e+07 disk.dm-1.out‚ 1339.26953125& disk.rbd0.in‚1.759218604441423e+13 disk.rbd0.out‚0$ disk.nvme0c0n1.in‚51072.21484375' disk.nvme0n1.out‚149065.04736328125$ memory_percent‚82.79731273651123) proc.memory.availableMB‚ 2818.48828125 network.recv‚ 5.306903e+06 disk.sda1.out‚0 disk.sda3.out‚ 1339.26953125 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.sda1.in‚0) disk.nvme0c0n1.out‚149065.04736328125 disk.rbd1.out‚ 55.7216796875 proc.cpu.threads‚17* disk./.usagePercent‚37.950462152993055 disk.dm-1.in‚ 108.921875 disk.rbd1.in‚ 0.52392578125* proc.memory.percent‚17.119598388671875 disk.sda.in‚ 108.921875¯ õÛ•Û3‚(:Š ´ÛéÒ²åãö gpu.0.gpu‚100.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚327.969' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚65.5938 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0†¸HÜ3‚(:½ ´ÛéÒ¤°«ù disk.sda.out‚ 1339.5234375& disk.rbd0.in‚1.759218604441423e+13) disk.nvme0c0n1.out‚149077.49853515625 proc.cpu.threads‚17 disk.rbd3.out‚ 0.4951171875$ memory_percent‚82.79626369476318) proc.memory.availableMB‚ 2818.66015625' disk.nvme0n1.out‚149077.49853515625* proc.memory.percent‚17.119598388671875 network.sent‚ 2.4941667e+07$ disk.dm-0.out‚149077.49853515625 disk.sda3.in‚ 108.921875 disk.sda2.out‚0 disk.rbd1.in‚ 0.52392578125 proc.memory.rssMB‚2804.875 disk.sda1.out‚0 disk.dm-1.in‚ 108.921875 disk.dm-1.out‚ 1339.5234375 disk.drbd2165.out‚0 disk.rbd3.in‚0$ disk.nvme0c0n1.in‚51072.41796875" disk.nvme0n1.in‚51072.41796875 disk.sda2.in‚0 disk.rbd1.out‚ 55.9404296875 network.recv‚ 5.311615e+06) disk./.usagePercent‚37.95073377698688 disk.sda.in‚ 108.921875 disk.sda3.out‚ 1339.5234375 disk.drbd2165.in‚0 cpu‚25.5894892690888 disk.dm-0.in‚51072.41796875 disk.rbd0.out‚0 disk.sda1.in‚0$ disk./.usageGB‚565.4933586120605ñéHžÝ3‚(:“ ÃÛéÒ˜¢ìö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚261.525' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚52.30499999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Œ°òHÞ3‚(:½ ÃÛéÒ«¢Ýù disk.sda1.out‚0 disk.drbd2165.out‚0 disk.sda.out‚ 1339.9140625) disk./.usagePercent‚37.95123375970125 disk.sda1.in‚0 disk.sda2.in‚0 disk.dm-1.out‚ 1339.9140625 disk.dm-0.in‚51072.63671875$ memory_percent‚82.79657363891602$ disk./.usageGB‚565.5008087158203 disk.rbd3.in‚0$ disk.nvme0c0n1.in‚51072.63671875) disk.nvme0c0n1.out‚149089.33642578125 disk.sda.in‚ 108.921875& disk.rbd0.in‚1.759218604441423e+13 network.recv‚ 5.315048e+06 disk.sda2.out‚0 disk.dm-1.in‚ 108.921875* proc.memory.percent‚17.119598388671875 network.sent‚ 2.4946325e+07 disk.sda3.in‚ 108.921875 disk.drbd2165.in‚0 disk.rbd1.out‚ 55.9404296875 disk.sda3.out‚ 1339.9140625 disk.rbd0.out‚0' proc.memory.availableMB‚ 2818.609375 cpu‚25.589332436442533' disk.nvme0n1.out‚149089.33642578125$ disk.dm-0.out‚149089.33642578125 proc.cpu.threads‚17 disk.rbd3.out‚ 0.4951171875" disk.nvme0n1.in‚51072.63671875 disk.rbd1.in‚ 0.52392578125 proc.memory.rssMB‚2804.875M?eá”ß3‚(:‰ ÒÛéÒÿ„×÷ gpu.0.gpu‚91.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚288.968' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚57.7936 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0k¶ƒ1Fà3‚(:» ÒÛéÒ²Îóú network.sent‚ 2.495195e+07 disk.sda1.out‚0 disk.drbd2165.in‚0& disk.nvme0n1.out‚149103.0673828125 disk.sda2.out‚0 disk.dm-1.out‚ 1340.07421875 disk.rbd0.out‚0# disk./.usageGB‚565.507495880127 disk.sda1.in‚0$ disk.nvme0c0n1.in‚51073.05859375" disk.nvme0n1.in‚51073.05859375 disk.sda2.in‚0 network.recv‚ 5.31874e+06 disk.sda3.out‚ 1340.07421875 disk.rbd1.in‚ 0.52392578125) disk./.usagePercent‚37.95168254090878 disk.sda3.in‚ 108.921875 disk.rbd3.out‚ 0.4951171875( disk.nvme0c0n1.out‚149103.0673828125 disk.rbd1.out‚ 56.0029296875 disk.sda.out‚ 1340.07421875$ memory_percent‚82.79814720153809 disk.rbd3.in‚0( proc.memory.availableMB‚ 2818.3515625 cpu‚25.589197178608334 proc.cpu.threads‚17 disk.dm-1.in‚ 108.921875& disk.rbd0.in‚1.759218604441423e+13# disk.dm-0.out‚149103.0673828125 disk.drbd2165.out‚0 disk.sda.in‚ 108.921875 proc.memory.rssMB‚2804.875* proc.memory.percent‚17.119598388671875 disk.dm-0.in‚51073.05859375>XbŒžá3‚(:“ áÛéÒõäòö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚272.302' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚54.46040000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚05É¥ìHâ3‚(:½ áÛéÒäÇàù$ disk.dm-0.out‚149113.82177734375 proc.memory.rssMB‚2804.875" disk.nvme0c0n1.in‚ 51159.796875 network.sent‚ 2.4957175e+07 disk.sda2.in‚0 disk.sda.out‚ 1340.43359375 disk.dm-1.in‚ 108.921875* proc.memory.percent‚17.119598388671875 disk.sda3.in‚ 108.921875 disk.sda3.out‚ 1340.43359375 disk.rbd3.in‚0 disk.rbd1.in‚ 0.52392578125 disk.rbd1.out‚ 56.0029296875 disk.rbd0.out‚0* disk./.usagePercent‚37.951997686183425 disk.rbd3.out‚ 0.4951171875 disk.sda.in‚ 108.921875 disk.dm-0.in‚ 51159.796875 cpu‚25.589067844585095 disk.drbd2165.in‚0 disk.nvme0n1.in‚ 51159.796875 disk.sda2.out‚0& disk.rbd0.in‚1.759218604441423e+13$ memory_percent‚82.79659748077393 proc.cpu.threads‚17 network.recv‚ 5.322075e+06$ disk./.usageGB‚565.5121917724609 disk.sda1.out‚0 disk.drbd2165.out‚0) disk.nvme0c0n1.out‚149113.82177734375 disk.sda1.in‚0 disk.dm-1.out‚ 1340.43359375) proc.memory.availableMB‚ 2818.60546875' disk.nvme0n1.out‚149113.82177734375E™”š4ã3/ _wandbruntime‚21460  _runtime‚21460šUÚÃÉä3à $ _timestamp‚1.784311276757553e+09 _runtime‚21460.974876668 _step‚1064 & lr-AdamW/pg1‚0.0001983860535759084 & lr-AdamW/pg2‚0.0001983860535759084 trainer/global_step‚12949u'dÎå3È & lr-AdamW/pg1‚0.0001983860535759084 & lr-AdamW/pg2‚0.0001983860535759084 trainer/global_step‚12949 $ _timestamp‚1.784311276757553e+09 _runtime‚21460.974876668 _step‚1064¨¨³Î4æ3/ _wandbruntime‚21460  _runtime‚21460a"…u(ç3”P HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3722692336934912 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚3722692363108352 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3733247210935422 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚3721084601626624 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1005180322713600 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12183430146560 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3734874421691904 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚3722692336934912 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚33221430 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚36700160 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 112153532 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3734875793254912 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12183430146560 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15710824462336 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3734875793254912 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 110496263 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12162583880830 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 110496257 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 222659768 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 112163505 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚69921319 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚3733245840907132 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 222649789 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚266 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 _runtime‚21460.976178446 _step‚1065 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 112163505 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3722692363108352 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12182084756992 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1020891110542336 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚3734874421691904 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1005180690804736 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15710787828736 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12161239280508 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 112153532 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1020891515267072 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36633600 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 % _timestamp‚1.7843112767588377e+09 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0Áþ/(è3™P bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1005180322713600 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12183430146560 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3734874421691904 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚3722692336934912 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚33221430 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚36700160 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 112153532 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3734875793254912 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12183430146560 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15710824462336 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3734875793254912 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 110496263 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12162583880830 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 110496257 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 222659768 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 112163505 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚69921319 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚3733245840907132 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 222649789 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚266 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 _runtime‚21460.976178446 _step‚1065 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 112163505 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3722692363108352 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12182084756992 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1020891110542336 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚3734874421691904 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1005180690804736 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15710787828736 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12161239280508 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 112153532 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1020891515267072 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36633600 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 % _timestamp‚1.7843112767588377e+09 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3722692336934912 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚3722692363108352 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3733247210935422 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚3721084601626624 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820©®›84é3/ _wandbruntime‚21462  _runtime‚21462S+‰·™ê3žN E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 trainer/global_step‚12949 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12184355189248 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1005258177212416 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3721362483404800 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7843112781477993e+09 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 222666473 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3735153347457536 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12184355189248 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 222676466 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 112162010 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1020969782108160 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 112171997 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12183009792512 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚69926581 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 112162010 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batchè¶í‹_end/segment.large_pool.allocated‚1175 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 112171997 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3735154719027712 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚3733524646151554 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12183009792512 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3733526016179956 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 _step‚1066 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 110504463 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚21462.365208348 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 110504469 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚33223968 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚36702878 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 222666473 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15712009613312 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚3722970337665024 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3722970337665024 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12163507347188 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚33223973 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1005257809121280 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 110504469 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 110504463 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0*¤Ì²)'ë3£N E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 trainer/global_step‚12949 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12184355189248 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1005258177212416 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3721362483404800 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7843112781477993e+09 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 222666473 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3735153347457536 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12184355189248 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 222676466 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 112162010 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1020969782108160 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 112171997 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12183009792512 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚69926581 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 112162010 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 112171997 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3735154719027712 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚3733524646151554 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12183009792512 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3733526016179956 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 _step‚1066 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 110504463 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚21462.365208348 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 110504469 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚33223968 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚36702878 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 222666473 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15712009613312 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚3722970337665024 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3722970337665024 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12163507347188 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚33223973 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1005257809121280 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 110504469 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 110504463 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0ªóª7l4ì3/ _wandbruntime‚21462  _runtime‚214622ZÂŒí3Œ 0loss/FRt1/assignment_loss‚0.9726218021757334 3loss/SRqqt2/assignment_loss‚0.19894041466292137 /loss/FBt1/detection_loss‚0.3583066772843511 % _timestamp‚1.7843112782309096e+09 _runtime‚21462.447895805 2loss/FBt1/assignment_loss‚0.001011573707609991 3loss/FBt2/assignment_loss‚0.0010342633806501185 /loss/FRt1/detection_loss‚0.4520581402138639 /loss/FRt2/detection_loss‚1.1965135480270344 epoch‚32 _step‚1067 2loss/SRqqt1/assignment_loss‚0.0842099980195459 1loss/SRqqt1/detection_loss‚0.6110354566090457 2loss/SRqqt2/detection_loss‚0.47048992310434024 /loss/FBt2/detection_loss‚0.2327057546348396 &loss/total_loss‚4.7806555463477975 trainer/global_step‚12949 1loss/FRt2/assignment_loss‚0.20172799452786197”¢)Ãî3‘ 2loss/SRqqt1/assignment_loss‚0.0842099980195459 1loss/SRqqt1/detection_loss‚0.6110354566090457 2loss/SRqqt2/detection_loss‚0.47048992310434024 /loss/FBt2/detection_loss‚0.2327057546348396 &loss/total_loss‚4.7806555463477975 trainer/global_step‚12949 1loss/FRt2/assignment_loss‚0.20172799452786197 0loss/FRt1/assignment_loss‚0.9726218021757334 3loss/SRqqt2/assignment_loss‚0.19894041466292137 /loss/FBt1/detection_loss‚0.3583066772843511 % _timestamp‚1.7843112782309096e+09 _runtime‚21462.447895805 2loss/FBt1/assignment_loss‚0.001011573707609991 3loss/FBt2/assignment_loss‚0.0010342633806501185 /loss/FRt1/detection_loss‚0.4520581402138639 /loss/FRt2/detection_loss‚1.1965135480270344 epoch‚32 _step‚1067«ì¿«”ï3‚(:‰ ðÛéÒâàéö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚278.198' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.6396 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ô¯6ð3‚(:« ðÛéÒ’Ñù disk.rbd0.out‚0 disk.sda2.out‚0* proc.memory.percent‚17.119598388671875 proc.cpu.threads‚17 disk.rbd1.in‚ 0.52392578125 disk.sda1.out‚0 disk.sda.out‚ 1340.6875 disk.dm-1.in‚ 108.921875 disk.sda1.in‚0 disk.drbd2165.in‚0 disk.rbd3.in‚0' disk.nvme0n1.out‚149150.45849609375 network.recv‚ 5.325634e+06 disk.sda.in‚ 108.921875& disk.rbd0.in‚1.759218604441423e+13$ disk.dm-0.out‚149150.45849609375 disk.rbd3.out‚ 0.4951171875( proc.memory.availableMB‚ 2818.3671875 proc.memory.rssMB‚2804.875$ disk./.usageGB‚566.2921714782715 disk.dm-1.out‚ 1340.6875 disk.sda3.out‚ 1340.6875 disk.sda2.in‚0) disk./.usagePercent‚38.00434277868693 disk.sda3.in‚ 108.921875! disk.nvme0c0n1.in‚ 51160.21875 network.sent‚ 2.4962261e+07 disk.dm-0.in‚ 51160.21875 disk.drbd2165.out‚0) disk.nvme0c0n1.out‚149150.45849609375 disk.nvme0n1.in‚ 51160.21875$ memory_percent‚82.79805183410645 cpu‚25.58891292361893 disk.rbd1.out‚ 56.0654296875‚HJf5ñ3‚(:ª ÿÛéÒááÀù network.sent‚ 2.5024661e+07! disk.dm-0.out‚149166.82421875 disk.rbd3.in‚0 disk.sda.in‚ 108.921875* proc.memory.percent‚17.119598388671875& disk.nvme0c0n1.out‚149166.82421875 disk.nvme0n1.in‚ 51160.4375 disk.dm-1.in‚ 108.921875 disk.sda2.out‚0# disk./.usageGB‚566.270149230957 disk.sda3.in‚ 108.921875 disk.rbd0.out‚0 disk.dm-1.out‚ 1340.94921875& disk.rbd0.in‚1.759218604441423e+13 disk.dm-0.in‚ 51160.4375 disk.sda1.in‚0 disk.nvme0c0n1.in‚ 51160.4375 disk.rbd1.out‚ 56.0654296875$ memory_percent‚82.79666900634766& proc.memory.availableMB‚ 2818.59375 disk.sda3.out‚ 1340.94921875 disk.drbd2165.in‚0 disk.rbd3.out‚ 0.4951171875$ disk.nvme0n1.out‚149166.82421875 disk.sda2.in‚0 disk.rbd1.in‚ 0.52392578125 disk.sda.out‚ 1340.94921875 proc.memory.rssMB‚2804.875 network.recv‚ 5.329057e+06 disk.drbd2165.out‚0 cpu‚25.588758187539305 proc.cpu.threads‚17* disk./.usagePercent‚38.002864847191766 disk.sda1.out‚0u §`”ò3‚(:‰ ÿÛéÒ‘îÔ‰ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚38.0 gpu.0.powerWatts‚111.642' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.3284 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÿ}M[žó3‚(:“ ŽÜéÒ§½æö gpu.0.gpu‚99.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚287.28' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚57.455999999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ô]üž>ô3‚(:³ ŽÜéÒÜÔêø disk.sda1.out‚0 disk.rbd0.out‚0* proc.memory.percent‚17.119598388671875) disk.nvme0c0n1.out‚149987.88134765625 disk.nvme0n1.in‚ 51160.640625( proc.memory.availableMB‚ 2818.7890625 disk.dm-0.in‚ 51160.640625 disk.rbd3.out‚ 0.4951171875 disk.dm-1.in‚ 108.921875 disk.dm-1.out‚ 1341.65625 network.recv‚ 5.332361e+06 disk.sda2.in‚0 disk.rbd1.in‚ 0.52392578125 disk.rbd1.out‚ 56.1279296875& disk.rbd0.in‚1.759218604441423e+13* disk./.usagePercent‚38.005102865056095 disk.sda3.in‚ 108.921875 disk.rbd3.in‚0 disk.sda.out‚ 1341.65625$ memory_percent‚82.79547691345215 cpu‚25.588616674760374 disk.drbd2165.out‚0 proc.memory.rssMB‚2804.875 disk.sda1.in‚0 disk.drbd2165.in‚0' disk.nvme0n1.out‚149987.88134765625 disk.sda2.out‚0 disk.sda.in‚ 108.921875 network.sent‚ 2.5029194e+07$ disk./.usageGB‚566.3034973144531 disk.sda3.out‚ 1341.65625" disk.nvme0c0n1.in‚ 51160.640625$ disk.dm-0.out‚149987.88134765625 proc.cpu.threads‚17… Ü<”õ3‚(:‰ ÜéÒ¥£þö gpu.0.gpu‚45.0 gpu.0.memory‚14, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚41.0 gpu.0.powerWatts‚109.884' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.9768 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0»gÃCö3‚(:¸ ÜéÒÚÞ¶ú disk.sda1.in‚0 network.sent‚ 2.5034837e+07 disk.rbd1.out‚ 56.1279296875 disk.dm-1.in‚ 108.921875) disk./.usagePercent‚38.00198162119864 disk.drbd2165.in‚0 disk.rbd3.out‚ 0.4951171875% disk.nvme0n1.out‚149997.916015625$ disk./.usageGB‚566.2569885253906 disk.dm-0.in‚51262.33203125 disk.sda1.out‚0$ memory_percent‚82.79542922973633 proc.memory.rssMB‚2804.875' disk.nvme0c0n1.out‚149997.916015625* proc.memory.percent‚17.119598388671875& disk.rbd0.in‚1.759218604441423e+13" disk.dm-0.out‚149997.916015625 disk.sda3.in‚ 108.921875 disk.drbd2165.out‚0 disk.rbd3.in‚0$ disk.nvme0c0n1.in‚51262.33203125 disk.sda.in‚ 108.921875 cpu‚25.58848134190282 network.recv‚ 5.33558e+06" disk.nvme0n1.in‚51262.33203125 disk.sda2.out‚0 disk.rbd1.in‚ 0.52392578125 disk.dm-1.out‚ 1342.24609375 disk.rbd0.out‚0 proc.cpu.threads‚17 disk.sda2.in‚0 disk.sda3.out‚ 1342.24609375 disk.sda.out‚ 1342.24609375' proc.memory.availableMB‚ 2818.796875n¬èŸ÷3‚(:” ¬ÜéÒÆý„÷ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚298.579' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚59.715799999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 :8Fø3‚(:» ¬Üéқϭù$ disk./.usageGB‚565.5174407958984' disk.nvme0c0n1.out‚150795.591796875 disk.rbd0.out‚0 disk.rbd3.in‚0 disk.rbd1.out‚ 56.1904296875& disk.rbd0.in‚1.759218604441423e+13 disk.sda3.out‚ 1342.87109375* proc.memory.percent‚17.119598388671875 disk.dm-1.out‚ 1342.87109375" disk.dm-0.out‚150795.591796875 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd2165.out‚0 proc.memory.rssMB‚2804.875 disk.sda.out‚ 1342.87109375 disk.dm-0.in‚51262.55078125 disk.sda3.in‚ 108.921875) proc.memory.availableMB‚ 2819.17578125 network.sent‚ 2.5040009e+07$ disk.nvme0c0n1.in‚51262.55078125% disk.nvme0n1.out‚150795.591796875 cpu‚25.58834116581134 network.recv‚ 5.339218e+06 disk.sda2.out‚0 disk.sda.in‚ 108.921875 disk.dm-1.in‚ 108.921875 disk.drbd2165.in‚0 proc.cpu.threads‚17 disk.rbd1.in‚ 0.52392578125) disk./.usagePercent‚37.95234995255054" disk.nvme0n1.in‚51262.55078125 disk.sda2.in‚0 disk.rbd3.out‚ 0.4951171875$ memory_percent‚82.79311656951904.K•H4ù3/ _wandbruntime‚21539  _runtime‚21539ƉUËú3Å _step‚1068 ' lr-AdamW/pg1‚0.00019588328691130157 ' lr-AdamW/pg2‚0.00019588328691130157 trainer/global_step‚12999 $ _timestamp‚1.784311355504791e+09 _runtime‚21539.721993233Z®Ð9Ðû3Ê ' lr-AdamW/pg2‚0.00019588328691130157 trainer/global_step‚12999 $ _timestamp‚1.784311355504791e+09 _runtime‚21539.721993233 _step‚1068 ' lr-AdamW/pg1‚0.00019588328691130157¬sz²4ü3/ _wandbruntime‚21539  _runtime‚21539©½Ú½5ý3”P MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15770099501056 _step‚1069 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36633600 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3748822145691136 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚33348575 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3748820774128128 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 % _timestamp‚1.7843113555059066e+09 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15770136134656 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3736592399613952 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚3736592373440512 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 112589257 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚36835880 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3748822145691136 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12229746077184 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 JDDeviceStatsMonitor.on_train_batch_stÝÓå art/oversize_allocations.current‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3747187536957330 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 110906557 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12228400687616 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 223485841 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 223495820 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚280 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚3734978690535424 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12228400687616 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 110906563 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 110906557 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820–¼·n(þ3™P TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚36835880 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3748822145691136 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12229746077184 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3747187536957330 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 110906557 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12228400687616 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 223485841 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 223495820 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚280 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚3734978690535424 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12228400687616 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 110906563 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 110906557 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15770099501056 _step‚1069 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36633600 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3748822145691136 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚33348575 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3748820774128128 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 % _timestamp‚1.7843113555059066e+09 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15770136134656 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3736592399613952 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚3736592373440512 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 112589257 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0­(Km¥”ÿ3‚(:‰ »ÜéÒ¼éíö gpu.0.gpu‚92.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚334.052' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚66.8104 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0”H xC€4‚(:¸ »ÜéÒ —Äú disk.sda2.in‚0 disk.sda1.out‚0 network.recv‚ 5.342742e+06) proc.memory.availableMB‚ 2818.93359375& disk.rbd0.in‚1.759218604441423e+13# disk.dm-0.out‚150805.9345703125 disk.sda3.out‚ 1343.171875$ disk.nvme0c0n1.in‚51262.75390625( disk.nvme0c0n1.out‚150805.9345703125 proc.memory.rssMB‚2804.875" disk.nvme0n1.in‚51262.75390625 disk.dm-1.in‚ 108.921875 disk.sda3.in‚ 108.921875* disk./.usagePercent‚37.951760623206724 disk.sda.in‚ 108.921875 disk.sda.out‚ 1343.171875 disk.rbd0.out‚0# disk./.usageGB‚565.508659362793& disk.nvme0n1.out‚150805.9345703125 disk.dm-1.out‚ 1343.171875 disk.rbd3.in‚0$ memory_percent‚82.79459476470947* proc.memory.percent‚17.119598388671875 cpu‚25.58820724604483 network.sent‚ 2.5045095e+07 disk.sda2.out‚0 disk.rbd1.in‚ 0.52392578125 disk.rbd1.out‚ 56.1904296875 disk.dm-0.in‚51262.75390625 disk.drbd2165.in‚0 disk.drbd2165.out‚0 proc.cpu.threads‚17 disk.sda1.in‚0 disk.rbd3.out‚ 0.4951171875„¥ßí44/ _wandbruntime‚21541  _runtime‚21541ÓÚ#'‚4N K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚3736870374170624 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚70190034 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12229325723136 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3736870400344064 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚3736870400344064 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 223502525 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1009132497596416 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 223512518 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 110914763 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 $ _timestamp‚1.784311356890377e+09 _step‚1070 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3749099699893760 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12208399859862 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1024903814749184 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15771280526336 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3735256597741568 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚21541.107854572 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1024903410031616 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚12999 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 223512518 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚269 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 110914769 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404717568 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12229325723136 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚264 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 112587762 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 112597749 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚36838647 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚3747464972173462 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 112597749 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12230671119872 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚36838911 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1009132129505280 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3749101071463936 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚70189765 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 110914769 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 223502525 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36626432 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3735256572313600 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 112587762 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚33351118 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15771317152768 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12209744460296 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12229325723136 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3736870400344064 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚3736870400344064 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 223502525 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1009132497596416 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 223512518 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 110914763 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 $ _timestamp‚1.784311356890377e+09 _step‚1070 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3749099699893760 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12208399859862 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1024903814749184 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15771280526336 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3735256597741568 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚21541.107854572 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1024903410031616 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚12999 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 223512518 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚269 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 110914769 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404717568 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12229325723136 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚264 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 112587762 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 112597749 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚36838647 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚3747464972173462 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 112597749 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12230671119872 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚36838911 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1009132129505280 M=DeviceStatsMonitor.on_ðÁu‹train_batch_end/reserved_bytes.all.peak‚ 49071259648 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3749101071463936 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚70189765 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 110914769 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 223502525 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36626432 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3735256572313600 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 112587762 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995®Ù”ø¹4„4/ _wandbruntime‚21541  _runtime‚21541Pì…4 /loss/FRt1/detection_loss‚0.4255893283037804 1loss/SRqqt1/detection_loss‚0.5665896023369031 % _timestamp‚1.7843113569770563e+09 0loss/FRt1/assignment_loss‚1.0506828746277053 3loss/SRqqt2/assignment_loss‚0.19415325651455645 0loss/FBt1/detection_loss‚0.49181328094939664 %loss/total_loss‚5.239407977735256 _step‚1071 1loss/FRt2/assignment_loss‚0.18798250798825863 3loss/SRqqt1/assignment_loss‚0.14485329534108998 3loss/FBt1/assignment_loss‚5.210288648901209e-06 /loss/FRt2/detection_loss‚1.3317984584328018 1loss/SRqqt2/detection_loss‚0.5360353888322201 epoch‚32 _runtime‚21541.19409048 4loss/FBt2/assignment_loss‚0.00024066189554605068 /loss/FBt2/detection_loss‚0.3096641122243485 trainer/global_step‚12999—– †4’ /loss/FRt2/detection_loss‚1.3317984584328018 1loss/SRqqt2/detection_loss‚0.5360353888322201 epoch‚32 _runtime‚21541.19409048 4loss/FBt2/assignment_loss‚0.00024066189554605068 /loss/FBt2/detection_loss‚0.3096641122243485 trainer/global_step‚12999 /loss/FRt1/detection_loss‚0.4255893283037804 1loss/SRqqt1/detection_loss‚0.5665896023369031 % _timestamp‚1.7843113569770563e+09 0loss/FRt1/assignment_loss‚1.0506828746277053 3loss/SRqqt2/assignment_loss‚0.19415325651455645 0loss/FBt1/detection_loss‚0.49181328094939664 %loss/total_loss‚5.239407977735256 _step‚1071 1loss/FRt2/assignment_loss‚0.18798250798825863 3loss/SRqqt1/assignment_loss‚0.14485329534108998 3loss/FBt1/assignment_loss‚5.210288648901209e-06¯YAÍ”‡4‚(:‰ ÊÜéÒ²ðö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚265.752' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.1504 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚05ø!+&ˆ4‚(:› ÊÜéÒ‹¦°ø$ disk.nvme0c0n1.out‚ 150815.921875 disk.drbd2165.out‚0 disk.sda.in‚ 108.921875 disk.sda.out‚1343.75 disk.dm-0.in‚51263.17578125 proc.memory.rssMB‚2804.875 network.recv‚ 5.347098e+06 disk.sda1.out‚0 disk.rbd3.in‚0* proc.memory.percent‚17.119598388671875 proc.cpu.threads‚17 disk.dm-1.in‚ 108.921875* disk./.usagePercent‚37.950457288849904 disk.drbd2165.in‚0" disk.nvme0n1.in‚51263.17578125 disk.sda2.out‚0 disk.dm-1.out‚1343.75 disk.sda3.in‚ 108.921875 disk.sda2.in‚0 disk.rbd1.in‚ 0.52392578125 disk.rbd1.out‚ 56.2529296875& disk.rbd0.in‚1.759218604441423e+13 disk.dm-0.out‚ 150815.921875 network.sent‚ 2.509768e+07" disk.nvme0n1.out‚ 150815.921875 cpu‚25.5880469891531$ disk./.usageGB‚565.4892387390137 disk.sda1.in‚0 disk.sda3.out‚1343.75$ disk.nvme0c0n1.in‚51263.17578125$ memory_percent‚82.79609680175781 disk.rbd3.out‚ 0.4951171875 disk.rbd0.out‚0% proc.memory.availableMB‚ 2818.6875p<,9Ÿ‰4‚(:” ÙÜéÒߺ÷ gpu.0.gpu‚100.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚337.288' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚67.45760000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚08™3Š4‚(:¨ ÙÜéÒ«ö‚ú$ memory_percent‚82.79588222503662 cpu‚25.58792385179287) disk./.usagePercent‚37.95076116979305" disk.nvme0c0n1.in‚ 51548.703125 disk.rbd1.in‚ 0.52392578125 disk.rbd1.out‚ 56.2529296875 disk.sda.out‚ 1343.921875 disk.dm-1.out‚ 1343.921875 disk.drbd2165.out‚0 disk.dm-0.in‚ 51548.703125 disk.sda3.in‚ 108.921875* proc.memory.percent‚17.119598388671875 disk.sda2.in‚0 disk.sda2.out‚0) proc.memory.availableMB‚ 2818.72265625% disk.nvme0c0n1.out‚150830.3828125 disk.rbd0.out‚0 network.sent‚ 2.5150282e+07# disk./.usageGB‚565.493766784668 disk.rbd3.out‚ 0.4951171875& disk.rbd0.in‚1.759218604441423e+13 disk.dm-0.out‚150830.3828125 disk.sda3.out‚ 1343.921875 proc.memory.rssMB‚2804.875 proc.cpu.threads‚17 disk.drbd2165.in‚0 disk.sda.in‚ 108.921875 disk.sda1.out‚0 disk.rbd3.in‚0 disk.nvme0n1.in‚ 51548.703125 network.recv‚ 5.350799e+06# disk.nvme0n1.out‚150830.3828125 disk.dm-1.in‚ 108.921875 disk.sda1.in‚0B]”‹4‚(:‰ èÜéÒÈ úö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚245.381' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚49.0762 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÁœQ(?Œ4‚(:´ èÜéÒ–Çíù disk.sda3.in‚ 108.921875' proc.memory.availableMB‚ 2818.734375 disk.sda2.in‚0 disk.nvme0n1.in‚ 51548.90625 disk.sda.out‚ 1344.23828125 disk.dm-1.out‚ 1344.23828125 disk.sda1.in‚0 network.recv‚ 5.354794e+06) disk./.usagePercent‚37.95112982064235( disk.nvme0c0n1.out‚150838.4013671875 disk.rbd0.out‚0 disk.drbd2165.out‚0! disk.nvme0c0n1.in‚ 51548.90625& disk.nvme0n1.out‚150838.4013671875& disk.rbd0.in‚1.759218604441423e+13 disk.sda3.out‚ 1344.23828125 disk.drbd2165.in‚0 disk.sda.in‚ 108.921875 disk.dm-1.in‚ 108.921875 disk.sda1.out‚0 network.sent‚ 2.5155603e+07$ disk./.usageGB‚565.4992599487305 disk.rbd3.out‚ 0.4951171875 disk.dm-0.in‚ 51548.90625# disk.dm-0.out‚150838.4013671875$ memory_percent‚82.79581069946289 disk.sda2.out‚0 disk.rbd1.in‚ 0.52392578125 proc.memory.rssMB‚2804.875* proc.memory.percent‚17.119598388671875 cpu‚25.587759102993115 proc.cpu.threads‚17 disk.rbd3.in‚0 disk.rbd1.out‚ 56.3154296875•¶­•4‚(:Š ÷ÜéÒ„¹ÿö gpu.0.gpu‚100.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚337.816' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚67.5632 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0bfowBŽ4‚(:· ÷ÜéÒ§õÈù proc.memory.rssMB‚2804.875 proc.cpu.threads‚17 disk.dm-1.out‚ 1344.52734375$ disk./.usageGB‚565.5244255065918 disk.rbd0.out‚0 cpu‚25.58764113984498 disk.dm-0.in‚ 51549.328125# disk.dm-0.out‚150849.2978515625 disk.sda1.in‚0" disk.nvme0c0n1.in‚ 51549.328125$ memory_percent‚82.79585838317871 network.sent‚ 2.5164311e+07) disk./.usagePercent‚37.95281870234573 disk.nvme0n1.in‚ 51549.328125& disk.nvme0n1.out‚150849.2978515625 disk.sda2.out‚0 disk.sda1.out‚0( proc.memory.availableMB‚ 2818.7265625 network.recv‚ 5.365278e+06 disk.sda.out‚ 1344.52734375& disk.rbd0.in‚1.759218604441423e+13 disk.rbd3.in‚0* proc.memory.percent‚17.119598388671875 disk.drbd2165.in‚0 disk.rbd1.in‚ 0.52392578125 disk.dm-1.in‚ 108.921875( disk.nvme0c0n1.out‚150849.2978515625 disk.sda.in‚ 108.921875 disk.sda2.in‚0 disk.rbd1.out‚ 56.3154296875 disk.sda3.in‚ 108.921875 disk.sda3.out‚ 1344.52734375 disk.drbd2165.out‚0 disk.rbd3.out‚ 0.4951171875]O ý”4‚(:‰ †ÝéÒ›š÷ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚271.718' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.3436 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¤p‡þD4‚(:¹ †ÝéÒ¾µù disk.drbd2165.in‚0* proc.memory.percent‚17.119598388671875 network.sent‚ 2.5169384e+07 disk.sda3.in‚ 108.921875 disk.sda3.out‚ 1344.66796875 disk.rbd0.out‚0 disk.sda1.in‚0 disk.sda1.out‚0 proc.memory.rssMB‚2804.875 disk.drbd2165.out‚0" disk.nvme0c0n1.in‚ 51549.546875 disk.nvme0n1.in‚ 51549.546875 disk.dm-1.in‚ 108.921875 disk.dm-1.out‚ 1344.66796875 disk.dm-0.in‚ 51549.546875$ disk.dm-0.out‚150862.16748046875) proc.memory.availableMB‚ 2818.89453125 network.recv‚ 5.368908e+06) disk./.usagePercent‚37.95312155925874 disk.rbd1.out‚ 56.3779296875 proc.cpu.threads‚17# disk./.usageGB‚565.528938293457' disk.nvme0n1.out‚150862.16748046875 disk.sda2.out‚0 disk.rbd1.in‚ 0.52392578125 disk.rbd3.in‚0 cpu‚25.5874883220548 disk.rbd3.out‚ 0.4951171875) disk.nvme0c0n1.out‚150862.16748046875 disk.sda.out‚ 1344.66796875& disk.rbd0.in‚1.759218604441423e+13 disk.sda2.in‚0 disk.sda.in‚ 108.921875$ memory_percent‚82.79483318328857Ò‚y4‘4/ _wandbruntime‚21618  _runtime‚21618¾ðàÄÌ’4Æ ' lr-AdamW/pg1‚0.00019339015709583449 ' lr-AdamW/pg2‚0.00019339015709583449 trainer/global_step‚13049 % _timestamp‚1.7843114343319936e+09 _runtime‚21618.549171396 _step‚1072!Ñ“4Ë _step‚1072 ' lr-AdamW/pg1‚0.00019339015709583449 ' lr-AdamW/pg2‚0.00019339015709583449 trainer/global_step‚13049 % _timestamp‚1.7843114343319936e+09 _runtime‚21618.549171396°#¥-4”4/ _wandbruntime‚21618  _runtime‚21618­S”](•4“P XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12274716618240 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 113015009 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚70445184 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3762767126564352 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚3750492409946112 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15829327423488 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3750492436119552 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12276062007808 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 224331872 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚3748872804872192 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚3762767126564352 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404724736 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3762768498127360 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 111316863 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1012928963481600 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1028758695629824 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚279 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚36969459 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 $ _timestamp‚1.784311434333083e+09 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 111316857 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12274716618240 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚3750492436119552 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12253713506724 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0È“Ð(–4˜P ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3750492409946112 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 111316857 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚3761126492950948 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 113005036 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚33475730 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15829364057088 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3762768498127360 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 113005036 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 trainer/global_step‚13049 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚70445468 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1028758290905088 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36633600 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 111316863 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1012928963481600 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1028758695629824 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚279 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚36969459 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 $ _timestamp‚1.784311434333083e+09 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 111316857 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12274716618240 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚3750492436119552 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12253713506724 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12274716618240 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 113015009 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚70445184 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3762767126564352 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚3750492409946112 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15829327423488 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3750492436119552 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12276062007808 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 224331872 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚3748872804872192 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚3762767126564352 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404724736 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3762768498127360 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12276987050496 ú}“&$]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12276987050496 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚268 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚3761405298195370 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1028836961753088 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 111325069 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 111325063 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12254636972970 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3749150661222400 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1013006817980416 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 111325069 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 113013514 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 224348570 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 224338577 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404717568 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1013006449889280 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15830511863808 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚273 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3750770410676224 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15830548490240 _step‚1074 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12275641653760 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 111325063 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3750770436849664 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3749150686650368 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 trainer/global_step‚13049 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 113013514 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1028837366470656 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3763047423900160 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚70450459 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚36972191 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 % _timestamp‚1.7843114357180889e+09 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 _runtime‚21619.935478937 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3763046052329984 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3761406668223772 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚33478268 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 113023501 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚70450732 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12255981573404ŒÙ?)'™4£N RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1028836961753088 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 111325069 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 111325063 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12254636972970 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3749150661222400 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1013006817980416 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 111325069 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 113013514 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 224348570 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 224338577 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404717568 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1013006449889280 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15830511863808 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚273 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3750770410676224 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15830548490240 _step‚1074 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12275641653760 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 111325063 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3750770436849664 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3749150686650368 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 trainer/global_step‚13049 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 113013514 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1028837366470656 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3763047423900160 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚70450459 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚36972191 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 % _timestamp‚1.7843114357180889e+09 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 _runtime‚21619.935478937 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3763046052329984 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3761406668223772 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚33478268 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 113023501 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚70450732 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12255981573404 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 224348570 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚3750770436849664 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12276987050496 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12276987050496 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚268 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚3761405298195370²<üŽ"4š4/ _wandbruntime‚21620  _runtime‚21620Îtð†›4 1loss/SRqqt1/detection_loss‚0.5017227107338187 1loss/SRqqt2/detection_loss‚0.4943434611857851 /loss/FBt1/detection_loss‚0.3525429517165041 /loss/FBt2/detection_loss‚0.3185424965995774 _runtime‚21620.021450392 0loss/FRt1/assignment_loss‚1.1063627313892797 3loss/SRqqt2/assignment_loss‚0.21790148292157238 _step‚1075 epoch‚32 1loss/FRt2/assignment_loss‚0.18615077699770768 3loss/FBt1/assignment_loss‚4.816391808030828e-05 0loss/FRt1/detection_loss‚0.44227651507405386 .loss/FRt2/detection_loss‚1.224954030195916 trainer/global_step‚13049 % _timestamp‚1.7843114358045554e+09 %loss/total_loss‚4.966923134603269 3loss/SRqqt1/assignment_loss‚0.12207636995311394 4loss/FBt2/assignment_loss‚1.4439178601164823e-06zSPîœ4’ 3loss/SRqqt2/assignment_loss‚0.21790148292157238 _step‚1075 epoch‚32 1loss/FRt2/assignment_loss‚0.18615077699770768 3loss/FBt1/assignment_loss‚4.816391808030828e-05 0loss/FRt1/detection_loss‚0.44227651507405386 .loss/FRt2/detection_loss‚1.224954030195916 trainer/global_step‚13049 % _timestamp‚1.7843114358045554e+09 %loss/total_loss‚4.966923134603269 3loss/SRqqt1/assignment_loss‚0.12207636995311394 4loss/FBt2/assignment_loss‚1.4439178601164823e-06 1loss/SRqqt1/detection_loss‚0.5017227107338187 1loss/SRqqt2/detection_loss‚0.4943434611857851 /loss/FBt1/detection_loss‚0.3525429517165041 /loss/FBt2/detection_loss‚0.3185424965995774 _runtime‚21620.021450392 0loss/FRt1/assignment_loss‚1.1063627313892797³"ÍX:”4‚(:‰ •ÝéÒÔ¥îö gpu.0.gpu‚51.0 gpu.0.memory‚16, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚317.399' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚63.4798 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0õêÃHž4‚(:½ •Ýéҙȕú$ disk.nvme0c0n1.in‚51677.87890625) disk.nvme0c0n1.out‚150873.98388671875 disk.dm-0.in‚51677.87890625) disk./.usagePercent‚37.95361078965667 disk.rbd0.out‚0 cpu‚25.58736817077513 disk.drbd2165.in‚0 disk.rbd3.in‚0 disk.rbd1.in‚ 0.52392578125 disk.sda.out‚ 1344.91015625 network.recv‚ 5.379565e+06$ disk./.usageGB‚565.5362281799316 disk.sda2.in‚0" disk.nvme0n1.in‚51677.87890625& disk.rbd0.in‚1.759218604441423e+13# memory_percent‚82.7965259552002 proc.memory.rssMB‚2804.875 network.sent‚ 2.51783e+07' disk.nvme0n1.out‚150873.98388671875 disk.drbd2165.out‚0* proc.memory.percent‚17.119598388671875 disk.rbd1.out‚ 56.3779296875( proc.memory.availableMB‚ 2818.6171875 disk.rbd3.out‚ 0.4951171875 disk.sda2.out‚0 disk.dm-1.in‚ 108.921875 disk.dm-1.out‚ 1344.91015625$ disk.dm-0.out‚150873.98388671875 disk.sda1.out‚0 disk.sda3.out‚ 1344.91015625 disk.sda.in‚ 108.921875 disk.sda1.in‚0 disk.sda3.in‚ 108.921875 proc.cpu.threads‚17œ®ŸŸ4‚(:” ¤ÝéÒîΆ÷ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚285.735' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚57.147000000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ŒÓ¢B 4‚(:· ¤ÝéÒ½˜ìù disk.sda1.in‚0 disk.drbd2165.in‚0& disk.rbd0.in‚1.759218604441423e+13) proc.memory.availableMB‚ 2818.33203125* proc.memory.percent‚17.119598388671875 disk.rbd1.out‚ 56.4404296875 proc.memory.rssMB‚2804.875 disk.rbd3.out‚ 0.4951171875 disk.sda.out‚ 1345.1796875" disk.nvme0n1.in‚51678.30078125$ disk.nvme0n1.out‚150882.98828125 network.sent‚ 2.5241619e+07$ disk./.usageGB‚565.5428352355957! disk.dm-0.out‚150882.98828125$ disk.nvme0c0n1.in‚51678.30078125* disk./.usagePercent‚37.954054194705975 disk.sda2.in‚0 disk.sda.in‚ 108.921875 disk.dm-1.in‚ 108.921875$ memory_percent‚82.79826641082764 cpu‚25.587216151689173 disk.rbd3.in‚0 disk.dm-0.in‚51678.30078125 disk.sda1.out‚0 disk.sda2.out‚0 disk.rbd0.out‚0 disk.drbd2165.out‚0 disk.dm-1.out‚ 1345.1796875 proc.cpu.threads‚17 disk.rbd1.in‚ 0.52392578125 network.recv‚ 5.384011e+06 disk.sda3.in‚ 108.921875 disk.sda3.out‚ 1345.1796875& disk.nvme0c0n1.out‚150882.988281254òQ”¡4‚(:‰ ³ÝéÒ¨‚Ñø gpu.0.gpu‚91.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚227.333' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚45.4666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÿýÊH¢4‚(:½ ³ÝéÒ­¾÷ù) proc.memory.availableMB‚ 2818.33203125 disk.sda.out‚ 1345.4609375 network.recv‚ 5.386904e+06 disk.dm-1.in‚ 108.921875 disk.rbd0.out‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd2165.out‚0 disk.rbd3.out‚ 0.4951171875$ disk.nvme0c0n1.in‚51678.51953125) disk./.usagePercent‚37.95434553127994 disk.sda3.in‚ 108.921875 disk.drbd2165.in‚0 disk.rbd3.in‚0 disk.rbd1.in‚ 0.52392578125" disk.nvme0n1.in‚51678.51953125 disk.sda2.in‚0 disk.rbd1.out‚ 56.4404296875 disk.dm-1.out‚ 1345.4609375 disk.dm-0.in‚51678.51953125$ memory_percent‚82.79826641082764 proc.memory.rssMB‚2804.875 cpu‚25.58707488894093 proc.cpu.threads‚17# disk./.usageGB‚565.547176361084$ disk.dm-0.out‚150899.38232421875* proc.memory.percent‚17.119598388671875 network.sent‚ 2.5246755e+07 disk.sda.in‚ 108.921875& disk.rbd0.in‚1.759218604441423e+13) disk.nvme0c0n1.out‚150899.38232421875' disk.nvme0n1.out‚150899.38232421875 disk.sda2.out‚0 disk.sda3.out‚ 1345.4609375Bé~q’£4‚(:‡ ÂÝéÒ›“†÷ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚280.04' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚56.008 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‘ƒ'uK¤4‚(:À ÂÝéÒ¸ÌÏù disk.rbd1.in‚ 0.52392578125) proc.memory.availableMB‚ 2818.33984375 disk.rbd0.out‚0 disk.sda3.in‚ 108.921875 disk.rbd3.in‚0' disk.nvme0n1.out‚150907.48779296875$ memory_percent‚82.79821872711182 disk.sda1.out‚0* disk./.usagePercent‚37.954825801414174 disk.dm-1.out‚ 1345.6796875 disk.dm-0.in‚51678.72265625 disk.drbd2165.out‚0 cpu‚25.586934811447044$ disk./.usageGB‚565.5543327331543 disk.sda.out‚ 1345.6796875 proc.memory.rssMB‚2804.875 disk.sda.in‚ 108.921875$ disk.dm-0.out‚150907.48779296875 disk.rbd1.out‚ 56.5029296875 network.sent‚ 2.5251299e+07 network.recv‚ 5.390464e+06 disk.sda3.out‚ 1345.6796875 disk.rbd3.out‚ 0.4951171875) disk.nvme0c0n1.out‚150907.48779296875" disk.nvme0n1.in‚51678.72265625 disk.sda2.out‚0& disk.rbd0.in‚1.759218604441423e+13 proc.cpu.threads‚17 disk.drbd2165.in‚0$ disk.nvme0c0n1.in‚51678.72265625 disk.sda2.in‚0* proc.memory.percent‚17.119598388671875 disk.dm-1.in‚ 108.921875 disk.sda1.in‚0Ô„¯p”¥4‚(:‰ ÑÝéÒË–€÷ gpu.0.gpu‚99.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚246.733' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚49.3466 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0k ¿C¦4‚(:¸ ÑÝéÒ•©Òù disk.sda1.out‚0 disk.sda3.in‚ 108.921875 disk.drbd2165.in‚0* proc.memory.percent‚16.889429092407227 cpu‚25.58680561480924 disk.sda.out‚ 1345.98828125 disk.dm-1.out‚ 1345.9921875 disk.sda1.in‚0' disk.nvme0c0n1.out‚150921.298828125$ memory_percent‚82.66735076904297& proc.memory.availableMB‚ 2839.78125" proc.memory.rssMB‚ 2767.1640625 disk.sda2.in‚0 disk.sda2.out‚0$ disk./.usageGB‚565.5612869262695 disk.rbd0.out‚0 disk.drbd2165.out‚0 disk.rbd3.out‚ 0.4951171875! disk.nvme0n1.in‚ 51846.1640625 disk.rbd1.in‚ 0.52392578125 network.recv‚ 5.393839e+06& disk.rbd0.in‚1.759218604441423e+13" disk.dm-0.out‚150921.548828125 disk.rbd3.in‚0 network.sent‚ 2.5256911e+07 disk.dm-0.in‚ 51846.1640625 proc.cpu.threads‚17# disk.nvme0c0n1.in‚ 51846.1640625% disk.nvme0n1.out‚150921.548828125 disk.rbd1.out‚ 56.5029296875* disk./.usagePercent‚37.955292503149096 disk.dm-1.in‚ 108.921875 disk.sda3.out‚ 1345.9921875 disk.sda.in‚ 108.921875Á »;4§4/ _wandbruntime‚21697  _runtime‚21697Ȫ˨4Å ' lr-AdamW/pg1‚0.00019090682425857631 ' lr-AdamW/pg2‚0.00019090682425857631 trainer/global_step‚13099 $ _timestamp‚1.784311512968953e+09 _runtime‚21697.186123089 _step‚1076\rõЩ4Ê ' lr-AdamW/pg1‚0.00019090682425857631 ' lr-AdamW/pg2‚0.00019090682425857631 trainer/global_step‚13099 $ _timestamp‚1.784311512968953e+09 _runtime‚21697.186123089 _step‚1076´A¿Þˆ4ª4/ _wandbruntime‚21697  _runtime‚21697ù€¯v«4”P _runtime‚21697.187343142 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 111727163 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 113440761 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1016803651956736 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3764392472625152 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 % _timestamp‚1.7843115129700165e+09 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15888591194112 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3776713479000576 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 111727157 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12322377938432 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3776714850563584 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚37103127 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12299950619832 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3764392446451712 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 225167924 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚33602875 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.smal™BÔX¤l_pool.allocated‚37103397 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 trainer/global_step‚13099 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 113440761 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3775068189001146 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1032692243150848 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚3762766868353024 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36633600 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚3775066818972856 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚3762766893780992 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚3764392472625152 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚275 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 113430788 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚3764392446451712 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15888554560512 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12322377938432 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚70706277 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚270 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1032691838426112 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12321032548864 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚3776713479000576 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3776714850563584 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3776713479000576 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 111727157 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12322377938432 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3776714850563584 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚37103127 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12299950619832 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3764392446451712 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 225167924 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚33602875 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚37103397 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 trainer/global_step‚13099 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 113440761 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3775068189001146 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1032692243150848 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚3762766868353024 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36633600 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚3775066818972856 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚3762766893780992 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚3764392472625152 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚275 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 113430788 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚3764392446451712 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15888554560512 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12322377938432 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚70706277 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚270 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1032691838426112 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12321032548864 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚3776713479000576 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3776714850563584 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175µÄý_O4­4/ _wandbruntime‚21698  _runtime‚21698ðàYÇ$'®4žN UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 225174629 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚37105923 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 _runtime‚21698.573081451 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 111735369 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 225184622 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1016881138364416 _step‚1078 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15889739076608 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 113439266 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12302218686512 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 113449253 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚3764670473355264 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3764670473355264 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3764670447181824 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12321957584384 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 225184622 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3763044750131200 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12300874086078 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 225174629 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 trainer/global_step‚13099 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12323302981120 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3776992404766208 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36626432 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3763044775559168 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚37105664 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1032770914067456 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404717568 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12323302981120 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15889775703040 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 111735363 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚264 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚70711082 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3775346994245680 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚3764670447181824 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3764670473355264 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3764670447181824 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12321957584384 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 225184622 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3763044750131200 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12300874086078 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 225174629 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 trainer/global_step‚13099 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12323302981120 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3776992404766208 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36626432 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3763044775559168 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚37105664 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1032770914067456 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404717568 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12323302981120 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15889775703040 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 111735363 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚264 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚70711082 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3775346994245680 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚3764670447181824 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 225184622 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1016881138364416 _step‚1078¶¿NÎé4°4/ _wandbruntime‚21698  _runtime‚21698ÐÀÉر4 epoch‚32 0loss/FRt1/assignment_loss‚1.0150294494343506 1loss/FRt2/assignment_loss‚0.20395949917534986 3loss/SRqqt2/assignment_loss‚0.22549158444211473 %loss/total_loss‚5.082916861788009 trainer/global_step‚13099 % _timestamp‚1.7843115144417691e+09 _runtime‚21698.65867282 _step‚1079 3loss/FBt2/assignment_loss‚0.0009650410430118581 1loss/SRqqt2/detection_loss‚0.5534941061209968 /loss/FBt1/detection_loss‚0.4507917458581182 /loss/FRt2/detection_loss‚1.2761868119675153 1loss/SRqqt1/detection_loss‚0.5937066431391155 3loss/SRqqt1/assignment_loss‚0.11952017364964915 4loss/FBt1/assignment_loss‚0.00021574133632877116 0loss/FRt1/detection_loss‚0.40903766841966466 /loss/FBt2/detection_loss‚0.2345183972017936îà¾ô²4’ 3loss/SRqqt2/assignment_loss‚0.22549158444211473 %loss/total_loss‚5.082916861788009 trainer/global_step‚13099 % _timestamp‚1.7843115144417691e+09 _runtime‚21698.65867282 _step‚1079 3loss/FBt2/assignment_loss‚0.0009650410430118581 1loss/SRqqt2/detection_loss‚0.5534941061209968 /loss/FBt1/detection_loss‚0.4507917458581182 /loss/FRt2/detection_loss‚1.2761868119675153 1loss/SRqqt1/detection_loss‚0.5937066431391155 3loss/SRqqt1/assignment_loss‚0.11952017364964915 4loss/FBt1/assignment_loss‚0.00021574133632877116 0loss/FRt1/detection_loss‚0.40903766841966466 /loss/FBt2/detection_loss‚0.2345183972017936 epoch‚32 0loss/FRt1/assignment_loss‚1.0150294494343506 1loss/FRt2/assignment_loss‚0.20395949917534986·^ñ½5Ÿ³4‚(:” àÝéÒšËëö gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚274.423' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚54.884600000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Lz¥”E´4‚(:º àÝéÒ§÷ëú* disk./.usagePercent‚37.955600224205256 disk.rbd3.out‚ 0.4951171875 disk.sda3.in‚ 108.921875 disk.drbd2165.out‚0# disk.nvme0c0n1.in‚ 51846.3828125 disk.sda.in‚ 108.921875 disk.sda.out‚ 1346.46875 disk.dm-1.out‚ 1346.46875 disk.dm-0.in‚ 51846.3828125 disk.sda3.out‚ 1346.46875$ memory_percent‚82.66613483428955 disk.sda2.out‚0 cpu‚25.58663921492774 network.sent‚ 2.5262038e+07 network.recv‚ 5.397156e+06 disk.rbd1.in‚ 0.52392578125 disk.sda1.out‚0$ disk./.usageGB‚565.5658721923828 disk.sda2.in‚0$ disk.dm-0.out‚150929.02880859375 disk.sda1.in‚0) disk.nvme0c0n1.out‚150929.02880859375! disk.nvme0n1.in‚ 51846.3828125 disk.rbd1.out‚ 56.5654296875 disk.drbd2165.in‚0 proc.cpu.threads‚17 disk.rbd3.in‚0' disk.nvme0n1.out‚150929.02880859375 disk.dm-1.in‚ 108.921875& disk.rbd0.in‚1.759218604441423e+13 disk.rbd0.out‚0) proc.memory.availableMB‚ 2839.98046875" proc.memory.rssMB‚ 2767.1640625* proc.memory.percent‚16.889429092407227XÒ®0Ÿµ4‚(:” ïÝéÒÜñÕö gpu.0.gpu‚90.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚273.894' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚54.778800000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ö¥P(G¶4‚(:¼ ïÝéÒ¾¤Œú disk.sda1.out‚0 disk.sda3.out‚ 1347.27734375% disk.nvme0n1.out‚150940.681640625 disk.sda.in‚ 108.921875 disk.dm-1.in‚ 108.921875" disk.dm-0.out‚150940.681640625 network.sent‚ 2.5325366e+07 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.sda2.out‚0 disk.rbd1.out‚ 56.5654296875 disk.dm-1.out‚ 1347.27734375 disk.rbd3.out‚ 0.4951171875* disk./.usagePercent‚37.956241011063454# disk.nvme0c0n1.in‚ 51846.8046875! disk.nvme0n1.in‚ 51846.8046875 disk.sda1.in‚0 proc.cpu.threads‚17$ disk./.usageGB‚565.5754203796387' disk.nvme0c0n1.out‚150940.681640625 disk.sda.out‚ 1347.27734375& disk.rbd0.in‚1.759218604441423e+13 cpu‚25.58652375654681 disk.rbd3.in‚0 disk.sda2.in‚0 disk.sda3.in‚ 108.921875# memory_percent‚82.6676607131958" proc.memory.rssMB‚ 2767.1640625 network.recv‚ 5.401768e+06 disk.rbd0.out‚0) proc.memory.availableMB‚ 2839.73046875* proc.memory.percent‚16.889429092407227 disk.rbd1.in‚ 0.52392578125 disk.dm-0.in‚ 51846.8046875*íQ]Ÿ·4‚(:” þÝéÒ®¾¥÷ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚131.754' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚26.350799999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ë3M¸4‚(: þÝéÒ˜‡‹ú disk.drbd2165.in‚0 disk.drbd2165.out‚0# disk.nvme0c0n1.in‚ 51847.0078125( proc.memory.availableMB‚ 2839.6796875 proc.cpu.threads‚17 disk.rbd0.out‚0 disk.sda3.in‚ 108.921875 disk.rbd1.out‚ 56.6904296875 network.sent‚ 2.5330557e+07' disk.nvme0n1.out‚150954.09423828125* proc.memory.percent‚16.889429092407227 disk.sda.out‚ 1347.49609375& disk.rbd0.in‚1.759218604441423e+13$ memory_percent‚82.66797065734863 network.recv‚ 5.405606e+06$ disk./.usageGB‚565.5788879394531 disk.sda1.in‚0 disk.rbd1.in‚ 0.52392578125" proc.memory.rssMB‚ 2767.1640625 cpu‚25.586360729774295) disk./.usagePercent‚37.95647372191208$ disk.dm-0.out‚150954.09423828125 disk.sda1.out‚0 disk.rbd3.in‚0! disk.nvme0n1.in‚ 51847.0078125 disk.sda.in‚ 108.921875 disk.dm-1.out‚ 1347.49609375 disk.dm-0.in‚ 51847.0078125 disk.sda3.out‚ 1347.49609375 disk.rbd3.out‚ 0.4951171875) disk.nvme0c0n1.out‚150954.09423828125 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-1.in‚ 108.921875DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3790659831436800 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚254 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1036625837074432 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 112137457 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12347532333262 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚33730030 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1020677972340736 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 _runtime‚21775.879299245 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1020677604249600 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12367348479488 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚70966901 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 112137463 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 trainer/global_step‚13149 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 113856540 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3790661202999808 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚70966647 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚3789007144994764 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12346187732940 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚3778292482957312 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚37236622 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚3790659831436800 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 112137457 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 % _timestamp‚1.7843115916619477e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3778292482957312 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 113866513 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3789008515023054 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 112137463 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 _step‚1081 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3790661202999808 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12368693869056 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 113856540 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 113866513 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚33730025 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚249 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚37236871 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640Ëܘ³.!Â4™P M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚70966647 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚3789007144994764 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12346187732940 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚3778292482957312 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚37236622 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚3790659831436800 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 112137457 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 % _timestamp‚1.7843115916619477e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3778292482957312 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 113866513 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3789008515023054 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 112137463 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 _step‚1081 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3790661202999808 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12368693869056 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 113856540 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 113866513 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚33730025 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚249 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚37236871 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12367348479488 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15947830197248 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚3778292509130752 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1036625434446848 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚3776660957261824 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3778292509130752 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 226003976 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15947864733696 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 225993997 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3790659831436800 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚254 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1036625837074432 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 112137457 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12347532333262 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚33730030 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 G?DeviceStatsMonitor.on_train_batch"ÕHvñ_start/inactive_split.all.peak‚572 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1020677972340736 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 _runtime‚21775.879299245 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1020677604249600 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12367348479488 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚70966901 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 112137463 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 trainer/global_step‚13149 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 113856540 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3790661202999808 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290¹ø4Ã4/ _wandbruntime‚21777  _runtime‚21777!Ñ#'Ä4N UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚70972136 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 trainer/global_step‚13149 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 112145663 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12347111199186 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1020755090657280 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 $ _timestamp‚1.784311593047849e+09 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3790938757202432 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 112145663 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12368273515008 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1036704519967744 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚37239563 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1020755458748416 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 226010681 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 226010681 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚249 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12369618911744 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3790940128772608 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚21777.265193424 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 113865018 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚70971887 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3778570483687424 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15949061219328 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15949026690048 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12348455799620 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3789287320267588 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚37239319 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 226020674 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 112145669 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12369618911744 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 226020674 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12368273515008 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1036704117347328 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3776938839040000 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3778570509860864 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 112145669 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚3778570509860864 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 113875005 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚244 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3776938864467968 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 _step‚1082gåd'('Å4¢N TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3790938757202432 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 112145663 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12368273515008 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1036704519967744 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚37239563 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1020755458748416 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 226010681 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 226010681 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚249 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12369618911744 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3790940128772608 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚21777.265193424 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 113865018 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚70971887 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3778570483687424 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15949061219328 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15949026690048 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12348455799620 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3789287320267588 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚37239319 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 226020674 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 112145669 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12369618911744 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 226020674 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12368273515008 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1036704117347328 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3776938839040000 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3778570509860864 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 112145669 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚3778570509860864 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 113875005 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚244 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3776938864467968 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 _step‚1082 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚70972136 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 trainer/global_step‚13149 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 112145663 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12347111199186 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1020755090657280 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 $ _timestamp‚1.784311593047849e+09º¥+=B4Æ4/ _wandbruntime‚21777  _runtime‚21777Tiø\Ç4‹ 0loss/FBt2/detection_loss‚0.30784546492671194 _step‚1083 1loss/FRt2/assignment_loss‚0.16386921305254282 3loss/FBt2/assignment_loss‚2.744799257433174e-05 /loss/FRt1/detection_loss‚0.4334268565083102 0loss/SRqqt1/detection_loss‚0.518179835143642 1loss/SRqqt2/detection_loss‚0.5167639594007506 %loss/total_loss‚5.278965560163536 _runtime‚21777.35109721 epoch‚32 0loss/FRt1/assignment_loss‚1.1104798134642118 /loss/FRt2/detection_loss‚1.3089789958542397 trainer/global_step‚13149 $ _timestamp‚1.784311593134196e+09 3loss/SRqqt1/assignment_loss‚0.15982344198841134 3loss/SRqqt2/assignment_loss‚0.22504288852272317 4loss/FBt1/assignment_loss‚0.00010219919187289247 /loss/FBt1/detection_loss‚0.5344254441175459¿\¯IÈ4 0loss/FRt1/assignment_loss‚1.1104798134642118 /loss/FRt2/detection_loss‚1.3089789958542397 trainer/global_step‚13149 $ _timestamp‚1.784311593134196e+09 3loss/SRqqt1/assignment_loss‚0.15982344198841134 3loss/SRqqt2/assignment_loss‚0.22504288852272317 4loss/FBt1/assignment_loss‚0.00010219919187289247 /loss/FBt1/detection_loss‚0.5344254441175459 0loss/FBt2/detection_loss‚0.30784546492671194 _step‚1083 1loss/FRt2/assignment_loss‚0.16386921305254282 3loss/FBt2/assignment_loss‚2.744799257433174e-05 /loss/FRt1/detection_loss‚0.4334268565083102 0loss/SRqqt1/detection_loss‚0.518179835143642 1loss/SRqqt2/detection_loss‚0.5167639594007506 %loss/total_loss‚5.278965560163536 _runtime‚21777.35109721 epoch‚32»“\~Ö’É4‚(:‡ «ÞéÒó‰±÷ gpu.0.gpu‚90.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚271.19' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.238 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0.´i=Ê4‚(:² «ÞéÒš±˜ù disk.rbd3.out‚ 0.4951171875 disk.sda.in‚ 108.921875 disk.dm-1.in‚ 108.921875 disk.sda2.out‚0 network.sent‚ 2.5345044e+07$ disk./.usageGB‚565.5972099304199 disk.drbd2165.out‚0 disk.rbd1.in‚ 0.52392578125 disk.rbd1.out‚ 56.7529296875 cpu‚25.58596683374308& disk.rbd0.in‚1.759218604441423e+13" proc.memory.rssMB‚ 2767.1640625 disk.dm-1.out‚ 1348.49609375 disk.sda3.in‚ 108.921875$ memory_percent‚82.66334533691406% proc.memory.availableMB‚ 2840.4375* proc.memory.percent‚16.889429092407227 disk.rbd3.in‚0 disk.sda2.in‚0 disk.dm-0.out‚150988.0546875 proc.cpu.threads‚17 disk.sda.out‚ 1348.49609375 disk.dm-0.in‚ 51954.3984375 disk.sda3.out‚ 1348.49609375) disk./.usagePercent‚37.95770332609904 disk.drbd2165.in‚0 network.recv‚ 5.414745e+06! disk.nvme0n1.in‚ 51954.3984375 disk.rbd0.out‚0 disk.sda1.out‚0# disk.nvme0c0n1.in‚ 51954.3984375% disk.nvme0c0n1.out‚150988.0546875# disk.nvme0n1.out‚150988.0546875 disk.sda1.in‚0rûMZ”Ë4‚(:‰ ºÞéÒƒ ˆ÷ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚333.229' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚66.6458 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‚ß3EÌ4‚(:º ºÞéÒ©çìù disk.sda.in‚ 108.921875 disk.dm-1.in‚ 108.921875* proc.memory.percent‚16.889429092407227 proc.cpu.threads‚17 disk.rbd3.in‚0 disk.sda.out‚ 1348.6796875 disk.dm-0.in‚ 51954.8203125$ memory_percent‚82.66329765319824 cpu‚25.58583738151133$ disk./.usageGB‚565.6064300537109 disk.rbd0.out‚0 disk.sda1.in‚0( disk./.usagePercent‚37.9583220963093 disk.rbd1.out‚ 56.8154296875& disk.rbd0.in‚1.759218604441423e+13 disk.sda3.in‚ 108.921875 network.recv‚ 5.419418e+06 disk.rbd3.out‚ 0.4951171875( disk.nvme0c0n1.out‚150997.7861328125! disk.nvme0n1.in‚ 51954.8203125# disk.dm-0.out‚150997.7861328125 disk.sda3.out‚ 1348.6796875" proc.memory.rssMB‚ 2767.1640625# disk.nvme0c0n1.in‚ 51954.8203125 disk.drbd2165.in‚0 disk.drbd2165.out‚0 network.sent‚ 2.5408344e+07 disk.sda2.in‚0 disk.dm-1.out‚ 1348.6796875 disk.sda1.out‚0( proc.memory.availableMB‚ 2840.4453125& disk.nvme0n1.out‚150997.7861328125 disk.sda2.out‚0 disk.rbd1.in‚ 0.52392578125­º±S”Í4‚(:‰ ÉÞéÒ·Øðö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚257.997' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚51.5994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0lÂHáCÎ4‚(:¸ ÉÞéÒÉÌ»ù disk.rbd0.out‚0 network.recv‚ 5.422762e+06 disk.sda1.out‚0 disk.sda3.out‚ 1349.0859375 disk.sda2.out‚0 disk.sda.in‚ 108.921875$ disk./.usageGB‚565.6102638244629 disk.sda2.in‚0 proc.cpu.threads‚17 disk.rbd3.out‚ 0.4951171875! disk.nvme0n1.in‚ 51955.0390625$ memory_percent‚82.66324996948242* proc.memory.percent‚16.889429092407227) disk./.usagePercent‚37.95857938388121 disk.sda1.in‚0 disk.drbd2165.in‚0 disk.rbd1.in‚ 0.52392578125 disk.dm-1.in‚ 108.921875& disk.rbd0.in‚1.759218604441423e+13 disk.sda.out‚ 1349.0859375 disk.dm-1.out‚ 1349.0859375 disk.drbd2165.out‚0 disk.rbd3.in‚0' disk.nvme0c0n1.out‚151011.787109375 disk.sda3.in‚ 108.921875# disk.nvme0c0n1.in‚ 51955.0390625 disk.dm-0.in‚ 51955.0390625" disk.dm-0.out‚151011.787109375' proc.memory.availableMB‚ 2840.453125" proc.memory.rssMB‚ 2767.1640625 cpu‚25.585676282937044 network.sent‚ 2.5413458e+07% disk.nvme0n1.out‚151011.787109375 disk.rbd1.out‚ 56.8154296875N7$+”Ï4‚(:‰ ØÞéÒ‚„÷ gpu.0.gpu‚92.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚47.0 gpu.0.powerWatts‚339.701' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚67.9402 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0n±’MOÐ4‚(:Ä ØÞéÒâÄŒù disk.drbd2165.out‚0' disk.nvme0n1.out‚151019.73779296875 disk.sda.out‚ 1349.44921875 disk.dm-1.out‚ 1349.44921875$ disk.nvme0c0n1.in‚52048.76953125 disk.dm-1.in‚ 108.921875 disk.dm-0.in‚52048.76953125" disk.nvme0n1.in‚52048.76953125 disk.rbd1.out‚ 56.8779296875# memory_percent‚82.6636552810669" proc.memory.rssMB‚ 2767.1640625 network.sent‚ 2.5418537e+07 disk.rbd3.in‚0 disk.rbd1.in‚ 0.53173828125 disk.rbd0.out‚0 disk.sda3.in‚ 108.921875 disk.sda3.out‚ 1349.44921875 disk.rbd3.out‚ 0.4951171875 disk.sda1.out‚0 network.recv‚ 5.426255e+06$ disk./.usageGB‚565.6168556213379 disk.sda1.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 cpu‚25.58556115084443) disk./.usagePercent‚37.95902176490038& disk.rbd0.in‚1.759218604441423e+13) disk.nvme0c0n1.out‚151019.73779296875) proc.memory.availableMB‚ 2840.38671875* proc.memory.percent‚16.889429092407227 disk.sda.in‚ 108.921875$ disk.dm-0.out‚151019.73779296875 disk.drbd2165.in‚0 proc.cpu.threads‚17Á]¡¼”Ñ4‚(:‰ çÞéÒôÔùö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚266.358' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.2716 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ú’=HÒ4‚(:½ çÞéÒ›·•ú network.sent‚ 2.5423715e+07 disk.sda.out‚ 1349.8046875 disk.sda3.in‚ 108.921875" disk.nvme0n1.in‚52049.19140625 proc.cpu.threads‚17$ disk./.usageGB‚565.6238250732422 disk.sda2.in‚0 disk.sda.in‚ 108.921875 disk.dm-1.in‚ 108.921875 disk.drbd2165.in‚0 cpu‚25.585408353719338 disk.rbd1.in‚ 0.53173828125& disk.rbd0.in‚1.759218604441423e+13 disk.dm-0.in‚52049.19140625 disk.sda3.out‚ 1349.8046875 disk.drbd2165.out‚0# disk.dm-0.out‚151035.0244140625 disk.rbd3.in‚0 disk.rbd3.out‚ 0.4951171875$ memory_percent‚82.66315460205078& proc.memory.availableMB‚ 2840.46875" proc.memory.rssMB‚ 2767.1640625 network.recv‚ 5.429244e+06 disk.rbd0.out‚0 disk.sda1.in‚0& disk.nvme0n1.out‚151035.0244140625* proc.memory.percent‚16.889429092407227) disk./.usagePercent‚37.95948949066543 disk.rbd1.out‚ 56.8857421875( disk.nvme0c0n1.out‚151035.0244140625 disk.sda2.out‚0 disk.dm-1.out‚ 1349.8046875 disk.sda1.out‚0$ disk.nvme0c0n1.in‚52049.19140625íÿÀ4Ó4/ _wandbruntime‚21854  _runtime‚21854°› ÊÔ4Ä % _timestamp‚1.7843116702140985e+09 _runtime‚21854.431310043 _step‚1084 & lr-AdamW/pg1‚0.0001859701868785109 & lr-AdamW/pg2‚0.0001859701868785109 trainer/global_step‚13199î¸ÑÏÕ4É _runtime‚21854.431310043 _step‚1084 & lr-AdamW/pg1‚0.0001859701868785109 & lr-AdamW/pg2‚0.0001859701868785109 trainer/global_step‚13199 % _timestamp‚1.7843116702140985e+09¼°ÎäÔ4Ö4/ _wandbruntime‚21854  _runtime‚21854•¦©\×4”P SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3804607555436032 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 trainer/global_step‚13199 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚261 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12392424846048 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1024551924633600 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12415009799680 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16007470836736 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 112547763 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚3790555071598592 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚37372433 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1040559798097920 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3804606183873024 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3792192545636352 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚37372689 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚1085 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 112547757 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚33857175 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 _runtime‚21854.435924375 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚71229869 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚3790555046170624 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚3792192545636352 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12393769446370 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 114292265 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1024552292724736 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3802948841044962 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 226840028 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 % _timestamp‚1.7843116702151856e+09}ºpÛ(Ø4™P QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 112547763 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚3790555071598592 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚37372433 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1040559798097920 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3804606183873024 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3792192545636352 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚37372689 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚1085 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 112547757 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚33857175 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 _runtime‚21854.435924375 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚71229869 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚3790555046170624 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚3792192545636352 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12393769446370 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 114292265 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1024552292724736 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3802948841044962 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 226840028 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 % _timestamp‚1.7843116702151856e+09 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3804607555436032 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 trainer/global_step‚13199 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚261 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12392424846048 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1024551924633600 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12415009799680 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16007470836736 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0½Ûí„ò•Ù4‚(:Š öÞéÒ ‹üö gpu.0.gpu‚100.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚248.621' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚49.7242 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0™.EÚ4‚(:º öÞéÒØÐÌù* proc.memory.percent‚16.889429092407227' disk.nvme0n1.out‚151043.35302734375 disk.sda2.in‚0 network.sent‚ 2.5428845e+07& disk.rbd0.in‚1.759218604441423e+13 disk.rbd0.out‚0 cpu‚25.585271187890918( disk./.usagePercent‚37.9600089299524$ disk.dm-0.out‚151043.35302734375 disk.sda1.in‚0" proc.memory.rssMB‚ 2767.1640625 proc.cpu.threads‚17 disk.dm-1.in‚ 108.921875 disk.sda3.in‚ 108.921875 disk.rbd3.out‚ 0.4951171875 disk.sda.in‚ 108.921875 network.recv‚ 5.43305e+06$ disk.nvme0c0n1.in‚52049.41015625 disk.rbd1.in‚ 0.53173828125 disk.rbd1.out‚ 56.9482421875" disk.nvme0n1.in‚52049.41015625$ disk./.usageGB‚565.6315650939941 disk.sda1.out‚0 disk.sda3.out‚ 1349.96875 disk.drbd2165.out‚0) disk.nvme0c0n1.out‚151043.35302734375 disk.dm-1.out‚ 1349.96875 disk.sda2.out‚0$ memory_percent‚82.66329765319824 disk.dm-0.in‚52049.41015625 disk.drbd2165.in‚0 disk.rbd3.in‚0 disk.sda.out‚ 1349.96875( proc.memory.availableMB‚ 2840.44531258Ï\4Û4/ _wandbruntime‚21855  _runtime‚218559N †$'Ü4žN D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 114290770 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3804886481208832 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚3792470520193024 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 226856726 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚71234897 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 114300757 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12394692912728 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3790832953376768 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16008697893888 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 trainer/global_step‚13199 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 226856726 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚33859718 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1040638074405888 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12414589445632 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚256 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 112555969 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1024629411041280 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 114300757 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 114290770 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 112555969 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚71235153 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3804885109638656 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3790832927948800 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 226846733 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12415934842368 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚37375430 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚251 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3804886481208832 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3792470520193024 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12415934842368 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 112555963 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1024629779132416 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16008663364608 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 % _timestamp‚1.7843116715994916e+09 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚37375179 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3803227646289496 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚33859723 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚3792470546366464 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 226846733 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 _runtime‚21855.816981053 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3792470546366464 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1040638477026304 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 112555963 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 _step‚1086 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12415934842368 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 112555963 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1024629779132416 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16008663364608 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 % _timestamp‚1.7843116715994916e+09 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚37375179 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3803227646289496 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚33859723 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 \GDeviceStatsMonitor.on_train_œti batch_end/active_bytes.large_pool.allocated‚3792470546366464 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 226846733 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 _runtime‚21855.816981053 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3792470546366464 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1040638477026304 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 112555963 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 _step‚1086 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 114290770 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3804886481208832 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚3792470520193024 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 226856726 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚71234897 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 114300757 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12394692912728 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3790832953376768 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16008697893888 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 trainer/global_step‚13199 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 226856726 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚33859718 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1040638074405888 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12414589445632 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚256 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 112555969 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1024629411041280 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 114300757 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 114290770 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 112555969 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚71235153 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3804885109638656 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3790832927948800 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 226846733 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12415934842368 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚37375430 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚251 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3804886481208832 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175¾eþ{Û4Þ4/ _wandbruntime‚21855  _runtime‚218555Œìžß4‹ 2loss/SRqqt2/detection_loss‚0.47439777636857033 0loss/FBt1/detection_loss‚0.37702212354821724 0loss/FBt2/detection_loss‚0.19740679985290396 %loss/total_loss‚4.922801419451892 0loss/FRt1/assignment_loss‚1.0066129712009788 /loss/FRt2/assignment_loss‚0.191172109459931 /loss/FRt1/detection_loss‚0.4034755543292532 % _timestamp‚1.7843116716861846e+09 3loss/SRqqt1/assignment_loss‚0.17878519145220315 trainer/global_step‚13199 3loss/SRqqt2/assignment_loss‚0.21700195133892447 4loss/FBt1/assignment_loss‚1.0539353640539165e-05 .loss/FRt2/detection_loss‚1.296692640817162 /loss/SRqqt1/detection_loss‚0.58021840392955 epoch‚32 _runtime‚21855.903095408 _step‚1087 3loss/FBt2/assignment_loss‚5.357800556598451e-06U24à4 0loss/FBt1/detection_loss‚0.37702212354821724 0loss/FBt2/detection_loss‚0.19740679985290396 %loss/total_loss‚4.922801419451892 0loss/FRt1/assignment_loss‚1.0066129712009788 /loss/FRt2/assignment_loss‚0.191172109459931 /loss/FRt1/detection_loss‚0.4034755543292532 % _timestamp‚1.7843116716861846e+09 3loss/SRqqt1/assignment_loss‚0.17878519145220315 trainer/global_step‚13199 3loss/SRqqt2/assignment_loss‚0.21700195133892447 4loss/FBt1/assignment_loss‚1.0539353640539165e-05 .loss/FRt2/detection_loss‚1.296692640817162 /loss/SRqqt1/detection_loss‚0.58021840392955 epoch‚32 _runtime‚21855.903095408 _step‚1087 3loss/FBt2/assignment_loss‚5.357800556598451e-06 2loss/SRqqt2/detection_loss‚0.47439777636857033¿7âËñ’á4‚(:‡ …ßéÒ鄉ù gpu.0.gpu‚1.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚36.0 gpu.0.powerWatts‚110.527' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.1054 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0“Ù©Kâ4‚(:À …ßéÒ‚•¦ú disk.rbd0.out‚0 disk.sda3.out‚ 1350.43359375 disk.sda.in‚ 108.921875) proc.memory.availableMB‚ 2808.01953125 network.recv‚ 5.437054e+06) disk./.usagePercent‚37.95928673269832 disk.drbd2165.in‚0 disk.rbd3.out‚ 0.4951171875' disk.nvme0c0n1.out‚151053.384765625* proc.memory.percent‚17.084932327270508 disk.dm-1.in‚ 108.921875" disk.nvme0n1.in‚52049.83203125$ memory_percent‚82.86120891571045" proc.memory.rssMB‚ 2799.1953125 disk.dm-0.in‚52049.83203125 disk.drbd2165.out‚0$ disk.nvme0c0n1.in‚52049.83203125 disk.rbd1.in‚ 0.53173828125 disk.rbd1.out‚ 56.9482421875 disk.dm-1.out‚ 1350.43359375 network.sent‚ 2.5481388e+07& disk.rbd0.in‚1.759218604441423e+13" disk.dm-0.out‚151053.384765625 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda2.out‚0 proc.cpu.threads‚17 disk.sda.out‚ 1350.43359375 disk.sda3.in‚ 108.921875 disk.rbd3.in‚0% disk.nvme0n1.out‚151053.384765625 cpu‚25.586798325412005$ disk./.usageGB‚565.6208038330078žòåæ4ã4/ _wandbruntime‚21882  _runtime‚21882y£Fç>ä4ú{ 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6024258136749268 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.025083769112825394 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.16012291610240936 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.16590802371501923 8"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.754478394985199 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6262297034263611 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4507390856742859 6 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.533516526222229 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6619876027107239 particle/accuracy_2_of_2‚1 particle/accuracy_2_of_3‚1 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.13507454097270966 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 particle/accuracy_1_of_1‚1 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9807115197181702 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.8125 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9971989393234253 +jet/accuracy_3_of_4‚0.05261075496673584 particle/accuracy_1_of_5‚1 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9961417317390442 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6446301937103271 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.033480554819107056 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 6validation_average_jet_accuracy‚0.3253196179866791 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5762330293655396 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.033997949212789536 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.26923078298568726 trainer/global_step‚13199 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.15418946743011475 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 particle/accuracy_1_of_3‚1 particle/accuracy_3_of_4‚1 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7142857313156128 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.14663942158222198 6 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.994621992111206 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7824074029922485 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.26923078298568726 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6080889105796814 jet/accuracy_5_of_5‚0 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9537036418914795 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 _step‚1088 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8585567474365234 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7734882831573486 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.11972472816705704 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5914109349250793 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.7987620830535889 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9935646057128906 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.6214250326156616 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9955089092254639 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 *jet/accuracy_1_of_2‚0.7211213707923889 'particle/accuracy‚0.831426739692688 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.2239583283662796 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.15123917162418365 *jet/accuracy_2_of_2‚0.1191931813955307 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 ,jet/accuracy_3_of_3‚0.047598518431186676 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.23914320766925812 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5762330293655396 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9915719032287598 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6262079477310181 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.009171264246106148 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.03989005833864212 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.8653846383094788 jet/accuracy_4_of_5‚0 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 - Purity/0FRt2SRqqt1FBt/FBt_purity‚0.921875 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 particle/accuracy_3_of_5‚1 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7850340604782104 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9901438355445862 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.023269450291991234 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.03989005833864212 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 epoch‚32 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2652522921562195 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.20209239423274994 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.49916666746139526 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 _runtime‚21882.492587176 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9849424958229065 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9970017671585083 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.16812703013420105 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 'particle/f_score‚0.6011072993278503 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8094277381896973 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.1266717165708542 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 ,jet/accuracy_4_of_4‚0.013899499550461769 particle/accuracy_3_of_3‚1 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6839219927787781 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7277453541755676 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.30000001192092896 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 particle/accuracy_4_of_5‚1 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.1745455116033554 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.7794406414031982 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.33740493655204773 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.1298273354768753 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.14610449969768524 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6377299427986145 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.7968124151229858 *jet/accuracy_1_of_1‚0.2837216556072235 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.23284152150154114 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6079440116882324 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9800085425376892 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_2_of_5‚1 $ _timestamp‚1.784311698274823e+09 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6561156511306763 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.6993327140808105 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7824074029922485 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 6 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.988606870174408 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6211401224136353 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6333602666854858 particle/accuracy_1_of_4‚1 particle/accuracy_2_of_4‚1 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.625 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.5611014366149902 -"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4375 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6605954170227051 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9989467859268188 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.1603991985321045 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9939056634902954 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6463041305541992 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6528264284133911 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9988662004470825 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.21296295523643494 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7214497327804565 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 *jet/accuracy_2_of_4‚0.6993950605392456 +particle/sensitivity‚0.5914840698242188 . Purity/2FRt0SRqqt1FBt/FRt_purity‚ 0.5859375 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9190019965171814 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.8935184478759766 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6418288350105286 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7116090059280396 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6057983040809631 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5613268613815308 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.1819215714931488 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.44919848442077637 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9950030446052551 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5456798672676086 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.16074025630950928 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7126620411872864 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.16080081462860107 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.1452375054359436 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.19304487109184265 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9782153367996216 - Purity/0FRt2SRqqt*FBt/FBt_purity‚0.921875 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.15914739668369293 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.40740740299224854 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 particle/accuracy_1_of_2‚1 /particle/accuracy_4_of_4‚0.9469330310821533 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9920619130134583 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.21296295523643494 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 jet/accuracy_1_of_5‚1 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.012289556674659252 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4166666567325592 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.8571428656578064 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 jet/accuracy_2_of_5‚1 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.03658873587846756 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7050496339797974 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9170029163360596 8"Purity/0FRt1SRqqt1FBt/event_purity‚0.311523973941803 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9937976002693176 8"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.733272135257721 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2049192488193512 jet/accuracy_3_of_5‚1 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7860733270645142 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.4188568592071533 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.4315289556980133 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.13630053400993347 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 +particle/specificity‚0.8970515131950378 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 particle/accuracy_5_of_5‚1 )loss/val_total_loss‚9.751955193935512 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.2229490578174591 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3319396674633026 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.5904577970504761 )jet/accuracy_1_of_4‚0.956942081451416 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.16074025630950928 9"Purity/1FRt1SRqqt*FBt/event_purity‚0.0299194548279047 9"Purity/2FRt*SRqqt0FBt/event_purity‚0.0049513247795403 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9537036418914795 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9871701002120972 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.18936856091022491 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6191508173942566 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.10499999672174454 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9985540509223938 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.7968124151229858 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5791295170783997 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.7843537926673889 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9743400812149048 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.12212320417165756 8"Purity/1FRt*SRqqt2FBt/event_purity‚0.706254780292511 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.17375971376895905 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 *jet/accuracy_1_of_3‚0.8945541381835938 +jet/accuracy_2_of_3‚0.47400742769241333 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9985540509223938 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6254305243492126 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.03682824969291687 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 8"Purity/1FRt0SRqqt1FBt/event_purity‚0.249641552567482 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.18946222960948944 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05k#´¿ìå4ÿ{ *jet/accuracy_2_of_4‚0.6993950605392456 +particle/sensitivity‚0.5914840698242188 . Purity/2FRt0SRqqt1FBt/FRt_purity‚ 0.5859375 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9190019965171814 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.8935184478759766 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6418288350105286 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7116090059280396 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6057983040809631 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5613268613815308 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.1819215714931488 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.44919848442077637 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9950030446052551 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5456798672676086 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.16074025630950928 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7126620411872864 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.16080081462860107 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.1452375054359436 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.19304487109184265 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9782153367996216 - Purity/0FRt2SRqqt*FBt/FBt_purity‚0.921875 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.15914739668369293 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.40740740299224854 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 particle/accuracy_1_of_2‚1 /particle/accuracy_4_of_4‚0.9469330310821533 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9920619130134583 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.21296295523643494 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 jet/accuracy_1_of_5‚1 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.012289556674659252 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4166666567325592 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.8571428656578064 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 jet/accuracy_2_of_5‚1 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.03658873587846756 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7050496339797974 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9170029163360596 8"Purity/0FRt1SRqqt1FBt/event_purity‚0.311523973941803 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9937976002693176 8"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.733272135257721 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.2049192488193512 jet/accuracy_3_of_5‚1 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7860733270645142 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.4188568592071533 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.4315289556980133 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.13630053400993347 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 +particle/specificity‚0.8970515131950378 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 particle/accuracy_5_of_5‚1 )loss/val_total_loss‚9.751955193935512 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.2229490578174591 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3319396674633026 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.5904577970504761 )jet/accuracy_1_of_4‚0.956942081451416 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.16074025630950928 9"Purity/1FRt1SRqqt*FBt/event_purity‚0.0299194548279047 9"Purity/2FRt*SRqqt0FBt/event_purity‚0.0049513247795403 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9537036418914795 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9871701002120972 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.18936856091022491 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6191508173942566 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.10499999672174454 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9985540509223938 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.7968124151229858 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5791295170783997 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.7843537926673889 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9743400812149048 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.12212320417165756 8"Purity/1FRt*SRqqt2FBt/event_purity‚0.706254780292511 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.17375971376895905 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 ,&Purity/2FRt]-$E)2SRqqt1FBt/event_proportion‚0 *jet/accuracy_1_of_3‚0.8945541381835938 +jet/accuracy_2_of_3‚0.47400742769241333 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9985540509223938 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6254305243492126 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.03682824969291687 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 8"Purity/1FRt0SRqqt1FBt/event_purity‚0.249641552567482 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.18946222960948944 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6024258136749268 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.025083769112825394 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.16012291610240936 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.16590802371501923 8"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.754478394985199 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6262297034263611 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4507390856742859 6 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.533516526222229 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6619876027107239 particle/accuracy_2_of_2‚1 particle/accuracy_2_of_3‚1 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.13507454097270966 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 particle/accuracy_1_of_1‚1 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9807115197181702 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.8125 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9971989393234253 +jet/accuracy_3_of_4‚0.05261075496673584 particle/accuracy_1_of_5‚1 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9961417317390442 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6446301937103271 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.033480554819107056 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 6validation_average_jet_accuracy‚0.3253196179866791 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5762330293655396 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.10000000149011612 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.033997949212789536 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.26923078298568726 trainer/global_step‚13199 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.15418946743011475 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 particle/accuracy_1_of_3‚1 particle/accuracy_3_of_4‚1 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7142857313156128 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.14663942158222198 6 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.994621992111206 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7824074029922485 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.26923078298568726 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6080889105796814 jet/accuracy_5_of_5‚0 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9537036418914795 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 _step‚1088 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8585567474365234 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7734882831573486 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.11972472816705704 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5914109349250793 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.7987620830535889 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9935646057128906 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.6214250326156616 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9955089092254639 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 *jet/accuracy_1_of_2‚0.7211213707923889 'particle/accuracy‚0.831426739692688 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.2239583283662796 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.15123917162418365 *jet/accuracy_2_of_2‚0.1191931813955307 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 ,jet/accuracy_3_of_3‚0.047598518431186676 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.23914320766925812 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5762330293655396 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9915719032287598 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6262079477310181 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.009171264246106148 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.03989005833864212 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.8653846383094788 jet/accuracy_4_of_5‚0 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 - Purity/0FRt2SRqqt1FBt/FBt_purity‚0.921875 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 particle/accuracy_3_of_5‚1 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7850340604782104 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9901438355445862 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.023269450291991234 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.03989005833864212 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 epoch‚32 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2652522921562195 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.20209239423274994 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.49916666746139526 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 _runtime‚21882.492587176 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9849424958229065 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9970017671585083 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.16812703013420105 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 'particle/f_score‚0.6011072993278503 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8094277381896973 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.1266717165708542 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 ,jet/accuracy_4_of_4‚0.013899499550461769 particle/accuracy_3_of_3‚1 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6839219927787781 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7277453541755676 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.30000001192092896 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 particle/accuracy_4_of_5‚1 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.1745455116033554 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.7794406414031982 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.33740493655204773 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.1298273354768753 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.14610449969768524 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6377299427986145 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.7968124151229858 *jet/accuracy_1_of_1‚0.2837216556072235 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.23284152150154114 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6079440116882324 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9800085425376892 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_2_of_5‚1 $ _timestamp‚1.784311698274823e+09 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6561156511306763 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.6993327140808105 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7824074029922485 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 6 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.988606870174408 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6211401224136353 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6333602666854858 particle/accuracy_1_of_4‚1 particle/accuracy_2_of_4‚1 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.625 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.5611014366149902 -"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4375 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6605954170227051 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9989467859268188 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.1603991985321045 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9939056634902954 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6463041305541992 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6528264284133911 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9988662004470825 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.21296295523643494 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7214497327804565 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0Àørܨžæ4‚(:“ ”ßéÒìå„÷ gpu.0.gpu‚99.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚283.318' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚56.66359999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0’~°‘Bë4‚(:· ²ßéÒ¿Þú) disk./.usagePercent‚37.96171112404757 disk.sda1.out‚0 network.sent‚ 2.5597061e+07 disk.rbd1.in‚ 0.53173828125 disk.sda1.in‚0 disk.drbd2165.out‚0$ disk.nvme0c0n1.out‚ 151089.484375 proc.cpu.threads‚17 disk.rbd0.out‚0" disk.nvme0n1.out‚ 151089.484375& disk.rbd0.in‚1.759218604441423e+13 disk.dm-0.out‚ 151089.484375 network.recv‚ 5.448525e+06 disk.rbd1.out‚ 57.2763671875 disk.dm-1.in‚ 108.921875 disk.drbd2165.in‚0 disk.rbd3.in‚0 disk.sda2.in‚0 disk.dm-1.out‚ 1351.18359375 disk.sda3.out‚ 1351.18359375 disk.rbd3.out‚ 0.4951171875$ disk.nvme0c0n1.in‚52184.93359375) proc.memory.availableMB‚ 2803.15234375* proc.memory.percent‚17.109036445617676 cpu‚25.587965123329226" disk.nvme0n1.in‚52184.93359375 disk.sda.in‚ 108.921875# memory_percent‚82.8909158706665# proc.memory.rssMB‚ 2803.14453125$ disk./.usageGB‚565.6569290161133 disk.sda2.out‚0 disk.sda.out‚ 1351.18359375 disk.dm-0.in‚52184.93359375 disk.sda3.in‚ 108.921875a í³”ì4‚(:‰ ÁßéÒäåø gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚333.229' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚66.6458 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0)||;í4‚(:° ÁßéÒ§¿Ÿú$ disk.nvme0c0n1.in‚52185.13671875& disk.rbd0.in‚1.759218604441423e+13 disk.rbd0.out‚0$ memory_percent‚82.88793563842773 network.recv‚ 5.451687e+06 disk.drbd2165.in‚0 disk.rbd3.in‚0' disk.nvme0c0n1.out‚151099.783203125 disk.sda.out‚1351.625 network.sent‚ 2.5601635e+07$ disk./.usageGB‚565.6614837646484 disk.sda3.in‚ 108.921875 disk.dm-0.in‚52185.13671875 disk.sda1.in‚0' proc.memory.availableMB‚ 2803.640625" disk.nvme0n1.in‚52185.13671875 disk.dm-1.out‚1351.625" disk.dm-0.out‚151099.783203125# proc.memory.rssMB‚ 2803.14453125 cpu‚25.58783665476626 proc.cpu.threads‚17 disk.rbd3.out‚ 0.4951171875 disk.sda1.out‚0 disk.drbd2165.out‚0 disk.sda2.in‚0 disk.rbd1.out‚ 57.2763671875 disk.dm-1.in‚ 108.921875 disk.sda3.out‚1351.625 disk.sda2.out‚0 disk.sda.in‚ 108.921875* disk./.usagePercent‚37.962016797043454% disk.nvme0n1.out‚151099.783203125 disk.rbd1.in‚ 0.53173828125* proc.memory.percent‚17.109036445617676u<µžî4‚(:“ ÐßéÒæ’íø gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚262.359' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚52.47179999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0}é1ï4‚(:¦ ÐßéÒˆÐéú network.sent‚ 2.5606708e+07 disk.sda2.out‚0 disk.dm-1.out‚ 1351.82421875 disk.dm-0.out‚ 151109.28125 disk.sda1.out‚0 disk.sda3.out‚ 1351.82421875* proc.memory.percent‚17.109036445617676 disk.rbd3.out‚ 0.4951171875 disk.nvme0n1.in‚ 52319.875& disk.rbd0.in‚1.759218604441423e+13 disk.sda2.in‚0 disk.rbd1.in‚ 0.53173828125 disk.sda.out‚ 1351.82421875 disk.dm-1.in‚ 108.921875 disk.rbd0.out‚0! disk.nvme0n1.out‚ 151109.28125 disk.rbd1.out‚ 57.3388671875 disk.sda.in‚ 108.921875 disk.dm-0.in‚ 52319.875 disk.nvme0c0n1.in‚ 52319.875 disk.sda1.in‚0 disk.drbd2165.in‚0$ memory_percent‚82.88800716400146$ disk./.usageGB‚565.6710777282715 disk.sda3.in‚ 108.921875 disk.drbd2165.out‚0) disk./.usagePercent‚37.96266065599207 disk.rbd3.in‚0 proc.cpu.threads‚17 network.recv‚ 5.455069e+06# disk.nvme0c0n1.out‚ 151109.28125) proc.memory.availableMB‚ 2803.62890625# proc.memory.rssMB‚ 2803.14453125 cpu‚25.587684234516264âœÈb4ð4/ _wandbruntime‚21959  _runtime‚21959îçøçËñ4Å ' lr-AdamW/pg1‚0.00018351719940669803 ' lr-AdamW/pg2‚0.00018351719940669803 trainer/global_step‚13249 $ _timestamp‚1.784311775629996e+09 _runtime‚21959.847901384 _step‚1089œï@¹Ðò4Ê _step‚1089 ' lr-AdamW/pg1‚0.00018351719940669803 ' lr-AdamW/pg2‚0.00018351719940669803 trainer/global_step‚13249 $ _timestamp‚1.784311775629996e+09 _runtime‚21959.847901384ÁŽúç4ó4/ _wandbruntime‚21959  _runtime‚21959ʨ]x(ô4”P QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 114776999 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 400530432 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚3820719535102625 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚3808275069440000 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 227867068 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1044549496930816 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 114767026 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3809920563576832 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚3809920537403392 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 trainer/global_step‚13249 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 227857089 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚33993248 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚37545596 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3822387731918848 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚32439296 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12445835690915 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12444491090593 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚37545821 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚33993243 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 % _timestamp‚1.7843117756307993e+09 _runtime‚21959.848141597 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 113090063 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚225 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3822386360355840 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12465822952448 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3820720905130915 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 _step‚1090 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚3822386360355840 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12467168342016 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚3808275044012032 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3809920537403392 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚33993243 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 % _timestamp‚1.7843117756307993e+09 _runtime‚21959.848141597 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 113090063 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚225 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3822386360355840 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12465822952448 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3820720905130915 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 _step‚1090 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0ÂÞ J”ö4‚(:‰ ßßéÒßÑö gpu.0.gpu‚54.0 gpu.0.memory‚16, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚318.173' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚63.6346 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ïCo:÷4‚(:¯ ßßéÒè¢ßø disk.rbd3.out‚ 0.4951171875# disk.nvme0n1.out‚151125.6484375 disk.sda2.out‚0 disk.rbd3.in‚0% disk.nvme0c0n1.out‚151125.6484375) disk./.usagePercent‚37.96295045652083 disk.drbd2165.out‚0 disk.nvme0n1.in‚ 52320.09375 disk.dm-1.out‚ 1352.18359375& disk.rbd0.in‚1.759218604441423e+13& proc.memory.availableMB‚ 2802.90625 disk.sda1.in‚0! disk.nvme0c0n1.in‚ 52320.09375 disk.rbd1.in‚ 0.53173828125# proc.memory.rssMB‚ 2803.14453125 network.sent‚ 2.5611921e+07 disk.sda3.in‚ 108.921875 disk.rbd0.out‚0 disk.dm-0.out‚151125.6484375* proc.memory.percent‚17.109036445617676 cpu‚25.587561228423265$ memory_percent‚82.89241790771484 disk.dm-0.in‚ 52320.09375 network.recv‚ 5.458465e+06$ disk./.usageGB‚565.6753959655762 disk.drbd2165.in‚0 disk.sda.in‚ 108.921875 proc.cpu.threads‚17 disk.sda1.out‚0 disk.dm-1.in‚ 108.921875 disk.sda3.out‚ 1352.18359375 disk.sda2.in‚0 disk.rbd1.out‚ 57.3388671875 disk.sda.out‚ 1352.18359375ø`ö¼4ø4/ _wandbruntime‚21961  _runtime‚21961ó‹Ëä$'ù4žN \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚3810198564306944 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 _runtime‚21961.231937522 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 227873773 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚37548621 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3820999710375449 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 % _timestamp‚1.7843117770144923e+09 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3822666657691648 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12446759157273 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚32432128 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3822666657691648 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3822665286121472 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚71544186 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 400523264 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚3810198538133504 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 227883766 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚226 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12468093384704 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 227883766 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3808552925790208 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 _step‚1091 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚33995786 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚221 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 114785491 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 113098269 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚37548400 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 227873773 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16076394256896 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1028551843672064 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16076361824768 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1044628606020096 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12445414556839 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 227873773 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚37548621 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3820999710375449 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 % _timestamp‚1.7843117770144923e+09 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3822666657691648 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12446759157273 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚32432128 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3822666657691648 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3822665286121472 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚71544186 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 400523264 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚3810198538133504 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 227883766 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚226 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12468093384704 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 227883766 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3808552925790208 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 _step‚1091 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚33995786 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚221 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 114785491 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 113098269 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚37548400 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 227873773 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16076394256896 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1028551843672064 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16076361824768 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1044628606020096 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12445414556839 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 _runtime‚21961.231937522à rÄ94û4/ _wandbruntime‚21961  _runtime‚21961.§×äü4 epoch‚33 0loss/FRt1/assignment_loss‚1.0074975834355016 1loss/FRt2/assignment_loss‚0.11440418229993347 0loss/FRt1/detection_loss‚0.44457403289274156 /loss/FRt2/detection_loss‚1.2915157076950028 0loss/FBt1/detection_loss‚0.33063123583873144 /loss/FBt2/detection_loss‚0.2549952854221361 % _timestamp‚1.7843117771011798e+09 3loss/SRqqt1/assignment_loss‚0.09793865397482303 1loss/SRqqt2/assignment_loss‚0.177913403246043 1loss/SRqqt1/detection_loss‚0.6023235426016919 %loss/total_loss‚4.838810629020393 3loss/FBt1/assignment_loss‚0.0004097699789034883 1loss/SRqqt2/detection_loss‚0.5165787351390358 _runtime‚21961.318102022 4loss/FBt2/assignment_loss‚2.8496495849263856e-05 trainer/global_step‚13249 _step‚1092ò¬ô„ý4’ 1loss/SRqqt1/detection_loss‚0.6023235426016919 %loss/total_loss‚4.838810629020393 3loss/FBt1/assignment_loss‚0.0004097699789034883 1loss/SRqqt2/detection_loss‚0.5165787351390358 _runtime‚21961.318102022 4loss/FBt2/assignment_loss‚2.8496495849263856e-05 trainer/global_step‚13249 _step‚1092 epoch‚33 0loss/FRt1/assignment_loss‚1.0074975834355016 1loss/FRt2/assignment_loss‚0.11440418229993347 0loss/FRt1/detection_loss‚0.44457403289274156 /loss/FRt2/detection_loss‚1.2915157076950028 0loss/FBt1/detection_loss‚0.33063123583873144 /loss/FBt2/detection_loss‚0.2549952854221361 % _timestamp‚1.7843117771011798e+09 3loss/SRqqt1/assignment_loss‚0.09793865397482303 1loss/SRqqt2/assignment_loss‚0.177913403246043Äu£xŠŸþ4‚(:” îßéÒÚ¥™÷ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚259.109' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚51.821799999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0m†¨Dÿ4‚(:¹ îßéÒáŽúø disk.dm-0.in‚ 52320.296875 disk.sda1.out‚0# proc.memory.rssMB‚ 2803.14453125 network.sent‚ 2.5665011e+07$ disk./.usageGB‚565.6822357177734 disk.sda2.in‚0* proc.memory.percent‚17.109036445617676 disk.rbd1.in‚ 0.53173828125 disk.rbd1.out‚ 57.4013671875 disk.dm-1.in‚ 108.921875& disk.rbd0.in‚1.759218604441423e+13 disk.nvme0n1.in‚ 52320.296875 disk.sda2.out‚0 cpu‚25.587398727592504 disk.sda3.in‚ 108.921875' disk.nvme0n1.out‚151133.84814453125 disk.sda.out‚ 1352.3671875 disk.drbd2165.in‚0" disk.nvme0c0n1.in‚ 52320.296875 network.recv‚ 5.463107e+06 disk.sda.in‚ 108.921875 disk.dm-1.out‚ 1352.3671875 disk.sda1.in‚0) disk.nvme0c0n1.out‚151133.84814453125$ memory_percent‚82.89108276367188$ disk.dm-0.out‚151133.84814453125 disk.sda3.out‚ 1352.3671875 disk.rbd3.in‚0 disk.rbd3.out‚ 0.4951171875$ proc.memory.availableMB‚2803.125) disk./.usagePercent‚37.96340947802972 disk.rbd0.out‚0 disk.drbd2165.out‚0 proc.cpu.threads‚17²î¬3€5‚(:’ ýßéÒÀ¤¥÷ gpu.0.gpu‚44.0 gpu.0.memory‚14, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚341.45' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚68.28999999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚09D(5‚(: ýßéÒ¨ù disk.rbd1.out‚ 57.4013671875( proc.memory.availableMB‚ 2803.1640625 proc.cpu.threads‚17 network.recv‚ 5.467254e+06 disk.dm-1.in‚ 108.921875 disk.dm-1.out‚ 1352.6875 disk.rbd3.out‚ 0.4951171875# proc.memory.rssMB‚ 2803.14453125* disk./.usagePercent‚37.963860051289984 disk.dm-0.out‚ 151148.5625 disk.sda.in‚ 108.921875 disk.sda3.in‚ 108.921875 disk.drbd2165.out‚0 disk.sda3.out‚ 1352.6875 disk.sda2.in‚0 disk.rbd1.in‚ 0.53173828125 network.sent‚ 2.5717601e+07& disk.rbd0.in‚1.759218604441423e+13 disk.rbd0.out‚0 disk.drbd2165.in‚0 disk.nvme0n1.out‚ 151148.5625 disk.sda2.out‚0$ memory_percent‚82.89084434509277$ disk./.usageGB‚565.6889495849609 disk.dm-0.in‚ 52320.71875 disk.sda1.in‚0 disk.sda1.out‚0* proc.memory.percent‚17.109036445617676 cpu‚25.587281540642174 disk.sda.out‚ 1352.6875 disk.rbd3.in‚0! disk.nvme0c0n1.in‚ 52320.71875 disk.nvme0n1.in‚ 52320.71875" disk.nvme0c0n1.out‚ 151148.5625$1óN‚5‚(:” ŒàéÒ°›ûö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.meMïò¢QmoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚300.697' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚60.139399999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¸_ÜmMƒ5‚(: ŒàéÒŠÔºù disk.drbd2165.out‚0 network.sent‚ 2.5722724e+07 network.recv‚ 5.470723e+06$ disk.nvme0c0n1.in‚52422.97265625 disk.sda.out‚ 1352.7734375 disk.dm-1.out‚ 1352.7734375* proc.memory.percent‚17.109036445617676 cpu‚25.587130935672562 disk.rbd1.out‚ 57.4794921875& disk.rbd0.in‚1.759218604441423e+13 disk.sda1.out‚0 disk.sda3.in‚ 108.921875$ memory_percent‚82.88958072662354 disk.drbd2165.in‚0) proc.memory.availableMB‚ 2803.37109375" disk.nvme0n1.in‚52422.97265625 disk.dm-0.in‚52422.97265625 disk.sda1.in‚0 proc.cpu.threads‚17* disk./.usagePercent‚37.964162652195455 disk.rbd3.out‚ 0.4951171875& disk.nvme0n1.out‚151156.2607421875 disk.sda.in‚ 108.921875 disk.dm-1.in‚ 108.921875 disk.sda3.out‚ 1352.7734375$ disk./.usageGB‚565.6934585571289# disk.dm-0.out‚151156.2607421875 disk.sda2.out‚0 disk.rbd1.in‚ 0.53173828125 disk.rbd3.in‚0( disk.nvme0c0n1.out‚151156.2607421875 disk.sda2.in‚0 disk.rbd0.out‚0# proc.memory.rssMB‚ 2803.14453125Ü  •„5‚(:Š ›àéÒôǤ÷ gpu.0.gpu‚100.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚334.052' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚66.8104 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ºP7;…5‚(:° ›àéÒù—­ú' disk.nvme0c0n1.out‚151168.744140625 disk.sda2.in‚0 disk.rbd1.out‚ 57.4794921875& disk.rbd0.in‚1.759218604441423e+13) disk./.usagePercent‚37.96482417566393 disk.dm-0.in‚52423.39453125 disk.sda1.out‚0 disk.sda3.in‚ 108.921875 disk.sda3.out‚1353.125* proc.memory.percent‚17.109036445617676 proc.cpu.threads‚17 network.sent‚ 2.5727824e+07 network.recv‚ 5.473912e+06" disk.dm-0.out‚151168.744140625% disk.nvme0n1.out‚151168.744140625 disk.sda.out‚1353.125 disk.rbd0.out‚0 cpu‚25.587001450767666 disk.drbd2165.out‚0 disk.rbd1.in‚ 0.53173828125 disk.dm-1.in‚ 108.921875$ disk./.usageGB‚565.7033157348633 disk.rbd3.out‚ 0.4951171875$ disk.nvme0c0n1.in‚52423.39453125" disk.nvme0n1.in‚52423.39453125 disk.sda2.out‚0 disk.sda.in‚ 108.921875' proc.memory.availableMB‚ 2803.328125 disk.sda1.in‚0 disk.drbd2165.in‚0 disk.rbd3.in‚0 disk.dm-1.out‚1353.125$ memory_percent‚82.88984298706055# proc.memory.rssMB‚ 2803.14453125Ê;á锆5‚(:‰ ªàéÒµÖš÷ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚298.284' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚59.6568 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0"Öêß@‡5‚(:µ ªàéҜޣù$ disk.nvme0c0n1.in‚52423.59765625 disk.rbd1.out‚ 57.5419921875 proc.cpu.threads‚17 disk.dm-0.in‚52423.59765625" disk.nvme0n1.in‚52423.59765625" disk.nvme0n1.out‚ 151182.234375 disk.sda2.out‚0 disk.sda.in‚ 108.921875 disk.rbd3.in‚0$ disk.nvme0c0n1.out‚ 151182.234375 disk.sda.out‚ 1353.3515625) proc.memory.availableMB‚ 2803.02734375 cpu‚25.586854230617647 network.sent‚ 2.5736779e+07& disk.rbd0.in‚1.759218604441423e+13 disk.rbd0.out‚0 disk.rbd3.out‚ 0.4951171875 network.recv‚ 5.484791e+06) disk./.usagePercent‚37.96510783201187 disk.sda1.out‚0 disk.drbd2165.in‚0* proc.memory.percent‚17.109036445617676$ disk./.usageGB‚565.7075424194336$ memory_percent‚82.89167881011963# proc.memory.rssMB‚ 2803.14453125 disk.dm-1.out‚ 1353.3515625 disk.dm-0.out‚ 151182.234375 disk.sda3.in‚ 108.921875 disk.sda2.in‚0 disk.dm-1.in‚ 108.921875 disk.drbd2165.out‚0 disk.rbd1.in‚ 0.53173828125 disk.sda1.in‚0 disk.sda3.out‚ 1353.3515625šÙ±z4ˆ5/ _wandbruntime‚22038  _runtime‚22038kmˉ5Å ' lr-AdamW/pg2‚0.00018107464303471717 trainer/global_step‚13299 $ _timestamp‚1.784311854210074e+09 _runtime‚22038.427265141 _step‚1093 ' lr-AdamW/pg1‚0.00018107464303471717Œk/Њ5Ê $ _timestamp‚1.784311854210074e+09 _runtime‚22038.427265141 _step‚1093 ' lr-AdamW/pg1‚0.00018107464303471717 ' lr-AdamW/pg2‚0.00018107464303471717 trainer/global_step‚13299Ån˃ÿ4‹5/ _wandbruntime‚22038  _runtime‚22038¿:û(Œ5“P JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3836334084355072 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3836332712792064 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚280 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1032349045739520 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚3823820573908992 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚3822169158348800 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚3834659861124533 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12512138883072 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚34120393 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12513484272640 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚71804504 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3834661231152823 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 $ _timestamp‚1.784311854211171e+09 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 113500363 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1048484051451392 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12490728203701 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚3822169132920832 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12492072804023 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 113500369 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚285 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404724736 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16135005711872 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 113500363 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3823820600082432 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12513484272640 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16134969078272 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 228703120 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚34120398 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0n¼(5˜P bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1032348677648384 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3836334084355072 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3836332712792064 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚280 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1032349045739520 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚3823820573908992 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚3822169158348800 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚3834659861124533 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12512138883072 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚34120393 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12513484272640 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚71804504 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3834661231152823 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 $ _timestamp‚1.784311854211171e+09 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 113500363 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1048484051451392 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12490728203701 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚3822169132920832 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12492072804023 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 113500369 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚285 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404724736 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16135005711872 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 113500363 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3823820600082432 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12513484272640 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16134969078272 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 228703120 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚34120398 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3836334084355072 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚34122941 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16136188867072 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3822447014699008 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1048562721014272 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 113508575 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 228709825 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12514409315328 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 _runtime‚22039.819290124 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 228719818 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3834940036397357 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1032426164056064 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚37687152 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1032426532147200 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 _step‚1095 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚279 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚37686878 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3836613010127872 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚274 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 115201256 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 228719818 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 113508569 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36626432 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12514409315328 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12513063918592 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 115211243 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3824098574639104 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12492996270381 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚3824098600812544 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 113508575 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3836613010127872 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 TEDeviceStatsMonitor.on_train_batch_end/req‰7{5 uested_bytes.small_pool.peak‚ 1679318926 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚3834938666368955 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3822447040126976 $ _timestamp‚1.784311855601896e+09 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚71810093 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚71809814 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404717568 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 115201256 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 trainer/global_step‚13299 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1048562316296704 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16136152240640 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 228719818 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 113508569 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36626432 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12514409315328 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12513063918592 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 115211243 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3824098574639104 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12492996270381 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚3824098600812544 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 113508575 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3836613010127872 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚3834938666368955 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3822447040126976 $ _timestamp‚1.784311855601896e+09 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚71810093 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚71809814 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404717568 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 115201256 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 trainer/global_step‚13299 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1048562316296704 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16136152240640 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚34122941 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16136188867072 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3822447014699008 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1048562721014272 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 113508575 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 228709825 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12514409315328 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 _runtime‚22039.819290124 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 228719818 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3834940036397357 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1032426164056064 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚37687152 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1032426532147200 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 _step‚1095 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚279 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚37686878 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3836613010127872 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚274Ç'äª4‘5/ _wandbruntime‚22039  _runtime‚22039“ Û’5‹ 0loss/FRt1/assignment_loss‚1.0182442826255513 3loss/FBt2/assignment_loss‚4.742015501924157e-05 /loss/FRt1/detection_loss‚0.4898503136464704 /loss/FBt1/detection_loss‚0.3125134043899986 _step‚1096 3loss/SRqqt1/assignment_loss‚0.13507657078556481 3loss/FBt1/assignment_loss‚0.0006222768595556261 /loss/SRqqt1/detection_loss‚0.49723151846483 /loss/FBt2/detection_loss‚0.3186022229219137 epoch‚33 trainer/global_step‚13299 3loss/SRqqt2/assignment_loss‚0.16231008655612283 %loss/total_loss‚4.712151528973255 1loss/FRt2/assignment_loss‚0.12781912538240542 /loss/FRt2/detection_loss‚1.1803742410540405 2loss/SRqqt2/detection_loss‚0.46946006613178193 % _timestamp‚1.7843118556872137e+09 _runtime‚22039.904155173¾‹IÏ“5 3loss/SRqqt2/assignment_loss‚0.16231008655612283 %loss/total_loss‚4.712151528973255 1loss/FRt2/assignment_loss‚0.12781912538240542 /loss/FRt2/detection_loss‚1.1803742410540405 2loss/SRqqt2/detection_loss‚0.46946006613178193 % _timestamp‚1.7843118556872137e+09 _runtime‚22039.904155173 0loss/FRt1/assignment_loss‚1.0182442826255513 3loss/FBt2/assignment_loss‚4.742015501924157e-05 /loss/FRt1/detection_loss‚0.4898503136464704 /loss/FBt1/detection_loss‚0.3125134043899986 _step‚1096 3loss/SRqqt1/assignment_loss‚0.13507657078556481 3loss/FBt1/assignment_loss‚0.0006222768595556261 /loss/SRqqt1/detection_loss‚0.49723151846483 /loss/FBt2/detection_loss‚0.3186022229219137 epoch‚33 trainer/global_step‚13299ȯpå ””5‚(:‰ ¹àéÒÇä­÷ gpu.0.gpu‚99.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚331.878' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚66.3756 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚01µ÷ÁB•5‚(:· ¹àéÒñäú disk.dm-1.out‚ 1353.5625 disk.sda3.in‚ 108.921875 cpu‚25.586712260079924 network.recv‚ 5.488024e+06# disk./.usageGB‚565.714412689209$ disk.nvme0c0n1.in‚52423.81640625' disk.nvme0n1.out‚151193.23095703125 disk.dm-1.in‚ 108.921875 disk.rbd3.in‚0 network.sent‚ 2.574189e+07& disk.rbd0.in‚1.759218604441423e+13 disk.rbd0.out‚0$ disk.dm-0.out‚151193.23095703125* proc.memory.percent‚17.109036445617676) disk.nvme0c0n1.out‚151193.23095703125 disk.rbd1.in‚ 0.53173828125 disk.rbd1.out‚ 57.5419921875 disk.sda1.out‚0 disk.sda3.out‚ 1353.5625 disk.drbd2165.in‚0 disk.rbd3.out‚ 0.4951171875$ memory_percent‚82.89022445678711 disk.sda2.in‚0 disk.dm-0.in‚52423.81640625 disk.drbd2165.out‚0 proc.cpu.threads‚17 disk.sda1.in‚0' proc.memory.availableMB‚ 2803.265625# proc.memory.rssMB‚ 2803.14453125) disk./.usagePercent‚37.96556890158104" disk.nvme0n1.in‚52423.81640625 disk.sda.in‚ 108.921875 disk.sda.out‚ 1353.5625 disk.sda2.out‚0(”š”–5‚(:‰ ÈàéҲυù gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚265.534' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.1068 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ë€Ð‡O—5‚(:Ä ÈàéÒžš¾û network.recv‚ 5.492518e+06 disk.rbd0.out‚0& disk.rbd0.in‚1.759218604441423e+13 disk.sda3.out‚ 1353.71484375 disk.rbd3.in‚0) disk.nvme0c0n1.out‚151205.98486328125' disk.nvme0n1.out‚151205.98486328125( proc.memory.availableMB‚ 2803.2421875# proc.memory.rssMB‚ 2803.14453125* proc.memory.percent‚17.109036445617676 disk.dm-1.in‚ 108.921875 disk.sda1.out‚0 disk.sda2.out‚0$ memory_percent‚82.89036750793457 disk.dm-1.out‚ 1353.71484375 disk.sda1.in‚0 disk.sda3.in‚ 108.921875 disk.drbd2165.in‚0# disk.nvme0c0n1.in‚ 52527.2578125! disk.nvme0n1.in‚ 52527.2578125 proc.cpu.threads‚17 disk.drbd2165.out‚0 disk.sda.out‚ 1353.71484375$ disk.dm-0.out‚151205.98486328125 disk.sda2.in‚0 disk.rbd1.in‚ 0.53173828125* disk./.usagePercent‚37.966000530283765$ disk./.usageGB‚565.7208442687988 disk.dm-0.in‚ 52527.2578125 network.sent‚ 2.5805185e+07 disk.rbd3.out‚ 0.4951171875 disk.rbd1.out‚ 57.6201171875 disk.sda.in‚ 108.921875 cpu‚25.586561142147886ÑÜÃK”˜5‚(:‰ ×àéÒôÉ­ù gpu.0.gpu‚90.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚273.606' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.7212 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ê(Q1™5‚(:¦ ×àéÒ»“Àû disk.drbd2165.in‚0) disk.nvme0c0n1.out‚151217.27880859375' disk.nvme0n1.out‚151217.27880859375 disk.sda1.in‚0 disk.drbd2165.out‚0 cpu‚25.58643447794829$ memory_percent‚82.88886547088623 disk.rbd1.in‚ 0.53173828125 network.recv‚ 5.4948e+06# proc.memory.rssMB‚ 2803.14453125 proc.cpu.threads‚17 disk.dm-0.in‚ 52527.4609375 disk.sda1.out‚0 disk.rbd1.out‚ 57.6201171875* proc.memory.percent‚17.109036445617676 disk.sda.in‚ 108.921875 disk.sda.out‚1354 disk.dm-1.out‚1354# disk.nvme0c0n1.in‚ 52527.4609375) proc.memory.availableMB‚ 2803.48828125 disk.rbd3.in‚0! disk.nvme0n1.in‚ 52527.4609375 disk.sda2.out‚0 network.sent‚ 2.5808855e+07$ disk./.usageGB‚565.7253074645996& disk.rbd0.in‚1.759218604441423e+13 disk.sda3.in‚ 108.921875 disk.sda3.out‚1354 disk.rbd3.out‚ 0.4951171875 disk.sda2.in‚0) disk./.usagePercent‚37.96630005909883 disk.dm-1.in‚ 108.921875 disk.rbd0.out‚0$ disk.dm-0.out‚151217.27880859375ÖõÑÚŸš5‚(:” æàéÒ–ÌŸù gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚276.599' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚55.319799999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0µßdM›5‚(: æàéÒåéŸû* disk./.usagePercent‚37.966856619478264 disk.sda1.out‚0) disk.nvme0c0n1.out‚151224.99365234375 disk.rbd0.out‚0 disk.dm-0.in‚ 52527.8828125$ disk.dm-0.out‚151224.99365234375& disk.rbd0.in‚1.759218604441423e+13# memory_percent‚82.8885555267334( proc.memory.availableMB‚ 2803.5390625 disk.drbd2165.in‚0' disk.nvme0n1.out‚151224.99365234375 disk.rbd1.out‚ 57.6826171875 disk.sda1.in‚0! disk.nvme0n1.in‚ 52527.8828125 disk.sda.in‚ 108.921875 cpu‚25.58629735394025$ disk./.usageGB‚565.7336006164551 disk.rbd3.in‚0 disk.rbd1.in‚ 0.53173828125 disk.sda.out‚ 1354.18359375* proc.memory.percent‚17.109036445617676 network.recv‚ 5.498187e+06 disk.sda3.in‚ 108.921875 disk.sda3.out‚ 1354.18359375 disk.dm-1.in‚ 108.921875 network.sent‚ 2.5814042e+07# disk.nvme0c0n1.in‚ 52527.8828125 disk.sda2.out‚0 disk.dm-1.out‚ 1354.18359375 disk.drbd2165.out‚0 disk.rbd3.out‚ 0.4951171875 disk.sda2.in‚0# proc.memory.rssMB‚ 2803.14453125 proc.cpu.threads‚17´*J”œ5‚(:‰ õàéÒžºÈù gpu.0.gpu‚99.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚171.512' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚34.3024 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0}Þß9N5‚(:à õàéÒÔèÝû disk.sda2.in‚0 disk.sda.out‚ 1354.87109375 cpu‚25.58616993075888 proc.cpu.threads‚17 disk.drbd2165.out‚0 disk.sda2.out‚0 disk.rbd1.in‚ 0.53173828125# proc.memory.rssMB‚ 2803.14453125* proc.memory.percent‚17.109036445617676 network.sent‚ 2.5819143e+07) disk.nvme0c0n1.out‚151239.10791015625 disk.rbd0.out‚0$ disk./.usageGB‚565.7381401062012 disk.drbd2165.in‚0 disk.dm-1.out‚ 1354.87109375 disk.sda3.in‚ 108.921875# disk.nvme0c0n1.in‚ 52528.1015625! disk.nvme0n1.in‚ 52528.1015625 disk.rbd1.out‚ 57.6826171875 disk.dm-0.in‚ 52528.1015625 disk.sda1.out‚0$ memory_percent‚82.89005756378174) proc.memory.availableMB‚ 2803.29296875 network.recv‚ 5.501491e+06 disk.sda3.out‚ 1354.87109375 disk.sda.in‚ 108.921875& disk.rbd0.in‚1.759218604441423e+13 disk.sda1.in‚0) disk./.usagePercent‚37.96716126844401 disk.rbd3.in‚0 disk.rbd3.out‚ 0.4951171875' disk.nvme0n1.out‚151239.10791015625 disk.dm-1.in‚ 108.921875$ disk.dm-0.out‚151239.10791015625–B-ä4ž5/ _wandbruntime‚22117  _runtime‚22117¼nwÌŸ5Æ % _timestamp‚1.7843119330485811e+09 _runtime‚22117.265880654 _step‚1097 ' lr-AdamW/pg1‚0.00017864267464340003 ' lr-AdamW/pg2‚0.00017864267464340003 trainer/global_step‚13349d@mÑ 5Ë ' lr-AdamW/pg1‚0.00017864267464340003 ' lr-AdamW/pg2‚0.00017864267464340003 trainer/global_step‚13349 % _timestamp‚1.7843119330485811e+09 _runtime‚22117.265880654 _step‚1097É|X&4¡5/ _wandbruntime‚22117  _runtime‚22117úð4Dk'¢5“P P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚3848600187146441 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚37820053 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 trainer/global_step‚13349 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12558454813696 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚3837720636588032 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚3837720610414592 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1052416762225664 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1036222998032384 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚38730752 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16193802924032 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚72067882 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 115618530 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1052417169047552 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16193764193280 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚37820334 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12536965316809 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 115618530 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚3836063221829632 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 113910663 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 229529193 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 115628503 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚281 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 113910669 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 115628503 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 113910663 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3850280436791296 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12559800203264 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12538309917131 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 113910669 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12559800203264 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12558454813696 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 229539172 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚286 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3848601557174731 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _runtime‚22117.269937853 _step‚1098 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚3850279065228288 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚72067596 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 S>DeviceStats| L¾®Monitor.on_train_batch_start/active_bytes.all.freed‚3850279065228288 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1036223366123520(ÆÏ»(£5˜P XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12558454813696 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚3837720636588032 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚3837720610414592 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1052416762225664 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1036222998032384 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚38730752 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16193802924032 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚72067882 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 115618530 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1052417169047552 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16193764193280 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚37820334 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12536965316809 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 115618530 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚3836063221829632 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 113910663 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 229529193 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 115628503 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚281 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 113910669 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 115628503 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 113910663 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3850280436791296 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12559800203264 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12538309917131 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 113910669 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12559800203264 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12558454813696 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 229539172 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚286 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3848601557174731 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _runtime‚22117.269937853 _step‚1098 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚3850279065228288 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚72067596 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3850279065228288 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1036223366123520 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚3848600187146441 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚37820053 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 trainer/global_step‚13349Êž°Õ®4¤5/ _wandbruntime‚22118  _runtime‚22118@KPá$'¥5žN T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3850557990993920 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12559379849216 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3837998611144704 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12559379849216 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3848880362419265 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 115627008 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16194980318208 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 _step‚1099 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 113918869 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚3837998637318144 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚271 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 229545877 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1052495426034688 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚3837998611144704 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3837998637318144 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚3848878992390863 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3850559362564096 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚72072770 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚38723584 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12560725245952 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 115636995 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚37822684 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 % _timestamp‚1.7843119344344444e+09 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 115636995 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1052495832849408 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 229555870 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 229555870 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚276 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3836341129035776 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 113918875 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚3837998611144704 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3837998637318144 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚3848878992390863 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3850559362564096 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚72072770 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚38723584 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12560725245952 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 115636995 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚37822684 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 % _timestamp‚1.7843119344344444e+09 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 115636995 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1052495832849408 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 229555870 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 229555870 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚276 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3836341129035776 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 113918875 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 115627008 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16194980318208 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 _step‚1099 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 113918869 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚3837998637318144 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚271 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008Ëj¢ç+4§5/ _wandbruntime‚22118  _runtime‚22118Od%…¨5‹ $loss/total_loss‚4.55958823870567 1loss/FRt2/assignment_loss‚0.15918041060242472 2loss/SRqqt2/assignment_loss‚0.1566956280735396 /loss/FRt2/detection_loss‚1.2214778088836318 1loss/SRqqt1/detection_loss‚0.6334741381313974 1loss/SRqqt2/detection_loss‚0.4537449433353641 3loss/SRqqt1/assignment_loss‚0.06996556836669765 3loss/FBt2/assignment_loss‚7.750558987107414e-06 0loss/FRt1/detection_loss‚0.42698577087581113 trainer/global_step‚13349 $ _timestamp‚1.784311934520688e+09 _runtime‚22118.737618521 4loss/FBt1/assignment_loss‚1.4845448145619922e-05 /loss/FBt1/detection_loss‚0.2156895677414598 /loss/FBt2/detection_loss‚0.1901923291328913 _step‚1100 0loss/FRt1/assignment_loss‚1.0321594775553202 epoch‚33¦ˆ—D©5 2loss/SRqqt2/assignment_loss‚0.1566956280735396 /loss/FRt2/detection_loss‚1.2214778088836318 1loss/SRqqt1/detection_loss‚0.6334741381313974 1loss/SRqqt2/detection_loss‚0.4537449433353641 3loss/SRqqt1/assignment_loss‚0.06996556836669765 3loss/FBt2/assignment_loss‚7.750558987107414e-06 0loss/FRt1/detection_loss‚0.42698577087581113 trainer/global_step‚13349 $ _timestamp‚1.784311934520688e+09 _runtime‚22118.737618521 4loss/FBt1/assignment_loss‚1.4845448145619922e-05 /loss/FBt1/detection_loss‚0.2156895677414598 /loss/FBt2/detection_loss‚0.1901923291328913 _step‚1100 0loss/FRt1/assignment_loss‚1.0321594775553202 epoch‚33 $loss/total_loss‚4.55958823870567 1loss/FRt2/assignment_loss‚0.15918041060242472Ì ©£“ª5‚(:ˆ „áéÒýà¬÷ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚287.615' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚57.523 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÌÔîý«5‚(:± „áéÒܺÃù network.recv‚ 5.50521e+06 disk.rbd1.in‚ 0.53173828125& disk.rbd0.in‚1.759218604441423e+13±Ó½ disk.sda3.in‚ 108.921875 disk.drbd2165.out‚0 disk.rbd1.out‚ 57.7451171875 disk.dm-1.in‚ 108.921875 disk.sda1.out‚0 disk.sda2.out‚0 disk.dm-0.in‚52634.98046875) disk./.usagePercent‚37.96769709221318 network.sent‚ 2.5824278e+07 disk.drbd2165.in‚0 disk.rbd3.out‚ 0.4951171875$ disk.nvme0c0n1.out‚ 151246.078125 disk.sda2.in‚0 disk.sda1.in‚0$ disk.nvme0c0n1.in‚52634.98046875* proc.memory.percent‚17.109036445617676 cpu‚25.58601531854604 disk.rbd0.out‚0 disk.rbd3.in‚0" disk.nvme0n1.in‚52634.98046875$ proc.memory.availableMB‚2804.375 proc.cpu.threads‚17$ disk./.usageGB‚565.7461242675781 disk.sda.in‚ 108.921875 disk.sda.out‚ 1354.96484375 disk.dm-1.out‚ 1354.96484375 disk.sda3.out‚ 1354.96484375" disk.nvme0n1.out‚ 151246.078125$ memory_percent‚82.88345336914062# proc.memory.rssMB‚ 2803.14453125 disk.dm-0.out‚ 151246.078125â3Ø•Ÿ¬5‚(:” “áéÒ†òŸø gpu.0.gpu‚99.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚245.379' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚49.075799999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0eÙÙ]N­5‚(:à “áéÒý·íù) disk.nvme0c0n1.out‚151261.12255859375 disk.sda2.in‚0# proc.memory.rssMB‚ 2803.14453125* proc.memory.percent‚17.109036445617676$ disk.dm-0.out‚151261.12255859375 disk.rbd3.out‚ 0.4951171875 disk.sda.in‚ 108.921875 disk.sda1.out‚0 disk.sda3.out‚ 1355.20703125 disk.rbd3.in‚0 network.sent‚ 2.5888222e+07) disk./.usagePercent‚37.96815944182001 disk.drbd2165.out‚0$ disk.nvme0c0n1.in‚52635.40234375' disk.nvme0n1.out‚151261.12255859375 disk.sda2.out‚0 disk.sda.out‚ 1355.20703125 cpu‚25.585888608664156 disk.dm-1.in‚ 108.921875 disk.dm-1.out‚ 1355.20703125& disk.rbd0.in‚1.759218604441423e+13 disk.rbd0.out‚0 disk.drbd2165.in‚0 disk.rbd1.out‚ 57.7451171875& proc.memory.availableMB‚ 2804.34375 proc.cpu.threads‚17" disk.nvme0n1.in‚52635.40234375 disk.sda3.in‚ 108.921875 network.recv‚ 5.509358e+06$ disk./.usageGB‚565.7530136108398 disk.dm-0.in‚52635.40234375 disk.sda1.in‚0 disk.rbd1.in‚ 0.53173828125# memory_percent‚82.8836441040039.öògž®5‚(:“ ¢áéÒ‰ËÙø gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚294.335' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚58.86699999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 OXK¯5‚(:À ¢áéÒ¶”›û network.recv‚ 5.512731e+06 disk.dm-0.in‚52635.62109375$ disk.dm-0.out‚151268.58837890625 disk.sda1.in‚0 disk.rbd3.in‚0 cpu‚25.585770914908974 disk.sda3.out‚ 1355.6640625 disk.drbd2165.out‚0 disk.sda2.in‚0# proc.memory.rssMB‚ 2803.14453125* proc.memory.percent‚17.109036445617676 network.sent‚ 2.5892803e+07) disk./.usagePercent‚37.96846101869535 disk.sda3.in‚ 108.921875 disk.drbd2165.in‚0' disk.nvme0n1.out‚151268.58837890625# memory_percent‚82.8836441040039 proc.cpu.threads‚17$ disk./.usageGB‚565.7575073242188 disk.sda1.out‚0& proc.memory.availableMB‚ 2804.34375 disk.rbd1.in‚ 0.53173828125 disk.sda.in‚ 108.921875 disk.dm-1.out‚ 1355.6640625& disk.rbd0.in‚1.759218604441423e+13" disk.nvme0n1.in‚52635.62109375 disk.sda.out‚ 1355.6640625$ disk.nvme0c0n1.in‚52635.62109375) disk.nvme0c0n1.out‚151268.58837890625 disk.rbd1.out‚ 57.8232421875 disk.dm-1.in‚ 108.921875 disk.rbd0.out‚0 disk.rbd3.out‚ 0.4951171875 disk.sda2.out‚0†?zÜž°5‚(:“ ±áéÒ›áÿö gpu.0.gpu‚90.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚287.28' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚57.455999999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0"ÆLM±5‚(: ±áéÒ¨‘Ùù network.recv‚ 5.515977e+06$ disk./.usageGB‚565.7643013000488 disk.rbd0.out‚0 disk.sda1.out‚0 cpu‚25.585615193072254 disk.sda2.out‚0$ disk.dm-0.out‚151277.10986328125 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.rbd3.in‚0) disk.nvme0c0n1.out‚151277.10986328125 network.sent‚ 2.5897903e+07 disk.dm-1.in‚ 108.921875# proc.memory.rssMB‚ 2803.14453125 proc.cpu.threads‚17 disk.dm-0.in‚52635.82421875 disk.sda1.in‚0 disk.sda3.out‚ 1355.9296875$ disk.nvme0c0n1.in‚52635.82421875 disk.sda.out‚ 1355.9296875 disk.sda3.in‚ 108.921875 disk.rbd3.out‚ 0.4951171875 disk.rbd1.in‚ 0.53564453125 disk.rbd1.out‚ 57.8271484375& disk.rbd0.in‚1.759218604441423e+13" disk.nvme0n1.in‚52635.82421875' disk.nvme0n1.out‚151277.10986328125" memory_percent‚82.883620262146* proc.memory.percent‚17.109036445617676 disk.sda.in‚ 108.921875) disk./.usagePercent‚37.96891696811383 disk.sda2.in‚0 disk.dm-1.out‚ 1355.9296875) proc.memory.availableMB‚ 2804.34765625 ;ꔲ5‚(:‰ ÀáéÒ⧓ø gpu.0.gpu‚94.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚339.701' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚67.9402 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0г£ S³5‚(:È ÀáéÒšòñø* disk./.usagePercent‚37.969366005328894 disk.sda2.in‚0 cpu‚25.585491766169913 disk.dm-1.out‚ 1356.07421875 disk.drbd2165.in‚0 network.sent‚ 2.5903577e+07 disk.rbd1.in‚ 0.53564453125 disk.sda.out‚ 1356.07421875 disk.sda1.in‚0 network.recv‚ 5.519193e+06$ disk./.usageGB‚565.7709922790527 disk.rbd1.out‚ 57.8896484375$ disk.dm-0.out‚151289.51025390625 disk.sda1.out‚0 disk.drbd2165.out‚0 disk.rbd3.out‚ 0.4951171875' disk.nvme0n1.out‚151289.51025390625 disk.dm-1.in‚ 108.921875 disk.rbd3.in‚0* proc.memory.percent‚17.109036445617676 disk.sda.in‚ 108.921875 disk.sda3.out‚ 1356.07421875$ memory_percent‚82.88242816925049 proc.cpu.threads‚17$ disk.nvme0c0n1.in‚52742.82421875 disk.sda2.out‚0 disk.rbd0.out‚0 disk.sda3.in‚ 108.921875) disk.nvme0c0n1.out‚151289.51025390625" disk.nvme0n1.in‚52742.82421875& disk.rbd0.in‚1.759218604441423e+13 disk.dm-0.in‚52742.82421875) proc.memory.availableMB‚ 2804.54296875# proc.memory.rssMB‚ 2803.14453125ƒïÛ4´5/ _wandbruntime‚22195  _runtime‚22195˜”]ɵ5à _step‚1101 & lr-AdamW/pg1‚0.0001762214504335316 & lr-AdamW/pg2‚0.0001762214504335316 trainer/global_step‚13399 % _timestamp‚1.7843120116224804e+09 _runtime‚22195.83964729; ú ζ5È & lr-AdamW/pg1‚0.0001762214504335316 & lr-AdamW/pg2‚0.0001762214504335316 trainer/global_step‚13399 % _timestamp‚1.7843120116224804e+09 _runtime‚22195.83964729 _step‚1101Íwýé™4·5/ _wandbruntime‚22195  _runtime‚22195´æË”(¸5”P [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12583202429917 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3864226789227520 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚275 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 trainer/global_step‚13399 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 116054255 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3864225417664512 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1040097686507520 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 114320963 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1056350484731904 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚3849957310738432 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 230375224 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 114320963 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚3864225417664512 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36633600 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 114320969 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0^jÂÙ(¹5™P G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3862541883196639 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚37957333 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3864226789227520 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12604770744320 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 116044282 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚3862540513168349 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 114320963 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1056350484731904 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚3849957310738432 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 230375224 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 114320963 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚3864225417664512 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36633600 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 114320969 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12583202429917 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3864226789227520 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚275 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 trainer/global_step‚13399 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 116054255 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3864225417664512 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1040097686507520 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚34377241 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1056429154973696 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404717568 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36626432 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12607041176576 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12605695779840 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3851898647650304 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 114329175 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1056428750256128 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚37959840 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 230391922 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12605695779840 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3850235217944576 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3862820688441173 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12584125896163 _runtime‚22197.224831399 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 116052760 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 E?DeviceStatsMonitor.on_train_batch_end/oversizµíÖ»le_allocations.peak‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3850235192516608 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚34377236 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚3851898647650304 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12607041176576 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 230391922 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚72337076 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16253945432064 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 114329169 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1040175172915200 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 116062747 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚3862819318412771 % _timestamp‚1.7843120130074446e+09 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 230381929 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 230381929 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3851898673823744 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚34377241 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1056429154973696 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404717568 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36626432 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12607041176576 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12605695779840 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3851898647650304 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 114329175 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1056428750256128 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚37959840 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 230391922 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12605695779840 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3850235217944576 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3862820688441173 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12584125896163 _runtime‚22197.224831399 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 116052760 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3850235192516608 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚34377236 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚3851898647650304 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12607041176576 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 230391922 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚72337076 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16253945432064 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 114329169 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1040175172915200 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 116062747 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚3862819318412771 % _timestamp‚1.7843120130074446e+09 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 230381929 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 230381929 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3851898673823744 EDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚3878171770100736 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 116480007 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 231201297 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12630784143347 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12652432064512 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16312257742848 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3865520709599232 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚72596803 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3878171770100736 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 231211276 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚252 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚3865520683425792 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3876482209218547 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 trainer/global_step‚13449 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3878173141663744 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 114731263 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚72597060 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚38095212 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚3863851399647232 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7843120903681626e+09 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚3876480839190257 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12651086674944 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 _step‚1106 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚34501848 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 QCDeviceStatsMonitor.on_traHëÂ] in_batch_start/allocation.small_pool.freed‚ 116470034 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚38094960 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12629439543025 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1043972006891520 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 114731263 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1060283896543232 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1043971638800384 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚3878171770100736 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 116480007 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 231201297 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12630784143347 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12652432064512 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16312257742848 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3865520709599232 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚72596803 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3878171770100736 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 231211276 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚252 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚3865520683425792 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3876482209218547 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 trainer/global_step‚13449 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3878173141663744 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 114731263 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚72597060 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚38095212 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚3863851399647232 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7843120903681626e+09 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚3876480839190257 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12651086674944 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 _step‚1106 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚34501848 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 116470034 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚38094960 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12629439543025 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008ÒaÆï"”Ð5‚(:‰ šâéÒÕ¾ãö gpu.0.gpu‚92.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚46.0 gpu.0.powerWatts‚335.402' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚67.0804 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0rnDèLÑ5‚(:Á šâéÒŽê¼ù disk.dm-1.in‚ 108.921875) proc.memory.availableMB‚ 2804.38671875' disk.nvme0c0n1.out‚151359.072265625" disk.nvme0n1.in‚52863.64453125 disk.sda2.in‚0 cpu‚25.584663200989485$ disk./.usageGB‚565.8091163635254$ disk.nvme0c0n1.in‚52863.64453125 disk.sda2.out‚0 disk.dm-0.in‚52863.64453125# proc.memory.rssMB‚ 2803.14453125 disk.drbd2165.out‚0& disk.rbd0.in‚1.759218604441423e+13* proc.memory.percent‚17.109036445617676 proc.cpu.threads‚17 network.sent‚ 2.5992582e+07 network.recv‚ 5.540426e+06 disk.rbd1.out‚ 58.1240234375 disk.rbd0.out‚0 disk.sda3.out‚ 1357.69140625# memory_percent‚82.8833818435669 disk.sda.in‚ 108.921875 disk.dm-1.out‚ 1357.69140625 disk.sda3.in‚ 108.921875* disk./.usagePercent‚37.971924544626084 disk.sda.out‚ 1357.69140625" disk.dm-0.out‚151359.072265625 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.rbd3.in‚0 disk.rbd3.out‚ 0.4951171875% disk.nvme0n1.out‚151359.072265625 disk.rbd1.in‚ 0.53564453125Zd4Ò5/ _wandbruntime‚22275  _runtime‚22275ú¹$'Ó5žN TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 114739475 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 116488499 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 231217981 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 116478512 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1060362575478784 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16313484800000 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1060362978099200 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚13449 _runtime‚22275.970220014 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3876761014463081 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3864129306853376 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 _step‚1107 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3878452067436544 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3878450695866368 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3864129281425408 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12652011710464 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 % _timestamp‚1.7843120917527392e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚34504391 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12652011710464 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚3876759644434679 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 116488499 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 116478512 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚34504386 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 114739469 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚241 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1044049493299200 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3878452067436544 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3865798710329344 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12653357107200 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 231227974 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 114739475 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 114739469 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 231227974 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚72602060 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚38097910 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12630363009271 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3865798684155904 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 231217981 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648×-6DXÔ5£N OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 116478512 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚34504386 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 114739469 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚241 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1044049493299200 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3878452067436544 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3865798710329344 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12653357107200 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 231227974 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 114739475 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 114739469 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 231227974 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚72602060 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚38097910 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12630363009271 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3865798684155904 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 231217981 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 114739475 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 116488499 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 231217981 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 116478512 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1060362575478784 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16313484800000 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1060362978099200 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚13449 _runtime‚22275.970220014 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3876761014463081 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3864129306853376 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 _step‚1107 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all1¹4Ñ.current‚ 49071259648 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3878452067436544 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3878450695866368 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3864129281425408 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12652011710464 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 % _timestamp‚1.7843120917527392e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚34504391 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12652011710464 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚3876759644434679 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 116488499 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176Ó9Xi4Õ5/ _wandbruntime‚22276  _runtime‚22276@tÆhÖ5Š 0loss/FRt1/assignment_loss‚1.0275904063791963 1loss/FRt2/assignment_loss‚0.20212770202172498 3loss/SRqqt2/assignment_loss‚0.16950266301653744 .loss/FRt2/detection_loss‚1.242602468252203 /loss/FBt1/detection_loss‚0.3087387936859159 2loss/SRqqt1/assignment_loss‚0.1315542616268293 3loss/FBt2/assignment_loss‚7.171149941669853e-06 0loss/FRt1/detection_loss‚0.43252561788813054 0loss/SRqqt1/detection_loss‚0.662344282201154 % _timestamp‚1.7843120918393283e+09 3loss/FBt1/assignment_loss‚0.0002074496695847472 /loss/FBt2/detection_loss‚0.3426430223574525 epoch‚33 trainer/global_step‚13449 _step‚1108 1loss/SRqqt2/detection_loss‚0.5282688018077868 %loss/total_loss‚5.048112640056457 _runtime‚22276.0562617448SÇ¥×5 2loss/SRqqt1/assignment_loss‚0.1315542616268293 3loss/FBt2/assignment_loss‚7.171149941669853e-06 0loss/FRt1/detection_loss‚0.43252561788813054 0loss/SRqqt1/detection_loss‚0.662344282201154 % _timestamp‚1.7843120918393283e+09 3loss/FBt1/assignment_loss‚0.0002074496695847472 /loss/FBt2/detection_loss‚0.3426430223574525 epoch‚33 trainer/global_step‚13449 _step‚1108 1loss/SRqqt2/detection_loss‚0.5282688018077868 %loss/total_loss‚5.048112640056457 _runtime‚22276.056261744 0loss/FRt1/assignment_loss‚1.0275904063791963 1loss/FRt2/assignment_loss‚0.20212770202172498 3loss/SRqqt2/assignment_loss‚0.16950266301653744 .loss/FRt2/detection_loss‚1.242602468252203 /loss/FBt1/detection_loss‚0.3087387936859159Ôt ÓOžØ5‚(:“ ©âéÒ”Îüö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚262.648' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚52.52960000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0šs¬JÙ5‚(:¿ ©âéÒã˶ù disk.sda.out‚ 1358.2265625" disk.nvme0n1.in‚52864.06640625 disk.sda2.in‚0) proc.memory.availableMB‚ 2804.06640625# proc.memory.rssMB‚ 2803.14453125 cpu‚25.584527969801833 disk.rbd1.out‚ 58.1240234375 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd3.out‚ 0.4951171875 disk.sda2.out‚0 disk.rbd1.in‚ 0.53564453125$ disk./.usageGB‚565.8182830810547 disk.drbd2165.in‚0$ memory_percent‚82.88533687591553 disk.sda.in‚ 108.921875& disk.rbd0.in‚1.759218604441423e+13 disk.dm-0.in‚52864.06640625 disk.drbd2165.out‚0$ disk.nvme0c0n1.in‚52864.06640625 disk.rbd0.out‚0" disk.dm-0.out‚151369.271484375% disk.nvme0n1.out‚151369.271484375* proc.memory.percent‚17.109036445617676 network.sent‚ 2.6045316e+07 network.recv‚ 5.544528e+06* disk./.usagePercent‚37.972539730730865 disk.dm-1.out‚ 1358.2265625 disk.sda3.in‚ 108.921875' disk.nvme0c0n1.out‚151369.271484375 disk.dm-1.in‚ 108.921875 disk.sda3.out‚ 1358.2265625 disk.rbd3.in‚0 proc.cpu.threads‚17n$+¬žÚ5‚(:“ ¸âéÒµÌèö gpu.0.gpu‚99.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚332.989' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚66.59779999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0E²ªœAÛ5‚(:¶ ¸âéÒûù‡ù" disk.nvme0n1.out‚ 151382.140625 disk.sda2.in‚0 disk.dm-1.in‚ 108.921875 disk.sda3.in‚ 108.921875 disk.sda.in‚ 108.921875 disk.dm-1.out‚ 1358.8203125 disk.dm-0.out‚ 151382.140625 disk.sda1.in‚0$ disk.nvme0c0n1.in‚52864.28515625$ disk.nvme0c0n1.out‚ 151382.140625$ memory_percent‚82.88381099700928) proc.memory.availableMB‚ 2804.31640625 network.sent‚ 2.6098018e+07 network.recv‚ 5.549042e+06$ disk./.usageGB‚565.8227119445801 disk.drbd2165.in‚0 disk.rbd3.out‚ 0.4951171875 disk.sda.out‚ 1358.8203125 disk.rbd1.out‚ 58.1865234375 disk.dm-0.in‚52864.28515625 disk.drbd2165.out‚0 disk.rbd3.in‚0 disk.sda2.out‚0& disk.rbd0.in‚1.759218604441423e+13 disk.rbd0.out‚0 disk.sda1.out‚0# proc.memory.rssMB‚ 2803.14453125* proc.memory.percent‚17.109036445617676 proc.cpu.threads‚17" disk.nvme0n1.in‚52864.28515625 disk.sda3.out‚ 1358.8203125 cpu‚25.584404876878633* disk./.usagePercent‚37.972836955478115 disk.rbd1.in‚ 0.53564453125¨¾"WŸÜ5‚(:” ÇâéÒ§°ø gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚273.894' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚54.778800000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚09Ðÿ<Ý5‚(:± ÇâéÒŒé¹ù disk.sda1.out‚0$ disk./.usageGB‚565.8294105529785" disk.dm-0.out‚151394.490234375# memory_percent‚82.8838586807251# proc.memory.rssMB‚ 2803.14453125 disk.dm-1.out‚ 1359.1875 disk.rbd0.out‚0 disk.sda2.in‚0) proc.memory.availableMB‚ 2804.30859375! disk.nvme0n1.in‚ 52955.7109375 disk.sda2.out‚0 network.recv‚ 5.552437e+06& disk.rbd0.in‚1.759218604441423e+13 disk.dm-0.in‚ 52955.7109375 disk.sda3.out‚ 1359.1875 disk.drbd2165.out‚0 disk.rbd1.in‚ 0.53564453125 disk.sda.out‚ 1359.1875 disk.sda3.in‚ 108.921875* proc.memory.percent‚17.109036445617676 proc.cpu.threads‚17 network.sent‚ 2.6103113e+07 disk.sda.in‚ 108.921875 disk.drbd2165.in‚0' disk.nvme0c0n1.out‚151394.490234375 disk.rbd3.out‚ 0.4951171875 disk.rbd1.out‚ 58.1865234375) disk./.usagePercent‚37.97328650470824 disk.sda1.in‚0 disk.rbd3.in‚0 cpu‚25.584246580905475 disk.dm-1.in‚ 108.921875# disk.nvme0c0n1.in‚ 52955.7109375% disk.nvme0n1.out‚151394.490234375°¨ßžÞ5‚(:“ ÖâéÒ¶¸Ûö gpu.0.gpu‚93.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚301.872' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚60.37440000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0HxÆGJß5‚(:¿ ÖâéÒ–‚®ù network.sent‚ 2.6107695e+07 disk.sda3.in‚ 108.921875 disk.sda.out‚ 1359.53515625# disk.dm-0.out‚151401.9267578125 disk.sda1.out‚0! disk.nvme0n1.in‚ 52956.1328125& disk.nvme0n1.out‚151401.9267578125 disk.dm-0.in‚ 52956.1328125$ memory_percent‚82.88388252258301 disk.rbd3.out‚ 0.4951171875 disk.sda1.in‚0 disk.sda3.out‚ 1359.53515625 disk.dm-1.in‚ 108.921875 disk.sda2.out‚0 disk.rbd1.in‚ 0.53564453125) disk./.usagePercent‚37.97376754286508 disk.rbd3.in‚0 disk.sda2.in‚0 disk.dm-1.out‚ 1359.53515625# proc.memory.rssMB‚ 2803.14453125 proc.cpu.threads‚17 disk.drbd2165.in‚0# disk.nvme0c0n1.in‚ 52956.1328125( disk.nvme0c0n1.out‚151401.9267578125 disk.sda.in‚ 108.921875( proc.memory.availableMB‚ 2804.3046875* proc.memory.percent‚17.109036445617676 cpu‚25.58413439013376 network.recv‚ 5.555825e+06$ disk./.usageGB‚565.8365783691406 disk.drbd2165.out‚0 disk.rbd1.out‚ 58.2646484375& disk.rbd0.in‚1.759218604441423e+13 disk.rbd0.out‚0¦¬!ƒ”à5‚(:‰ åâéÒ­âÚö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚259.116' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚51.8232 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÒÙÛNá5‚(:à åâéÒ¨Ùåú' disk.nvme0n1.out‚151418.56787109375 disk.sda2.out‚0 disk.drbd2165.in‚0* proc.memory.percent‚17.109036445617676 disk.sda.out‚ 1359.78515625$ disk.dm-0.out‚151418.56787109375 disk.sda3.out‚ 1359.78515625 disk.rbd1.out‚ 58.2646484375$ disk./.usageGB‚565.8411483764648 disk.sda3.in‚ 108.921875 disk.drbd2165.out‚0# proc.memory.rssMB‚ 2803.14453125 disk.rbd3.out‚ 0.4951171875 disk.sda2.in‚0& disk.rbd0.in‚1.759218604441423e+13 disk.rbd0.out‚0 proc.cpu.threads‚17 network.sent‚ 2.6113319e+07 network.recv‚ 5.558766e+06# disk.nvme0c0n1.in‚ 52956.3515625 disk.rbd1.in‚ 0.53564453125( disk./.usagePercent‚37.9740742398911) disk.nvme0c0n1.out‚151418.56787109375 disk.dm-1.in‚ 108.921875$ memory_percent‚82.88538455963135 disk.rbd3.in‚0 disk.sda.in‚ 108.921875 disk.dm-0.in‚ 52956.3515625 disk.sda1.out‚0 cpu‚25.583996346844245! disk.nvme0n1.in‚ 52956.3515625 disk.dm-1.out‚ 1359.78515625 disk.sda1.in‚0) proc.memory.availableMB‚ 2804.05859375E—ñ74â5/ _wandbruntime‚22353  _runtime‚22353‡]àyÌã5Æ ' lr-AdamW/pg2‚0.00017141185590090107 trainer/global_step‚13499 % _timestamp‚1.7843121689196062e+09 _runtime‚22353.136985011 _step‚1109 ' lr-AdamW/pg1‚0.00017141185590090107W½\%Ñä5Ë ' lr-AdamW/pg1‚0.00017141185590090107 ' lr-AdamW/pg2‚0.00017141185590090107 trainer/global_step‚13499 % _timestamp‚1.7843121689196062e+09 _runtime‚22353.136985011 _step‚1109ÕGÚ,4å5/ _wandbruntime‚22353  _runtime‚22353·Ð&¶(æ5“P ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚34628998 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12698747995136 _step‚1110 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3879420719931392 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚3879420746104832 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3892118122536960 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 232037349 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚3892118122536960 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚256 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3879420746104832 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚38232559 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚38232308 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 % _timestamp‚1.7843121689209347e+09 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1064218385916928 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1047846327275520 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 trainer/global_step‚13499 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 232047328 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 116895786 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12698747995136 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16372024104960 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12697402605568 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3892119494099968 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚3879420719931392 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚34628993 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1047845959184384 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3892119494099968 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚72861301 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 116905759 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 115141569 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12697402605568 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192-:Q$(ç5˜P QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 116905759 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 115141569 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12697402605568 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚34628998 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12698747995136 _step‚1110 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3879420719931392 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚3879420746104832 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3892118122536960 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 232037349 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚3892118122536960 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚256 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3879420746104832 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚38232559 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚38232308 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 % _timestamp‚1.7843121689209347e+09 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1064218385916928 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1047846327275520 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 trainer/global_step‚13499 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 232047328 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 116895786 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12698747995136 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16372024104960 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12697402605568 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3892119494099968 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚3879420719931392 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚34628993 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1047845959184384 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3892119494099968 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚72861301 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995Öæ`é5žN @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 % _timestamp‚1.7843121703067014e+09 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚251 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 TEDeviceStatsÀ:±ÒÄ%Monitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1047923813683200 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12677944722813 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 trainer/global_step‚13499 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12698327641088 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚38235035 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 232054033 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1064297059405824 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12676600122379 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚1111 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 115149775 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 232064026 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚72866571 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚38235281 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1047923445592064 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16373245722624 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3890701340484989 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 116914251 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚72866822 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416Z<´*)'ê5£N VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12698327641088 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚38235035 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 232054033 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1064297059405824 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12676600122379 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚1111 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 115149775 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 232064026 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚72866571 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚38235281 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1047923445592064 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16373245722624 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3890701340484989 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 116914251 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚72866822 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 % _timestamp‚1.7843121703067014e+09 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚251 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1047923813683200 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12677944722813 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 trainer/global_step‚13499 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128×ðôí4ë5/ _wandbruntime‚22354  _runtime‚22354ø`;ì5Œ 2loss/SRqqt2/assignment_loss‚0.2081871364400794 3loss/FBt2/assignment_loss‚5.639142299611874e-06 /loss/FRt1/detection_loss‚0.4333848530410045 .loss/FRt2/detection_loss‚1.279154089706922 /loss/FBt1/detection_loss‚0.3969649275791567 trainer/global_step‚13499 _runtime‚22354.609272229 %loss/total_loss‚5.016846773245515 epoch‚33 _step‚1112 0loss/FRt1/assignment_loss‚1.1052975462182089 1loss/FRt2/assignment_loss‚0.12539753368839365 3loss/SRqqt1/assignment_loss‚0.10506397901153912 4loss/FBt1/assignment_loss‚0.00032863327787578255 1loss/SRqqt1/detection_loss‚0.5865094758436645 % _timestamp‚1.7843121703923187e+09 1loss/SRqqt2/detection_loss‚0.4635746425638924 0loss/FBt2/detection_loss‚0.31297831673247806/0Åí5‘ _runtime‚22354.609272229 %loss/total_loss‚5.016846773245515 epoch‚33 _step‚1112 0loss/FRt1/assignment_loss‚1.1052975462182089 1loss/FRt2/assignment_loss‚0.12539753368839365 3loss/SRqqt1/assignment_loss‚0.10506397901153912 4loss/FBt1/assignment_loss‚0.00032863327787578255 1loss/SRqqt1/detection_loss‚0.5865094758436645 % _timestamp‚1.7843121703923187e+09 1loss/SRqqt2/detection_loss‚0.4635746425638924 0loss/FBt2/detection_loss‚0.31297831673247806 2loss/SRqqt2/assignment_loss‚0.2081871364400794 3loss/FBt2/assignment_loss‚5.639142299611874e-06 /loss/FRt1/detection_loss‚0.4333848530410045 .loss/FRt2/detection_loss‚1.279154089706922 /loss/FBt1/detection_loss‚0.3969649275791567 trainer/global_step‚13499ØêÚ]”î5‚(:‰ ôâéÒÈöçö gpu.0.gpu‚91.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚250.509' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚50.1018 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0£©Þ,Iï5‚(:¾ ôâéÒÇàãù# disk.nvme0c0n1.in‚ 52956.7734375! disk.nvme0n1.in‚ 52956.7734375 disk.sda2.in‚0 disk.rbd0.out‚0 disk.dm-0.in‚ 52956.7734375 disk.rbd3.in‚0 disk.rbd3.out‚ 0.4951171875 disk.sda3.in‚ 108.921875$ disk./.usageGB‚565.8504524230957 disk.rbd1.in‚ 0.53564453125 disk.rbd1.out‚ 58.3896484375 disk.sda.out‚ 1360.109375 network.recv‚ 5.562147e+06 disk.sda2.out‚0& disk.rbd0.in‚1.759218604441423e+13 disk.drbd2165.out‚0* proc.memory.percent‚17.109036445617676 disk.dm-1.out‚ 1360.109375# proc.memory.rssMB‚ 2803.14453125) disk.nvme0c0n1.out‚151426.62255859375 proc.cpu.threads‚17 disk.sda1.in‚0 disk.sda1.out‚0 network.sent‚ 2.6118504e+07* disk./.usagePercent‚37.974698642267114 disk.sda.in‚ 108.921875$ disk.dm-0.out‚151426.62255859375$ memory_percent‚82.88524150848389 disk.dm-1.in‚ 108.921875 disk.sda3.out‚ 1360.109375 disk.drbd2165.in‚0) proc.memory.availableMB‚ 2804.08203125 cpu‚25.58387546561186' disk.nvme0n1.out‚151426.62255859375ƒ'Šé”ð5‚(:‰ ƒãéÒèÊàö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚39.0 gpu.0.powerWatts‚112.997' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.5994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0[å}Mñ5‚(: ƒãéÒ’àêù proc.cpu.threads‚17 network.recv‚ 5.566233e+06& disk.rbd0.in‚1.759218604441423e+13 disk.dm-0.in‚53066.25390625 disk.sda1.in‚0' disk.nvme0n1.out‚151442.20458984375 disk.rbd1.out‚ 58.3896484375$ memory_percent‚82.88536071777344 cpu‚25.583729922309544 disk.drbd2165.out‚0 disk.sda2.in‚0 disk.sda2.out‚0% proc.memory.availableMB‚ 2804.0625# proc.memory.rssMB‚ 2803.14453125) disk./.usagePercent‚37.97497077827601 disk.dm-1.out‚ 1360.45703125$ disk.dm-0.out‚151442.20458984375 disk.sda3.out‚ 1360.45703125" disk.nvme0n1.in‚53066.25390625 disk.sda.out‚ 1360.45703125 disk.dm-1.in‚ 108.921875 disk.sda3.in‚ 108.921875 disk.rbd0.out‚0 disk.rbd1.in‚ 0.53564453125 network.sent‚ 2.618186e+07 disk.drbd2165.in‚0$ disk.nvme0c0n1.in‚53066.25390625$ disk./.usageGB‚565.8545074462891 disk.sda.in‚ 108.921875 disk.sda1.out‚0 disk.rbd3.in‚0 disk.rbd3.out‚ 0.4951171875) disk.nvme0c0n1.out‚151442.20458984375* proc.memory.percent‚17.109036445617676˜k8Äžò5‚(:“ ’ãéÒÇ¡Š÷ gpu.0.gpu‚92.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚300.808' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚60.16160000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¸êËZIó5‚(:¾ ’ãéÒÔôíù disk.sda.out‚ 1360.62890625 disk.sda2.in‚0 disk.rbd1.out‚ 58.4677734375* proc.memory.percent‚17.109036445617676 disk.rbd0.out‚0$ disk.nvme0c0n1.in‚53066.47265625& disk.nvme0n1.out‚151449.9755859375( proc.memory.availableMB‚ 2803.8984375 disk.sda1.out‚0 network.recv‚ 5.569785e+06 disk.dm-1.out‚ 1360.62890625& disk.rbd0.in‚1.759218604441423e+13 disk.sda3.in‚ 108.921875 disk.drbd2165.in‚0 disk.rbd3.in‚0" disk.nvme0n1.in‚53066.47265625 disk.dm-1.in‚ 108.921875 disk.sda2.out‚0$ memory_percent‚82.88636207580566! disk./.usageGB‚565.8603515625# disk.dm-0.out‚151449.9755859375 disk.rbd3.out‚ 0.4951171875 network.sent‚ 2.618694e+07# proc.memory.rssMB‚ 2803.14453125 cpu‚25.58361805480709 disk.rbd1.in‚ 0.53564453125 disk.sda.in‚ 108.921875 disk.dm-0.in‚53066.47265625 disk.drbd2165.out‚0( disk.nvme0c0n1.out‚151449.9755859375) disk./.usagePercent‚37.97536298181846 disk.sda1.in‚0 disk.sda3.out‚ 1360.62890625 proc.cpu.threads‚17Ú_jí’ô5‚(:‡ ¡ãéÒîâàö gpu.0.gpu‚93.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚261.23' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.246 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ø®×Oõ5‚(:Ä ¡ãéÒŠŽÑù disk.dm-0.in‚53066.89453125 disk.rbd3.in‚0 disk.sda2.in‚0 cpu‚25.583495815039846 disk.sda.out‚ 1361.1953125$ disk.dm-0.out‚151463.96240234375 disk.sda1.in‚0) disk.nvme0c0n1.out‚151463.96240234375* proc.memory.percent‚17.109036445617676 network.sent‚ 2.6192031e+07$ disk./.usageGB‚565.8672142028809 disk.dm-1.out‚ 1361.1953125 proc.cpu.threads‚17 disk.sda.in‚ 108.921875 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.rbd1.in‚ 0.53564453125 disk.dm-1.in‚ 108.921875& disk.rbd0.in‚1.759218604441423e+13 disk.sda2.out‚0# proc.memory.rssMB‚ 2803.14453125) disk./.usagePercent‚37.97582353937257 disk.drbd2165.out‚0" disk.nvme0n1.in‚53066.89453125' disk.nvme0n1.out‚151463.96240234375 disk.rbd1.out‚ 58.4677734375 disk.sda3.in‚ 108.921875) proc.memory.availableMB‚ 2803.90234375 network.recv‚ 5.573093e+06 disk.sda3.out‚ 1361.1953125 disk.rbd3.out‚ 0.4951171875$ disk.nvme0c0n1.in‚53066.89453125$ memory_percent‚82.88633823394775 disk.rbd0.out‚0ú¢J2”ö5‚(:‰ °ãéÒΩ°÷ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚265.534' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.1068 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0à$¾M÷5‚(: °ãéÒ€‡½ú$ disk./.usageGB‚565.8717956542969& disk.rbd0.in‚1.759218604441423e+13 disk.dm-0.in‚53067.09765625 disk.sda.out‚ 1361.37890625# disk.dm-0.out‚151472.0830078125 disk.sda3.in‚ 108.921875 cpu‚25.58334897550198 disk.rbd0.out‚0 disk.drbd2165.out‚0( disk.nvme0c0n1.out‚151472.0830078125 disk.sda2.out‚0) disk./.usagePercent‚37.97613100442119 disk.rbd1.out‚ 58.4677734375 disk.sda3.out‚ 1361.37890625 disk.rbd3.in‚0 disk.rbd3.out‚ 0.4951171875$ disk.nvme0c0n1.in‚53067.09765625" disk.nvme0n1.in‚53067.09765625 disk.sda2.in‚0$ memory_percent‚82.88521766662598 proc.cpu.threads‚17 network.recv‚ 5.575812e+06 disk.dm-1.out‚ 1361.37890625& disk.nvme0n1.out‚151472.0830078125 disk.sda.in‚ 108.921875 disk.dm-1.in‚ 108.921875 disk.sda1.out‚0 disk.drbd2165.in‚0* proc.memory.percent‚17.109036445617676 network.sent‚ 2.6196195e+07 disk.rbd1.in‚ 0.53564453125 disk.sda1.in‚0( proc.memory.availableMB‚ 2804.0859375# proc.memory.rssMB‚ 2803.14453125xçÜ×4ø5/ _wandbruntime‚22431  _runtime‚22431™F@9Êù5Ä trainer/global_step‚13549 % _timestamp‚1.7843122476907308e+09 _runtime‚22431.907992349 _step‚1113 & lr-AdamW/pg1‚0.0001690237944894081 & lr-AdamW/pg2‚0.0001690237944894081gø.Ïú5É % _timestamp‚1.7843122476907308e+09 _runtime‚22431.907992349 _step‚1113 & lr-AdamW/pg1‚0.0001690237944894081 & lr-AdamW/pg2‚0.0001690237944894081 trainer/global_step‚13549ÙŒõîR4û5/ _wandbruntime‚22431  _runtime‚22431ۈ ü5”P KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 % _timestamp‚1.7843122476918964e+09 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 115551869 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 117321538 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚73126825 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚34756143 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1068152386907136 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16431706808320 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 115551869 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1051720279568384 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚34756148 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3904362861262363 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚3891639602892800 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚3891639577464832 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚3893320782610432 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚242 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 _runtime‚22431.909224476 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚3893320756436992 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3893320782610432 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚38370677 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚3904361491234073 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 _step‚1114 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 115551863 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚38370435 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 trainer/global_step‚13549 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 117331511 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3906064474973184 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12721913769241 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3893320756436992 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12745063925760 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12743718536192 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3906065846536192 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 115551863 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚38370435 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 trainer/global_step‚13549 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 117331511 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3906064474973184 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12721913769241 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3893320756436992 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12745063925760 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12743718536192 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3906065846536192 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚73126825 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚34756143 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1068152386907136 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16431706808320 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 115551869 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1051720279568384 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚34756148 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3904362861262363 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚3891639602892800 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚3891639577464832 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚3893320782610432 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚242 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 _runtime‚22431.909224476 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚3893320756436992 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3893320782610432 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚38370677 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚3904361491234073 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 _step‚1114Ú€¤ 4þ5/ _wandbruntime‚22433  _runtime‚22433ÅT!|#'ÿ5N _step‚1115 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3893598757167104 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚73131881 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚238 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1068230669342720 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 117330016 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚3893598783340544 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚38373195 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 _runtime‚22433.29543408 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 115560075 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3904641666506897 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3893598783340544 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 trainer/global_step‚13549 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚3893598757167104 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1051798134067200 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1051797765976064 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3906344772308992 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 117340003 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16432937895936 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3906344772308992 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3906343400738816 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚38373433 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 232900078 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 117340003 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚34758686 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1068231071963136 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3891917484670976 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12745988968448 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12724181835921 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12722837235487 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3893598783340544 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 trainer/global_step‚13549 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚3893598757167104 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1051798134067200 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1051797765976064 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3906344772308992 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 117340003 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16432937895936 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3906344772308992 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3906343400738816 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.¼¤mTsmall_pool.allocated‚38373433 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 232900078 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 117340003 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚34758686 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1068231071963136 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3891917484670976 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12745988968448 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12724181835921 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12722837235487 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚73131881 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚238 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1068230669342720 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 117330016ÛÔÞ„Î46/ _wandbruntime‚22433  _runtime‚22433SÓ/Ú‚6‰ /loss/FRt1/assignment_loss‚1.076309503657321 2loss/SRqqt2/detection_loss‚0.46349277684719575 /loss/FBt1/detection_loss‚0.4162166796635244 # _timestamp‚1.78431224916383e+09 3loss/SRqqt1/assignment_loss‚0.09527398334619412 3loss/SRqqt2/assignment_loss‚0.17193971022432558 4loss/FBt1/assignment_loss‚0.00013292977951338193 .loss/FRt1/detection_loss‚0.431945807681727 /loss/FBt2/detection_loss‚0.2263934440883054 epoch‚33 _runtime‚22433.380756009 4loss/FBt2/assignment_loss‚2.2412153385930456e-06 /loss/FRt2/detection_loss‚1.2392320457425687 1loss/SRqqt1/detection_loss‚0.5836333115834813 $loss/total_loss‚4.86792681617195 trainer/global_step‚13549 0loss/FRt2/assignment_loss‚0.1633543823424544 _step‚1116òæ iƒ6Ž _step‚1116 /loss/FRt1/assignment_loss‚1.076309503657321 2loss/SRqqt2/detection_loss‚0.46349277684719575 /loss/FBt1/detection_loss‚0.4162166796635244 # _timestamp‚1.78431224916383e+09 3loss/SRqqt1/assignment_loss‚0.09527398334619412 3loss/SRqqt2/assignment_loss‚0.17193971022432558 4loss/FBt1/assignment_loss‚0.00013292977951338193 .loss/FRt1/detection_loss‚0.431945807681727 /loss/FBt2/detection_loss‚0.2263934440883054 epoch‚33 _runtime‚22433.380756009 4loss/FBt2/assignment_loss‚2.2412153385930456e-06 /loss/FRt2/detection_loss‚1.2392320457425687 1loss/SRqqt1/detection_loss‚0.5836333115834813 $loss/total_loss‚4.86792681617195 trainer/global_step‚13549 0loss/FRt2/assignment_loss‚0.1633543823424544Ü8)Ñž„6‚(:“ ¿ãéÒêÄñ÷ gpu.0.gpu‚91.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚333.764' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚66.75280000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0†ˆ B…6‚(:· ¿ãéÒŽ³ßù disk.nvme0n1.in‚ 53157.984375 disk.rbd0.out‚0$ disk./.usageGB‚565.8820304870605 disk.sda2.in‚0 disk.sda.in‚ 108.921875 disk.sda3.out‚ 1361.78125$ memory_percent‚82.88567066192627# proc.memory.rssMB‚ 2803.14453125* proc.memory.percent‚17.109036445617676 disk.rbd1.out‚ 58.5302734375& disk.rbd0.in‚1.759218604441423e+13 disk.dm-0.in‚ 53157.984375 disk.sda1.in‚0' disk.nvme0n1.out‚151483.86279296875 network.sent‚ 2.6201326e+07 disk.sda.out‚ 1361.78125$ disk.dm-0.out‚151483.86279296875) disk.nvme0c0n1.out‚151483.86279296875) proc.memory.availableMB‚ 2804.01171875) disk./.usagePercent‚37.97681787263556 proc.cpu.threads‚17 disk.rbd1.in‚ 0.53564453125 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.drbd2165.out‚0 cpu‚25.58321712850237 network.recv‚ 5.578862e+06 disk.dm-1.in‚ 108.921875 disk.dm-1.out‚ 1361.78125 disk.sda3.in‚ 108.921875 disk.rbd3.in‚0 disk.rbd3.out‚ 0.4951171875 disk.sda2.out‚0" disk.nvme0c0n1.in‚ 53157.984375>—ÍØ”†6‚(:‰ ÎãéÒ º‚ø gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚271.718' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.3436 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0º“HK‡6‚(:À ÎãéÒ¬ú™ù network.sent‚ 2.6264639e+07 disk.sda.out‚ 1362.13671875 disk.drbd2165.in‚0 disk.rbd0.out‚0$ disk.dm-0.out‚151496.78955078125 disk.sda1.out‚0 disk.sda3.in‚ 108.921875 disk.nvme0n1.in‚ 53158.203125# proc.memory.rssMB‚ 2803.14453125 disk.sda.in‚ 108.921875 disk.rbd3.in‚0 disk.sda2.out‚0$ memory_percent‚82.88564682006836' proc.memory.availableMB‚ 2804.015625* proc.memory.percent‚17.109036445617676$ disk./.usageGB‚565.8859634399414 disk.dm-1.in‚ 108.921875 disk.rbd3.out‚ 0.4951171875' disk.nvme0n1.out‚151496.78955078125* disk./.usagePercent‚37.977081816403356 disk.drbd2165.out‚0 disk.rbd1.in‚ 0.53564453125 disk.rbd1.out‚ 58.5302734375 cpu‚25.583073920409596" disk.nvme0c0n1.in‚ 53158.203125) disk.nvme0c0n1.out‚151496.78955078125 proc.cpu.threads‚17 disk.sda1.in‚0 disk.sda3.out‚ 1362.13671875 disk.sda2.in‚0 network.recv‚ 5.583356e+06 disk.dm-1.out‚ 1362.13671875& disk.rbd0.in‚1.759218604441423e+13 disk.dm-0.in‚ 53158.203125z>½“ˆ6‚(:ˆ ÝãéÒèêçö gpu.0.gpu‚100.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚329.76' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚65.952 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0_@WF‰6‚(:» ÝãéÒ®¥ôù disk.sda2.out‚0 disk.dm-1.out‚ 1362.7265625' disk.nvme0n1.out‚151514.19677734375 disk.rbd1.out‚ 58.6083984375 cpu‚25.582950139518296$ disk.dm-0.out‚151514.19677734375 disk.sda1.out‚0 network.recv‚ 5.586965e+06$ disk./.usageGB‚565.8960227966309 disk.rbd3.out‚ 0.4951171875 proc.cpu.threads‚17 disk.sda1.in‚0) disk./.usagePercent‚37.97775690827115 disk.sda.in‚ 108.921875 disk.sda.out‚ 1362.7265625 disk.dm-1.in‚ 108.921875$ memory_percent‚82.88638591766357# proc.memory.rssMB‚ 2803.14453125* proc.memory.percent‚17.109036445617676& disk.rbd0.in‚1.759218604441423e+13 disk.nvme0n1.in‚ 53158.40625) proc.memory.availableMB‚ 2803.89453125 disk.dm-0.in‚ 53158.40625 disk.sda2.in‚0 disk.rbd0.out‚0 disk.sda3.in‚ 108.921875 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.rbd3.in‚0) disk.nvme0c0n1.out‚151514.19677734375 disk.rbd1.in‚ 0.53564453125 network.sent‚ 2.6269914e+07 disk.sda3.out‚ 1362.7265625! disk.nvme0c0n1.in‚ 53158.40625XÄ «”Š6‚(:‰ ìãéÒê±òö gpu.0.gpu‚88.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚262.589' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.5178 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0{P–B‹6‚(:· ìãéÒÈé„ù network.recv‚ 5.590151e+06 disk.rbd1.out‚ 58.6083984375 disk.sda1.in‚0 network.sent‚ 2.6275036e+07 disk.rbd3.out‚ 0.4951171875 disk.sda2.out‚0 disk.rbd1.in‚ 0.53564453125 disk.drbd2165.in‚0! disk.nvme0c0n1.in‚ 53160.15625( disk.nvme0c0n1.out‚151527.4716796875# disk.dm-0.out‚151527.4716796875# memory_percent‚82.8869104385376 cpu‚25.582830272679757& disk.nvme0n1.out‚151527.4716796875 disk.dm-1.out‚ 1364.7578125 disk.sda1.out‚0$ disk./.usageGB‚565.9045867919922 disk.rbd3.in‚0 disk.nvme0n1.in‚ 53160.15625 disk.sda2.in‚0 disk.sda.in‚ 108.921875* proc.memory.percent‚17.109036445617676) disk./.usagePercent‚37.97833164518552 disk.drbd2165.out‚0 proc.cpu.threads‚17 disk.sda.out‚ 1364.7578125 disk.dm-1.in‚ 108.921875& disk.rbd0.in‚1.759218604441423e+13 disk.dm-0.in‚ 53160.15625 disk.sda3.in‚ 108.921875) proc.memory.availableMB‚ 2803.80859375 disk.rbd0.out‚0 disk.sda3.out‚ 1364.7578125# proc.memory.rssMB‚ 2803.14453125¼¥æŸŒ6‚(:” ûãéÒ¯¿÷ gpu.0.gpu‚98.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚285.446' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚57.089200000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 žD)M6‚(: ûãéÒÝøù network.sent‚ 2.6280111e+07& disk.rbd0.in‚1.759218604441423e+13 disk.rbd0.out‚0$ disk.dm-0.out‚151544.00341796875 disk.sda3.in‚ 108.921875* proc.memory.percent‚17.109036445617676 disk.dm-1.out‚ 1365.12890625 disk.rbd3.in‚0 network.recv‚ 5.593303e+06$ disk./.usageGB‚565.9087409973145 disk.sda.in‚ 108.921875 disk.drbd2165.in‚0 disk.rbd3.out‚ 0.4951171875 disk.rbd1.in‚ 0.53564453125 disk.rbd1.out‚ 58.6708984375( disk./.usagePercent‚37.9786104373903 disk.dm-0.in‚ 53240.5859375 disk.sda3.out‚ 1365.12890625' disk.nvme0n1.out‚151544.00341796875 disk.sda2.in‚0# proc.memory.rssMB‚ 2803.14453125 disk.sda1.in‚0 disk.sda1.out‚0) disk.nvme0c0n1.out‚151544.00341796875# disk.nvme0c0n1.in‚ 53240.5859375$ memory_percent‚82.88846015930176( proc.memory.availableMB‚ 2803.5546875 cpu‚25.582706363883442 proc.cpu.threads‚17 disk.sda.out‚ 1365.12890625 disk.dm-1.in‚ 108.921875 disk.drbd2165.out‚0! disk.nvme0n1.in‚ 53240.5859375 disk.sda2.out‚0Úç\4Ž6/ _wandbruntime‚22512  _runtime‚22512SªžGÌ6Æ _runtime‚22512.520347194 _step‚1117 ' lr-AdamW/pg1‚0.00016664709506206197 ' lr-AdamW/pg2‚0.00016664709506206197 trainer/global_step‚13599 % _timestamp‚1.7843123283031504e+09™ÞýÑ6Ë trainer/global_step‚13599 % _timestamp‚1.7843123283031504e+09 _runtime‚22512.520347194 _step‚1117 ' lr-AdamW/pg1‚0.00016664709506206197 ' lr-AdamW/pg2‚0.00016664709506206197ÝÆGzµ4‘6/ _wandbruntime‚22512  _runtime‚22512[¡†(’6”P LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 % _timestamp‚1.7843123283046343e+09 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3920010827409408 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3920012198972416 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16491483085824 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 115962163 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12768150882349 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12791379856384 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚248 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚3918301817255981 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚3920010827409408 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 117757263 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1055594968043520 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1055594599952384 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12790034466816 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚34883298 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1072086083038208 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3920012198972416 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16491517622272 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 trainer/global_step‚13599 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12791379856384 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚73391835 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 _step‚1118 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 115962163 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚38508542 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 115962169 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚253 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3918303187284271 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚3907220792942592 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1072086485665792 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175-oˆ‹“6™P A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 115962169 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚3905533691801600 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3907220792942592 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12790034466816 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚34883298 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1072086083038208 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3920012198972416 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16491517622272 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 trainer/global_step‚13599 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12791379856384 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚73391835 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 _step‚1118 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 115962163 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚38508542 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 115962169 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚253 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3918303187284271 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚3907220792942592 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1072086485665792 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 % _timestamp‚1.7843123283046343e+09 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3920010827409408 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3920012198972416 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16491483085824 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 115962163 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12768150882349 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12791379856384 QIDeviceStats+Ä”Monitor.on_train_batch_start/inactive_split.small_pool.current‚248 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚3918301817255981 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚3920010827409408 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 117757263 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1055594968043520 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1055594599952384 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚73397098 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16492675359744 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚34885836 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚34885841 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1072164761719808 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3920289753175040 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 233726137 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚236 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3920291124745216 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 233736130 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3905811573579776 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚13599 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 117765755 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12790959502336 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 115970375 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 117765755 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚73397339 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16492709889024 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640L@1*)'–6£N FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16492709889024 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 233736130 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 115970369 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚73397098 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16492675359744 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚34885836 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚34885841 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1072164761719808 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3920289753175040 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 233726137 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚236 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3920291124745216 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 233736130 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3905811573579776 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚13599 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 117765755 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12790959502336 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 115970375 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 117765755 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚73397339 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175ßœÄ1{4—6/ _wandbruntime‚22514  _runtime‚22514gX(”˜6Œ /loss/FRt2/detection_loss‚1.3336702617249827 $ _timestamp‚1.784312329793749e+09 2loss/FBt2/assignment_loss‚0.009616222338871475 1loss/SRqqt1/detection_loss‚0.5092319131109655 0loss/FBt1/detection_loss‚0.34604875537722063 0loss/FBt2/detection_loss‚0.24224587202093412 2loss/SRqqt1/assignment_loss‚0.1598610553712301 3loss/SRqqt2/assignment_loss‚0.19394083616687136 /loss/FRt1/detection_loss‚0.4245638008307123 2loss/SRqqt2/detection_loss‚0.48260043941088276 epoch‚33 _runtime‚22514.011030836 _step‚1120 0loss/FRt1/assignment_loss‚1.0839313392318086 1loss/FRt2/assignment_loss‚0.20774346668263433 3loss/FBt1/assignment_loss‚9.572788633163603e-07 %loss/total_loss‚4.993454919545977 trainer/global_step‚13599<åLh™6‘ %loss/total_loss‚4.993454919545977 trainer/global_step‚13599 /loss/FRt2/detection_loss‚1.3336702617249827 $ _timestamp‚1.784312329793749e+09 2loss/FBt2/assignment_loss‚0.009616222338871475 1loss/SRqqt1/detection_loss‚0.5092319131109655 0loss/FBt1/detection_loss‚0.34604875537722063 0loss/FBt2/detection_loss‚0.24224587202093412 2loss/SRqqt1/assignment_loss‚0.1598610553712301 3loss/SRqqt2/assignment_loss‚0.19394083616687136 /loss/FRt1/detection_loss‚0.4245638008307123 2loss/SRqqt2/detection_loss‚0.48260043941088276 epoch‚33 _runtime‚22514.011030836 _step‚1120 0loss/FRt1/assignment_loss‚1.0839313392318086 1loss/FRt2/assignment_loss‚0.20774346668263433 3loss/FBt1/assignment_loss‚9.572788633163603e-07àPì$’š6‚(:‡ ŠäéÒ˜öö gpu.0.gpu‚98.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚329.76' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚65.952 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0^þGÉ<›6‚(:± ŠäéÒšÙÝú$ disk./.usageGB‚565.9153594970703! disk.nvme0n1.in‚ 53240.7890625 disk.sda1.in‚0 disk.rbd3.out‚ 0.4951171875# proc.memory.rssMB‚ 2820.38671875* proc.memory.percent‚17.214274406433105 network.recv‚ 5.604066e+06* disk./.usagePercent‚37.979054610462214# disk.nvme0c0n1.in‚ 53240.7890625 disk.rbd0.out‚0 disk.dm-0.in‚ 53240.7890625' disk.nvme0c0n1.out‚151552.673828125" disk.dm-0.out‚151552.673828125 disk.sda1.out‚0 disk.rbd3.in‚0% disk.nvme0n1.out‚151552.673828125# memory_percent‚82.9925537109375 network.sent‚ 2.6289069e+07 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd1.in‚ 0.53564453125 disk.rbd1.out‚ 58.6708984375 disk.dm-1.in‚ 108.921875 disk.drbd2165.in‚0 disk.dm-1.out‚ 1365.359375 disk.sda3.out‚ 1365.359375 proc.cpu.threads‚17 disk.sda.in‚ 108.921875 disk.sda.out‚ 1365.359375& disk.rbd0.in‚1.759218604441423e+13 disk.drbd2165.out‚0 disk.sda3.in‚ 108.921875" proc.memory.availableMB‚2786.5 cpu‚25.582582926149573~t”œ6‚(:‰ ™äéÒˆ‰Äù gpu.0.gpu‚98.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚40.0 gpu.0.powerWatts‚342.642' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚68.5284 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0_ëÑN6‚(:à ™äéÒ¤ô©û disk.sda2.in‚0 disk.sda2.out‚0 disk.sda.in‚ 108.921875 disk.drbd2165.in‚0 disk.rbd3.out‚ 0.4951171875 disk.rbd1.out‚ 58.7490234375$ memory_percent‚83.20677280426025) proc.memory.availableMB‚ 2751.40234375 disk.rbd3.in‚0# disk.nvme0c0n1.in‚ 53241.2109375' disk.nvme0n1.out‚151568.07861328125 network.recv‚ 5.608514e+06) disk./.usagePercent‚37.97833932541155 disk.rbd1.in‚ 0.53564453125 disk.dm-1.out‚ 1365.82421875 disk.dm-0.in‚ 53241.2109375$ disk./.usageGB‚565.9047012329102 disk.sda.out‚ 1365.82421875 disk.sda3.in‚ 108.921875 disk.sda3.out‚ 1365.82421875! disk.nvme0n1.in‚ 53241.2109375 disk.sda1.in‚0 cpu‚25.584300178802817& disk.rbd0.in‚1.759218604441423e+13# proc.memory.rssMB‚ 2855.25390625) disk.nvme0c0n1.out‚151568.07861328125 proc.cpu.threads‚17 disk.sda1.out‚0 disk.dm-1.in‚ 108.921875$ disk.dm-0.out‚151568.07861328125 network.sent‚ 2.6352423e+07 disk.drbd2165.out‚0) proc.memory.percent‚17.42708683013916 disk.rbd0.out‚0¤í4ž6/ _wandbruntime‚22542  _runtime‚22542 û*pOŸ6ø{ ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_1‚1 particle/accuracy_3_of_3‚1 particle/accuracy_1_of_4‚1 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.6942709684371948 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 *jet/accuracy_1_of_4‚0.9611973762512207 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 jet/accuracy_2_of_5‚1 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.23362919688224792 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9983765482902527 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.036282215267419815 jet/accuracy_1_of_5‚1 jet/accuracy_5_of_5‚0 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7154194712638855 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7186991572380066 *jet/accuracy_2_of_3‚0.5192664265632629 * Purity/2FRt*SRqqt2FBt/FRt_purity‚0.625 jet/accuracy_4_of_5‚0 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9026336669921875 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 . Purity/2FRt0SRqqt1FBt/FRt_purity‚ 0.5859375 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8404529094696045 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9840746521949768 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 particle/accuracy_1_of_2‚1 particle/accuracy_2_of_4‚1 particle/accuracy_2_of_5‚1 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7183843851089478 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9719157218933105 _step‚1121 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 +jet/accuracy_3_of_4‚0.06704101711511612 + Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9375 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9879744648933411 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 /particle/accuracy_4_of_4‚0.9434183239936829 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.12210109084844589 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9903962016105652 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.41203704476356506 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6406445503234863 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9862955808639526 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.11356587707996368 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.19230769574642181 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.1556880623102188 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9930450916290283 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8559363484382629 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.15554645657539368 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 9"Purity/1FRt1SRqqt*FBt/event_purity‚0.0353284552693367 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.11499999463558197 *jet/accuracy_1_of_2‚0.7217177152633667 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.6428571343421936 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7165573239326477 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8432815074920654 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9537036418914795 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7210375666618347 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.42824074625968933 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7481002807617188 6 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.491666704416275 particle/accuracy_1_of_5‚1 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.02125358395278454 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8077256083488464 *jet/accuracy_2_of_4‚0.7331549525260925 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6948369145393372 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.18954609334468842 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 particle/accuracy_5_of_5‚1 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5226060152053833 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9928704500198364 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6659384965896606 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9937976002693176 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3499999940395355 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.03428864851593971 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5973998308181763 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.006806667894124985 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.2552083134651184 % _timestamp‚1.7843123582797968e+09 *loss/val_total_loss‚10.048317675002371 :"PurGï½Î¯'ity/*FRt0SRqqt*FBt/event_purity‚0.22441962361335754 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9166666865348816 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7127863168716431 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9537036418914795 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6717851161956787 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5312874913215637 6 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.158451646566391 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.15705294907093048 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 /particle/accuracy_3_of_4‚0.9965986609458923 jet/accuracy_3_of_5‚1 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9978824853897095 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2705225646495819 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6113886833190918 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9922001957893372 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7535344958305359 +jet/accuracy_3_of_3‚0.06253387778997421 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.19230769574642181 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6388929486274719 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.43752846121788025 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.05000000074505806 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6338998079299927 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.15812669694423676 particle/accuracy_4_of_5‚1 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.17428450286388397 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9951749444007874 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7157029509544373 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.003683497430756688 7validation_average_jet_accuracy‚0.32861199975013733 6 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.709901750087738 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 + Purity/0FRt2SRqqt1FBt/FBt_purity‚0.9375 particle/accuracy_2_of_2‚1 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt2SRqqt0FBt/event_purity‚0.2004629373550415 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.6501700282096863 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.5751368403434753 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.029256083071231842 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8077256083488464 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9038461446762085 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9953287839889526 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.45218536257743835 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 8"Purity/1FRt0SRqqt0FBt/event_purity‚0.158451646566391 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.17305199801921844 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9942217469215393 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4545869827270508 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 (particle/accuracy‚0.8367860913276672 trainer/global_step‚13599 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5823284983634949 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.17066538333892822 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.3735355734825134 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.034623801708221436 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5685039162635803 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 particle/accuracy_1_of_3‚1 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9825929999351501 8"Purity/0FRt2SRqqt0FBt/event_purity‚0.450819194316864 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.36039090156555176 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9975056648254395 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.1403181254863739 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9801885485649109 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6483431458473206 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.1666666716337204 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.01119150035083294 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6458994746208191 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.15390732884407043 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.17102396488189697 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.34117835760116577 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.12064921110868454 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 ,jet/accuracy_4_of_4‚0.013899499550461769 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.1666666716337204 8"Purity/0FRt*SRqqt2FBt/event_purity‚0.914798378944397 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.029256083071231842 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 particle/accuracy_3_of_5‚1 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9944142699241638 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.4724408686161041 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6203545331954956 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.16769495606422424 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.026027830317616463 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6094124913215637 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.19668813049793243 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.1292293667793274 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6717178821563721 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.03584723919630051 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.5714285969734192 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.5852727293968201 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.7154194712638855 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 epoch‚33 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9906582832336426 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6322909593582153 +jet/accuracy_2_of_2‚0.13656660914421082 *particle/sensitivity‚0.599128007888794 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6151785254478455 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 particle/accuracy_2_of_3‚1 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.4505877196788788 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 *jet/accuracy_1_of_3‚0.8986655473709106 +particle/specificity‚0.9017857313156128 'particle/f_score‚0.6118834614753723 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.12932856380939484 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.2385440617799759 _runtime‚22542.497893203 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.1380058228969574 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6332917213439941 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.14265620708465576 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9978824853897095 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7083333134651184 +jet/accuracy_1_of_1‚0.28387153148651123 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7083333134651184 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9921268224716187 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5842704176902771 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 *"Purity/2FRt*SRqqt2FBt/event_purity‚0.5 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6336050033569336 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.7129629850387573 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3420753479003906 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.05000000074505806 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.5972224473953247 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9957806468009949 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.15190160274505615 4 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9859579205513 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6569737195968628 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6634495854377747 -"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4375 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.14296269416809082 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.5902141332626343 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5823284983634949ý¯w+> 6ý{ +jet/accuracy_3_of_4‚0.06704101711511612 + Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9375 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9879744648933411 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 /particle/accuracy_4_of_4‚0.9434183239936829 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.12210109084844589 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9903962016105652 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.41203704476356506 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6406445503234863 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9862955808639526 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.11356587707996368 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.19230769574642181 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.1556880623102188 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9930450916290283 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8559363484382629 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.15554645657539368 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 9"Purity/1FRt1SRqqt*FBt/event_purity‚0.0353284552693367 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.11499999463558197 *jet/accuracy_1_of_2‚0.7217177152633667 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.6428571343421936 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7165573239326477 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8432815074920654 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9537036418914795 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7210375666618347 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.42824074625968933 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7481002807617188 6 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.491666704416275 particle/accuracy_1_of_5‚1 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.02125358395278454 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8077256083488464 *jet/accuracy_2_of_4‚0.7331549525260925 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6948369145393372 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.18954609334468842 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 particle/accuracy_5_of_5‚1 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5226060152053833 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9928704500198364 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6659384965896606 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9937976002693176 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3499999940395355 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.03428864851593971 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5973998308181763 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.006806667894124985 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.2552083134651184 % _timestamp‚1.7843123582797968e+09 *loss/val_total_loss‚10.048317675002371 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.22441962361335754 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9166666865348816 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7127863168716431 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9537036418914795 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6717851161956787 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5312874913215637 6 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.158451646566391 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.15705294907093048 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 /particle/accuracy_3_of_4‚0.9965986609458923 jet/accuracy_3_of_5‚1 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9978824853897095 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2705225646495819 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6113886833190918 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9922001957893372 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7535344958305359 +jet/accuracy_3_of_3‚0.06253387778997421 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.19230769574642181 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6388929486274719 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.43752846121788025 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.05000000074505806 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6338998079299927 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.15812669694423676 particle/accuracy_4_of_5‚1 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.17428450286388397 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9951749444007874 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7157029509544373 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.003683497430756688 7validation_average_jet_accuracy‚0.32861199975013733 6 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.709901750087738 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 + Purity/0FRt2SRqqt1FBt/FBt_purity‚0.9375 particle/accuracy_2_of_2‚1 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt2SRqqt0FBt/event_purity‚0.2004629373550415 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.6501700282096863 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.5751368403434753 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.029256083071231842 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8077256083488464 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9038461446762085 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9953287839889526 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.45218536257743835 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9928571581840515 8"Purity/1FRt0SRqqt0FBt/event_purity‚0.158451646566391 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.17305199801921844 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9942217469215393 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4545869827270508 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 (particle/accuracy‚0.8367860913276672 trainer/global_step‚13599 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5823284983634949 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.17066538333892822 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.3735355734825134 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.034623801708221436 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5685039162635803 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 particle/accuracy_1_of_3‚1 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9825929999351501 8"Purity/0FRt2SRqqt0FBt/event_purity‚0.450819194316864 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.36039090156555176 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9975056648254395 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.1403181254863739 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9801885485649109 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6483431458473206 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.1666666716337204 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.01119150035083294 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6458994746208191 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.15390732884407043 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.17102396488189697 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.34117835760116577 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.12064921110868454 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 ,jet/accuracy_4_of_4‚0.013899499550461769 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.1666666716337204 8"Purity/0FRt*SRqqt2FBt/event_purity‚0.914798378944397 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.029256083071231842 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 particle/accuracy_3_of_5‚1 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9944142699241638 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.4724408686161041 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6203545331954956 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.16769495606422424 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.026027830317616463 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6094124913215637 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.19668813049793243 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.1292293667793274 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6717178821563721 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.03584723919630051 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.5714285969734192 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.5852727293968201 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.7154194712638855 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 epoch‚33 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9906582832336426 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6322909593582153 +jet/accuracy_2_of_2‚0.13656660914421082 *particle/sensitivity‚0.599128007888794 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6151785254478455 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 particle/accuracy_2_of_3‚1 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.4505877196788788 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 *jet/accuracy_1_of_3‚0.8986655473709106 +particle/specificity‚0.9017857313156128 'particle/f_score‚0.6118834614753723 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.12932856380939484 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.2385440617799759 _runtime‚22542.497893203 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.1380058228969574 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6332917213439941 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.14265620708465576 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9978824853897095 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7083333134651184 +jet/accuracy_1_of_1‚0.28387153148651123 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7083333134651184 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9921268224716187 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5842704176902771 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 *"Purity/2FRt*SRqqt2FBt/event_purity‚0.5 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6336050033569336 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.7129629850387573 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3420753479003906 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.05000000074505806 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.5972224473953247 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9957806468009949 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.15190160274505615 4 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9859579205513 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6569737195968628 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6634495854377747 -"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4375 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.14296269416809082 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.5902141332626343 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5823284983634949 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_1‚1 particle/accuracy_3_of_3‚1 particle/accuracy_1_of_4‚1 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.6942709684371948 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 *jet/accuracy_1_of_4‚0.9611973762512207 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 jet/accuracy_2_of_5‚1 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.23362919688224792 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9983765482902527 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.036282215267419815 jet/accuracy_1_of_5‚1 jet/accuracy_5_of_5‚0 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7154194712638855 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7186991572380066 *jet/accuracy_2_of_3‚0.5192664265632629 * Purity/2FRt*SRqqt2FBt/FRt_purity‚0.625 jet/accuracy_4_of_5‚0 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9026336669921875 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 . Purity/2FRt0SRqqt1FBt/FRt_purity‚ 0.5859375 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8404529094696045 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9840746521949768 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 particle/accuracy_1_of_2‚1 particle/accuracy_2_of_4‚1 particle/accuracy_2_of_5‚1 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7183843851089478 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9719157218933105 _step‚1121 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172áBHG|ž¡6‚(:“ ¨äéÒÁ÷·÷ gpu.0.gpu‚87.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚326.236' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚65.24719999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 bVpE¢6‚(:º ¨äéÒ÷ˆÉø disk.sda2.in‚0$ disk./.usageGB‚565.9103012084961 disk.drbd2165.in‚0# disk.nvme0c0n1.in‚ 53241.4296875 cpu‚25.58560212725867 proc.cpu.threads‚17 disk.sda.out‚ 1365.9296875 disk.dm-0.in‚ 53241.4296875 disk.sda3.in‚ 108.921875* proc.memory.percent‚17.113566398620605# disk.dm-0.out‚151576.3759765625 network.sent‚ 2.6356958e+07 disk.dm-1.out‚ 1365.9296875 disk.sda1.in‚0 disk.rbd3.in‚0( disk.nvme0c0n1.out‚151576.3759765625! disk.nvme0n1.in‚ 53241.4296875& disk.nvme0n1.out‚151576.3759765625# proc.memory.rssMB‚ 2803.88671875 disk.sda.in‚ 108.921875% proc.memory.availableMB‚ 2802.9375 network.recv‚ 5.611944e+06* disk./.usagePercent‚37.978715144471806 disk.sda2.out‚0 disk.rbd1.in‚ 0.53564453125 disk.rbd1.out‚ 58.7490234375& disk.rbd0.in‚1.759218604441423e+13 disk.rbd0.out‚0 disk.sda1.out‚0$ memory_percent‚82.89222717285156 disk.dm-1.in‚ 108.921875 disk.sda3.out‚ 1365.9296875 disk.drbd2165.out‚0 disk.rbd3.out‚ 0.4951171875‰|Ÿb”£6‚(:‰ ·äéÒ»†÷ gpu.0.gpu‚99.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚338.878' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚67.7756 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0!æ¨2B¤6‚(:· ·äéÒ¨Öù) disk./.usagePercent‚37.98055609465053 disk.sda3.in‚ 108.921875# memory_percent‚82.8922986984253# proc.memory.rssMB‚ 2803.88671875 cpu‚25.585476201387994 network.sent‚ 2.6415332e+07 disk.rbd0.out‚0( disk.nvme0c0n1.out‚151587.8232421875 disk.rbd1.in‚ 0.53564453125 disk.rbd1.out‚ 58.9052734375 disk.sda2.out‚0 disk.drbd2165.in‚0 disk.rbd3.out‚ 0.4951171875 disk.nvme0n1.in‚ 53330.9375& disk.nvme0n1.out‚151587.8232421875 disk.dm-1.out‚ 1366.40234375 disk.dm-0.in‚ 53330.9375 disk.sda1.out‚0* proc.memory.percent‚17.113566398620605 network.recv‚ 5.616329e+06 disk.sda1.in‚0 disk.nvme0c0n1.in‚ 53330.9375 disk.sda.out‚ 1366.40234375) proc.memory.availableMB‚ 2802.92578125 disk.sda3.out‚ 1366.40234375 disk.sda2.in‚0 disk.sda.in‚ 108.921875$ disk./.usageGB‚565.9377326965332 disk.dm-1.in‚ 108.921875& disk.rbd0.in‚1.759218604441423e+13 disk.drbd2165.out‚0 proc.cpu.threads‚17# disk.dm-0.out‚151587.8232421875 disk.rbd3.in‚0p*(,ž¥6‚(:“ ÆäéÒôúö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚297.575' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚59.51499999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0"R‹A¦6‚(:¶ ÆäéÒé’ú disk.nvme0n1.in‚ 53331.140625& disk.nvme0n1.out‚151600.1669921875) disk./.usagePercent‚37.98110471879641$ disk./.usageGB‚565.9459075927734& disk.rbd0.in‚1.759218604441423e+13 disk.sda3.in‚ 108.921875 disk.drbd2165.in‚0# proc.memory.rssMB‚ 2803.88671875 disk.rbd1.out‚ 58.9052734375$ memory_percent‚82.89222717285156* proc.memory.percent‚17.113566398620605" disk.nvme0c0n1.in‚ 53331.140625 disk.sda2.in‚0% proc.memory.availableMB‚ 2802.9375 proc.cpu.threads‚17 disk.dm-1.out‚ 1366.6171875# disk.dm-0.out‚151600.1669921875 disk.sda1.in‚0 disk.drbd2165.out‚0 disk.sda.out‚ 1366.6171875 network.recv‚ 5.619882e+06 disk.rbd0.out‚0 disk.sda2.out‚0 disk.sda.in‚ 108.921875 network.sent‚ 2.6420407e+07 disk.dm-1.in‚ 108.921875 disk.sda3.out‚ 1366.6171875 disk.rbd3.in‚0 disk.rbd3.out‚ 0.4951171875( disk.nvme0c0n1.out‚151600.1669921875 cpu‚25.58534091357989 disk.rbd1.in‚ 0.53564453125 disk.dm-0.in‚ 53331.140625 disk.sda1.out‚0åóG•§6‚(:Š ÕäéÒñŠ€÷ gpu.0.gpu‚100.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚249.397' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚49.8794 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ôû“H¨6‚(:½ Õäé҃˃ú disk.drbd2165.out‚0 disk.rbd0.out‚0 disk.sda3.in‚ 108.921875 cpu‚25.585207184111052' disk.nvme0n1.out‚151619.12255859375$ disk./.usageGB‚565.9497871398926" disk.nvme0c0n1.in‚ 53331.359375 disk.nvme0n1.in‚ 53331.359375 disk.dm-0.in‚ 53331.359375 disk.sda1.out‚0# memory_percent‚82.8922986984253# proc.memory.rssMB‚ 2803.88671875 disk.sda2.in‚0 disk.rbd1.in‚ 0.53564453125 disk.sda3.out‚ 1367.7421875 network.recv‚ 5.623398e+06 disk.drbd2165.in‚0 disk.sda.out‚ 1367.7421875 disk.sda1.in‚0* proc.memory.percent‚17.113566398620605 disk.rbd1.out‚ 58.9677734375 disk.sda.in‚ 108.921875 disk.dm-1.in‚ 108.921875& disk.rbd0.in‚1.759218604441423e+13 proc.cpu.threads‚17 network.sent‚ 2.6425532e+07 disk.rbd3.out‚ 0.4951171875 disk.dm-1.out‚ 1367.7421875$ disk.dm-0.out‚151619.12255859375 disk.sda2.out‚0) disk./.usagePercent‚37.98136507845873 disk.rbd3.in‚0) disk.nvme0c0n1.out‚151619.12255859375) proc.memory.availableMB‚ 2802.92578125]æ9UŸ©6‚(:” ääéÒ§¥÷ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚283.559' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚56.711800000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0\ÙÍæª6‚(:» ääéÒ¨”îú disk.sda1.in‚0 disk.rbd1.in‚ 0.53564453125$ memory_percent‚82.89268016815186 disk.sda1.out‚0* proc.memory.percent‚17.113566398620605 disk.rbd1.out‚ 58.9677734375 disk.sda.in‚:¸UÁ` 108.921875 disk.dm-1.out‚ 1369.32421875& disk.rbd0.in‚1.759218604441423e+13( disk.nvme0c0n1.out‚151632.0791015625) proc.memory.availableMB‚ 2802.86328125# proc.memory.rssMB‚ 2803.88671875 disk.rbd0.out‚0 disk.dm-0.in‚ 53331.78125# disk.dm-0.out‚151632.0791015625 disk.drbd2165.in‚0 disk.rbd3.in‚0! disk.nvme0c0n1.in‚ 53331.78125 network.recv‚ 5.626924e+06 disk.nvme0n1.in‚ 53331.78125& disk.nvme0n1.out‚151632.0791015625 disk.sda2.in‚0 network.sent‚ 2.6430605e+07 disk.rbd3.out‚ 0.4951171875 cpu‚25.585060387924884 disk.sda2.out‚0$ disk./.usageGB‚565.9566421508789 disk.dm-1.in‚ 108.921875 disk.sda3.in‚ 108.921875 disk.drbd2165.out‚0) disk./.usagePercent‚37.98182512399776 disk.sda.out‚ 1369.32421875 disk.sda3.out‚ 1369.32421875 proc.cpu.threads‚17ELE(Ÿ«6‚(:” óäéÒ¯Èêö gpu.0.gpu‚90.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚298.392' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚59.678399999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0^#ÝG¬6‚(:¼ óäéÒŠ¢ù disk.sda.in‚ 108.921875 disk.dm-1.in‚ 108.921875) proc.memory.availableMB‚ 2802.83203125 network.recv‚ 5.630566e+06 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.rbd3.in‚0 cpu‚25.58494187073525 disk.sda1.in‚0 network.sent‚ 2.643606e+07$ disk./.usageGB‚565.9615859985352 disk.dm-1.out‚ 1369.9765625' disk.nvme0c0n1.out‚151651.052734375 proc.cpu.threads‚17 disk.rbd0.out‚0 disk.sda1.out‚0* proc.memory.percent‚17.113566398620605 disk.sda.out‚ 1369.9765625& disk.rbd0.in‚1.759218604441423e+13% disk.nvme0n1.out‚151651.052734375 disk.sda2.in‚0 disk.rbd1.in‚ 0.53564453125# proc.memory.rssMB‚ 2803.88671875 disk.sda3.in‚ 108.921875 disk.sda3.out‚ 1369.9765625 disk.sda2.out‚0 disk.rbd1.out‚ 59.0302734375" disk.dm-0.out‚151651.052734375 disk.rbd3.out‚ 0.4951171875" disk.nvme0n1.in‚53397.05078125$ memory_percent‚82.89287090301514) disk./.usagePercent‚37.98215690976214 disk.dm-0.in‚53397.05078125$ disk.nvme0c0n1.in‚53397.05078125nž4­6/ _wandbruntime‚22620  _runtime‚22620—€êË®6Å ' lr-AdamW/pg2‚0.00016428191026982583 trainer/global_step‚13649 % _timestamp‚1.7843124367050116e+09 _runtime‚22620.92220071 _step‚1122 ' lr-AdamW/pg1‚0.00016428191026982583¹¶“²Ð¯6Ê trainer/global_step‚13649 % _timestamp‚1.7843124367050116e+09 _runtime‚22620.92220071 _step‚1122 ' lr-AdamW/pg1‚0.00016428191026982583 ' lr-AdamW/pg2‚0.00016428191026982583â®84°6/ _wandbruntime‚22620  _runtime‚22620–8ä“(±6”P [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3937791003892224 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3937792375455232 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3924948810883072 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚35019366 _step‚1123 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 118241997 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 116504469 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚250 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1059517032583168 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16558704139264 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 116504469 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3924948837056512 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3936075251370224 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚3924948837056512 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1059517400674304 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚3936073881341934 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12843538398720 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚38681123 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚3923253689643008 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1076076137252864 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚35019361 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12843538398720 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16558736578560 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚73700489 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 118241997 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 400530432 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128eÑô(²6™P MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12842193009152 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚3937791003892224 trainer/global_step‚13649 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚245 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚38680878 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1076075736722432 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 118232024 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚3924948810883072 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 116504475 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚73700239 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12821561727216 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16558736578560 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚73700489 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 118241997 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 400530432 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3937791003892224 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3937792375455232 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3924948810883072 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚35019366 _step‚1123 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 118241997 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 116504469 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚250 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1059517032583168 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16558704139264 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 116504469 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3924948837056512 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3936075251370224 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚3924948837056512 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1059517400674304 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚3936073881341934 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12843538398720 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚38681123 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚3923253689643008 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1076076137252864 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚35019361 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12843538398720 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 JDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚22622.307622809 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 118240502 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3925226811613184 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 400523264 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 116512681 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12844463441408 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚32432128 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚3936352686586356 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1059594887081984 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 116512675 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 234763170 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3925226837786624 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3938071301228032 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 234753177 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚38683640 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12843118044672 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚38683875 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16559905883136 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3936354056614758 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1076154825397248 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1076154424873984 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚35021909 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12822485193574 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚73705544 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 234763170 _step‚1124 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3938069929657856 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12844463441408 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 118240502 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_e+ÕÇ'nd/reserved_bytes.small_pool.allocated‚ 1719664640 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 234753177 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12821140593140 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚3925226837786624 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3938071301228032 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 G5n)'µ6£N EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚22622.307622809 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 118240502 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3925226811613184 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 400523264 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 116512681 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12844463441408 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚32432128 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚3936352686586356 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1059594887081984 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 116512675 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 234763170 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3925226837786624 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3938071301228032 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 234753177 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚38683640 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12843118044672 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚38683875 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16559905883136 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3936354056614758 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1076154825397248 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1076154424873984 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚35021909 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12822485193574 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚73705544 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 234763170 _step‚1124 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3938069929657856 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12844463441408 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 118240502 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 234753177 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12821140593140 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚3925226837786624 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3938071301228032 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚240 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1059594518990848 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 116512675 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736äííOã4¶6/ _wandbruntime‚22622  _runtime‚22622q½Ú·6Œ _step‚1125 0loss/FRt1/assignment_loss‚0.9846138242155986 3loss/SRqqt1/assignment_loss‚0.08390169608284327 4loss/FBt1/assignment_loss‚3.1684411295809936e-07 /loss/FRt2/detection_loss‚1.3054387231890812 1loss/SRqqt1/detection_loss‚0.4872942714835611 .loss/FBt1/detection_loss‚0.208501115192408 epoch‚34 3loss/SRqqt2/assignment_loss‚0.13180690722950233 4loss/FBt2/assignment_loss‚0.00038101622270666954 0loss/FRt1/detection_loss‚0.44906121411391003 0loss/FBt2/detection_loss‚0.17254360135615526 trainer/global_step‚13649 $ _timestamp‚1.784312438177382e+09 _runtime‚22622.394313119 0loss/FRt2/assignment_loss‚0.2334006043969008 0loss/SRqqt2/detection_loss‚0.490789579908058 %loss/total_loss‚4.547732870234838˜[)¢¸6‘ 0loss/SRqqt2/detection_loss‚0.490789579908058 %loss/total_loss‚4.547732870234838 _step‚1125 0loss/FRt1/assignment_loss‚0.9846138242155986 3loss/SRqqt1/assignment_loss‚0.08390169608284327 4loss/FBt1/assignment_loss‚3.1684411295809936e-07 /loss/FRt2/detection_loss‚1.3054387231890812 1loss/SRqqt1/detection_loss‚0.4872942714835611 .loss/FBt1/detection_loss‚0.208501115192408 epoch‚34 3loss/SRqqt2/assignment_loss‚0.13180690722950233 4loss/FBt2/assignment_loss‚0.00038101622270666954 0loss/FRt1/detection_loss‚0.44906121411391003 0loss/FBt2/detection_loss‚0.17254360135615526 trainer/global_step‚13649 $ _timestamp‚1.784312438177382e+09 _runtime‚22622.394313119 0loss/FRt2/assignment_loss‚0.2334006043969008åùÐk”¹6‚(:‰ ‚åéÒËŠùö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚290.516' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚58.1032 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ó9©Oº6‚(:Ä ‚åéÒæ›êù network.recv‚ 5.641287e+06 disk.rbd3.out‚ 0.4951171875 disk.dm-1.out‚ 1372.6953125$ memory_percent‚82.89453983306885) proc.memory.availableMB‚ 2802.55859375 network.sent‚ 2.6444666e+07& disk.rbd0.in‚1.759218604441423e+13 disk.sda1.in‚0 disk.rbd1.in‚ 0.53564453125 cpu‚25.584817761956675$ disk./.usageGB‚565.9724922180176 disk.dm-0.in‚53397.47265625 disk.sda3.out‚ 1372.6953125 disk.rbd3.in‚0) disk.nvme0c0n1.out‚151665.11181640625 disk.sda2.in‚0 disk.drbd2165.out‚0) disk./.usagePercent‚37.98288883530253 disk.sda3.in‚ 108.921875$ disk.nvme0c0n1.in‚53397.47265625 disk.rbd1.out‚ 59.0302734375' disk.nvme0n1.out‚151665.11181640625 disk.sda.in‚ 108.921875 disk.sda.out‚ 1372.6953125* proc.memory.percent‚17.113566398620605$ disk.dm-0.out‚151665.11181640625 disk.sda1.out‚0 disk.rbd0.out‚0 disk.drbd2165.in‚0" disk.nvme0n1.in‚53397.47265625 disk.sda2.out‚0 disk.dm-1.in‚ 108.921875# proc.memory.rssMB‚ 2803.88671875 proc.cpu.threads‚17ºë¢ÄŸ»6‚(:” ‘åéÒÉÅö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚263.406' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚52.681200000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0^Š¬ìJ¼6‚(:¿ ‘åéÒ²âÊù network.sent‚ 2.6507437e+07 disk.drbd2165.out‚0 disk.sda2.in‚0# proc.memory.rssMB‚ 2803.88671875$ disk./.usageGB‚565.9738426208496 disk.rbd0.out‚0$ disk.nvme0c0n1.in‚53397.69140625% disk.nvme0n1.out‚151688.935546875 disk.rbd1.out‚ 59.1083984375* proc.memory.percent‚17.113566398620605 proc.cpu.threads‚17" disk.dm-0.out‚151688.935546875 disk.rbd3.in‚0" disk.nvme0n1.in‚53397.69140625 disk.sda2.out‚0 disk.rbd1.in‚ 0.53564453125 network.recv‚ 5.646332e+06) disk./.usagePercent‚37.98297946196965 disk.dm-1.out‚ 1375.08984375& disk.rbd0.in‚1.759218604441423e+13 cpu‚25.5846850632762 disk.sda1.in‚0 disk.sda3.in‚ 108.921875 disk.sda3.out‚ 1375.08984375 disk.drbd2165.in‚0' disk.nvme0c0n1.out‚151688.935546875$ memory_percent‚82.89620876312256) proc.memory.availableMB‚ 2802.28515625 disk.dm-1.in‚ 108.921875 disk.sda.in‚ 108.921875 disk.sda.out‚ 1375.08984375 disk.rbd3.out‚ 0.4951171875 disk.dm-0.in‚53397.69140625 disk.sda1.out‚0ÿ`xŸŸ½6‚(:”  åéÒ­¿Ïö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚38.0 gpu.0.powerWatts‚110.415' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.083000000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0)4ÔM¾6‚(:  åéÒä¢Äø disk.sda3.in‚ 108.921875 network.sent‚ 2.651307e+07$ disk./.usageGB‚565.9806442260742 disk.drbd2165.in‚0 disk.rbd3.out‚ 0.4951171875 disk.sda2.out‚0 disk.sda1.in‚0* proc.memory.percent‚17.113566398620605 proc.cpu.threads‚17 network.recv‚ 5.649672e+06 disk.dm-0.in‚53397.89453125# proc.memory.rssMB‚ 2803.88671875' disk.nvme0n1.out‚151696.21630859375 disk.sda3.out‚ 1375.9296875 disk.rbd1.out‚ 59.1083984375$ memory_percent‚82.89620876312256" disk.nvme0n1.in‚53397.89453125 disk.sda2.in‚0 disk.sda.in‚ 108.921875 disk.rbd0.out‚0) proc.memory.availableMB‚ 2802.28515625 cpu‚25.584554408800695) disk.nvme0c0n1.out‚151696.21630859375 disk.rbd1.in‚ 0.53564453125$ disk.dm-0.out‚151696.21630859375$ disk.nvme0c0n1.in‚53397.89453125 disk.dm-1.in‚ 108.921875 disk.dm-1.out‚ 1375.9296875& disk.rbd0.in‚1.759218604441423e+13 disk.sda1.out‚0( disk./.usagePercent‚37.9834359234032 disk.drbd2165.out‚0 disk.rbd3.in‚0 disk.sda.out‚ 1375.9296875óA`§”¿6‚(:‰ ¯åéÒ͉Ëø gpu.0.gpu‚90.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚273.366' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.6732 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0}µ?OÀ6‚(:Ä ¯åéÒá²Øù disk.sda2.out‚0 disk.dm-0.in‚53514.40234375 disk.sda3.in‚ 108.921875# proc.memory.rssMB‚ 2803.88671875& disk.rbd0.in‚1.759218604441423e+13 disk.sda1.out‚0 disk.sda.in‚ 108.921875 disk.sda.out‚ 1376.32421875$ disk.nvme0c0n1.in‚53514.40234375" disk.nvme0n1.in‚53514.40234375$ memory_percent‚82.89494514465332* proc.memory.percent‚17.113566398620605 cpu‚25.58441872854966$ disk.dm-0.out‚151712.38427734375 disk.sda1.in‚0 network.sent‚ 2.6518208e+07 disk.sda2.in‚0 disk.rbd0.out‚0 disk.drbd2165.out‚0 disk.rbd3.in‚0' disk.nvme0n1.out‚151712.38427734375( proc.memory.availableMB‚ 2802.4921875) disk./.usagePercent‚37.98393488208743 disk.rbd1.in‚ 0.53564453125 disk.sda3.out‚ 1376.32421875 disk.drbd2165.in‚0 disk.rbd3.out‚ 0.4951171875 disk.dm-1.in‚ 108.921875 proc.cpu.threads‚17 network.recv‚ 5.65277e+06$ disk./.usageGB‚565.9880790710449 disk.rbd1.out‚ 59.1708984375 disk.dm-1.out‚ 1376.32421875) disk.nvme0c0n1.out‚151712.38427734375¸“Á6‚(:ˆ ¾åéÒ¡™Ýö gpu.0.gpu‚76.0 gpu.0.memory‚23, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚335.265' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚67.053 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0J ÎàEÂ6‚(:º ¾åéÒΔãø disk.rbd0.out‚0 disk.dm-0.in‚53514.62109375" disk.nvme0n1.in‚53514.62109375 disk.rbd1.out‚ 59.1708984375 network.recv‚ 5.656216e+06 disk.dm-1.out‚ 1376.6484375! disk.dm-0.out‚151720.47265625 disk.drbd2165.in‚0* proc.memory.percent‚17.113566398620605$ disk./.usageGB‚565.9918632507324 disk.rbd3.in‚0$ disk.nvme0c0n1.in‚53514.62109375& disk.nvme0c0n1.out‚151720.47265625$ memory_percent‚82.89616107940674) proc.memory.availableMB‚ 2802.29296875( disk./.usagePercent‚37.9841888415614 disk.sda.out‚ 1376.6484375 disk.dm-1.in‚ 108.921875& disk.rbd0.in‚1.759218604441423e+13 disk.sda2.in‚0 disk.sda.in‚ 108.921875 disk.sda3.in‚ 108.921875 disk.rbd1.in‚ 0.53564453125 network.sent‚ 2.6522764e+07 disk.sda3.out‚ 1376.6484375 disk.drbd2165.out‚0 disk.sda2.out‚0# proc.memory.rssMB‚ 2803.88671875 disk.sda1.in‚0 disk.rbd3.out‚ 0.4951171875$ disk.nvme0n1.out‚151720.47265625 cpu‚25.584299137213392 proc.cpu.threads‚17 disk.sda1.out‚0ÇÙUt4Ã6/ _wandbruntime‚22699  _runtime‚22699*û(ËÄ6Å _runtime‚22699.51584408 _step‚1126 ' lr-AdamW/pg1‚0.00016192839202409885 ' lr-AdamW/pg2‚0.00016192839202409885 trainer/global_step‚13699 % _timestamp‚1.7843125152986343e+09£êß“ÐÅ6Ê ' lr-AdamW/pg1‚0.00016192839202409885 ' lr-AdamW/pg2‚0.00016192839202409885 trainer/global_step‚13699 % _timestamp‚1.7843125152986343e+09 _runtime‚22699.51584408 _step‚1126æšèr 4Æ6/ _wandbruntime‚22699  _runtime‚22699ßs!(Ç6”P M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 118657776 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚38819842 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1080010480941056 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚35146516 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3950015577392132 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 116914775 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚3951737356328448 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 116914769 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3938848847388672 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3951737356328448 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12888508939776 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 118667749 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 235572545 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚3938848847388672 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚73966358 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12888508939776 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1063391352967168 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚3937147778551808 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 trainer/global_step‚13699 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16618759882752 % _timestamp‚1.7843125152996984e+09 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 116914769 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚35146511 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1063391721058304 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 118667749 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 _step‚1127 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚73966086 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 118657776 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627¤§éý0È6™P QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 116914769 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚35146511 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1063391721058304 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 118667749 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 _step‚1127 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚73966086 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 118657776 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 118657776 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚38819842 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1080010480941056 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚35146516 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3950015577392132 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 116914775 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚3951737356328448 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 116914769 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3938848847388672 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3951737356328448 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12888508939776 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 118667749 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 235572545 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚3938848847388672 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚73966358 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12888508939776 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1063391352967168 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚3937147778551808 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 trainer/global_step‚13699 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16618759882752 % _timestamp‚1.7843125152996984e+09ç[ð÷ú4É6/ _wandbruntime‚22700  _runtime‚22700Š7ÿ{$'Ê6žN E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12889433975296 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3937425634902016 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3952016282094080 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 235599222 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 235589229 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16619921563648 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3937425660329984 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚73971396 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1063469207465984 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 trainer/global_step‚13699 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3950294382636666 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 QDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 235599222 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚35149059 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 116922975 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 _step‚1128 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 116922981 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 116922981 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12890779372032 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12890779372032 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚259 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 235589229 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 116922975 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16619956092928 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚38822342 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 118666254 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12889433975296 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚3950293012608264 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12867377706248 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1080088760938496 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 _runtime‚22700.903453138÷<)'Ë6£N ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚3950293012608264 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12867377706248 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1080088760938496 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 _runtime‚22700.903453138 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12889433975296 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3937425634902016 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3952016282094080 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 235599222 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 235589229 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16619921563648 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3937425660329984 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚73971396 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1063469207465984 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 trainer/global_step‚13699 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3950294382636666 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 QDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 235599222 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚35149059 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 116922975 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 _step‚1128 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 116922981 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 116922981 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12890779372032 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12890779372032 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚259 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 235589229 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 116922975 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16619956092928 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚38822342 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 118666254 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12889433975296 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0èÁЮ4Ì6/ _wandbruntime‚22700  _runtime‚22700j‡Í6Ž 2loss/SRqqt2/detection_loss‚0.44868275234928384 _step‚1129 2loss/SRqqt2/assignment_loss‚0.1387573916204778 /loss/FRt1/detection_loss‚0.4063989247017718 /loss/FRt2/detection_loss‚1.2476358776118779 /loss/FBt1/detection_loss‚0.3325204610300716 trainer/global_step‚13699 0loss/FRt1/assignment_loss‚0.9753985936202968 2loss/SRqqt1/detection_loss‚0.48691284940310103 %loss/total_loss‚4.534906456118749 1loss/FRt2/assignment_loss‚0.16272810582191713 3loss/SRqqt1/assignment_loss‚0.15647044485606604 2loss/FBt1/assignment_loss‚0.003946628609106995 4loss/FBt2/assignment_loss‚0.00025069200260056896 0loss/FBt2/detection_loss‚0.17520373449217783 epoch‚34 % _timestamp‚1.7843125167720463e+09 _runtime‚22700.989006836ZÅÞ…Î6“ _step‚1129 2loss/SRqqt2/assignment_loss‚0.1387573916204778 /loss/FRt1/detection_loss‚0.4063989247017718 /loss/FRt2/detection_loss‚1.2476358776118779 /loss/FBt1/detection_loss‚0.3325204610300716 trainer/global_step‚13699 0loss/FRt1/assignment_loss‚0.9753985936202968 2loss/SRqqt1/detection_loss‚0.48691284940310103 %loss/total_loss‚4.534906456118749 1loss/FRt2/assignment_loss‚0.16272810582191713 3loss/SRqqt1/assignment_loss‚0.15647044485606604 2loss/FBt1/assignment_loss‚0.003946628609106995 4loss/FBt2/assignment_loss‚0.00025069200260056896 0loss/FBt2/detection_loss‚0.17520373449217783 epoch‚34 % _timestamp‚1.7843125167720463e+09 _runtime‚22700.989006836 2loss/SRqqt2/detection_loss‚0.44868275234928384é*¾å Ï6‚(:’ ÍåéÒ©‘áù gpu.0.gpu‚98.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚283.03' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚56.60599999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ¬Z?GÐ6‚(:¼ ÍåéÒÅÕû& disk.nvme0n1.out‚151730.3330078125 disk.dm-0.in‚53515.04296875$ memory_percent‚82.89785385131836 cpu‚25.584148747775554( disk.nvme0c0n1.out‚151730.3330078125 disk.sda.out‚ 1377.671875 disk.drbd2165.out‚0 disk.sda.in‚ 108.921875 disk.dm-1.in‚ 108.921875# disk.dm-0.out‚151730.3330078125 proc.cpu.threads‚17 disk.sda2.in‚0 disk.rbd3.out‚ 0.4951171875* proc.memory.percent‚17.113566398620605" disk.nvme0n1.in‚53515.04296875 disk.sda2.out‚0 disk.dm-1.out‚ 1377.671875 network.sent‚ 2.6528385e+07 disk.rbd3.in‚0 disk.rbd1.in‚ 0.53564453125 disk.rbd1.out‚ 59.2490234375' proc.memory.availableMB‚ 2802.015625 network.recv‚ 5.659561e+06$ disk./.usageGB‚566.0031967163086 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda3.in‚ 108.921875& disk.rbd0.in‚1.759218604441423e+13 disk.rbd0.out‚0# proc.memory.rssMB‚ 2803.88671875) disk./.usagePercent‚37.98494943994563 disk.sda3.out‚ 1377.671875 disk.drbd2165.in‚0$ disk.nvme0c0n1.in‚53515.04296875ò`•”Ñ6‚(:‰ ÜåéÒáÃéö gpu.0.gpu‚91.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚323.288' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚64.6576 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0»¦?jÒ6‚(:Å ÜåéÒ£ˆÖù disk.drbd2165.out‚0) disk.nvme0c0n1.out‚151748.06103515625 cpu‚25.58403199598552 disk.rbd3.in‚0" disk.nvme0n1.in‚53515.24609375 disk.sda.out‚ 1379.69140625* proc.memory.percent‚17.113566398620605 network.sent‚ 2.6591302e+07' disk.nvme0n1.out‚151748.06103515625$ memory_percent‚82.89744853973389# disk./.usageGB‚566.013786315918$ disk.nvme0c0n1.in‚53515.24609375 disk.rbd0.out‚0 disk.sda2.out‚0 disk.rbd1.in‚ 0.53564453125 disk.rbd1.out‚ 59.2490234375 disk.dm-0.in‚53515.24609375 disk.sda1.in‚0 proc.cpu.threadsŠü·åæ‚17 network.recv‚ 5.664192e+06 disk.sda1.out‚0 disk.rbd3.out‚ 0.4951171875 disk.sda2.in‚0 disk.dm-1.in‚ 108.921875 disk.dm-1.out‚ 1379.69140625$ disk.dm-0.out‚151748.06103515625# proc.memory.rssMB‚ 2803.88671875) disk./.usagePercent‚37.98566011686068 disk.sda3.in‚ 108.921875 disk.drbd2165.in‚0 disk.sda.in‚ 108.921875& disk.rbd0.in‚1.759218604441423e+13) proc.memory.availableMB‚ 2802.08203125 disk.sda3.out‚ 1379.69140625¦6›m”Ó6‚(:‰ ëåéÒ„Ýéö gpu.0.gpu‚89.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚270.099' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.0198 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ƨùc=Ô6‚(:² ëåéÒ ¯²ù proc.cpu.threads‚17 disk.sda1.in‚0 disk.drbd2165.out‚0# proc.memory.rssMB‚ 2803.88671875 network.recv‚ 5.667678e+06 disk.dm-1.in‚ 108.921875 disk.rbd0.out‚0 disk.dm-0.in‚ 53574.71875 disk.rbd3.in‚0$ memory_percent‚82.89813995361328 disk.rbd1.in‚ 0.53564453125 disk.drbd2165.in‚0 disk.nvme0n1.in‚ 53574.71875& proc.memory.availableMB‚ 2801.96875) disk./.usagePercent‚37.98604105607163 disk.sda1.out‚0 disk.sda3.in‚ 108.921875 disk.rbd3.out‚ 0.4951171875! disk.nvme0c0n1.in‚ 53574.71875 cpu‚25.583889419457652& disk.nvme0c0n1.out‚151760.19921875 disk.sda2.in‚0 disk.rbd1.out‚ 59.3115234375 network.sent‚ 2.6596919e+07 disk.sda.out‚ 1380.74609375 disk.dm-1.out‚ 1380.74609375 disk.sda.in‚ 108.921875& disk.rbd0.in‚1.759218604441423e+13 disk.sda3.out‚ 1380.74609375 disk.sda2.out‚0* proc.memory.percent‚17.113566398620605! disk.dm-0.out‚151760.19921875$ disk./.usageGB‚566.0194625854492$ disk.nvme0n1.out‚151760.19921875 m…”Õ6‚(:‰ úåéÒж‘÷ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚291.723' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚58.3446 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0’lIULÖ6‚(:Á úåéÒÞüù$ disk.nvme0c0n1.in‚53575.15234375* proc.memory.percent‚17.113566398620605' disk./.usagePercent‚37.986484973136$ disk./.usageGB‚566.0260772705078" disk.nvme0n1.in‚53575.15234375$ memory_percent‚82.89835453033447# proc.memory.rssMB‚ 2803.88671875 cpu‚25.58374703152789 disk.sda.in‚ 108.921875 disk.drbd2165.out‚0' disk.nvme0n1.out‚151777.68212890625 network.recv‚ 5.670971e+06 disk.sda3.out‚ 1381.0390625 disk.rbd3.in‚0) proc.memory.availableMB‚ 2801.93359375$ disk.dm-0.out‚151777.68212890625 disk.sda1.out‚0 disk.rbd3.out‚ 0.4951171875 disk.sda2.in‚0 disk.rbd1.in‚ 0.53564453125 disk.dm-1.in‚ 108.921875& disk.rbd0.in‚1.759218604441423e+13 disk.dm-0.in‚53575.15234375 disk.rbd1.out‚ 59.3115234375 network.sent‚ 2.6602009e+07 disk.sda.out‚ 1381.0390625 disk.dm-1.out‚ 1381.0390625 disk.rbd0.out‚0 disk.sda3.in‚ 108.921875 disk.drbd2165.in‚0 disk.sda2.out‚0 disk.sda1.in‚0) disk.nvme0c0n1.out‚151777.68212890625 proc.cpu.threads‚178ÕX|Ÿ×6‚(:” ‰æéÒæŸì÷ gpu.0.gpu‚90.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚284.911' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚56.982200000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¡³CPØ6‚(:Å ‰æéÒðý‚ù disk.dm-1.in‚ 108.921875# disk.dm-0.out‚151789.0693359375$ disk.nvme0c0n1.in‚53575.35546875" disk.nvme0n1.in‚53575.35546875 disk.dm-1.out‚ 1383.48828125( disk.nvme0c0n1.out‚151789.0693359375 proc.cpu.threads‚17 disk.drbd2165.in‚0 disk.rbd3.in‚0 disk.rbd3.out‚ 0.4951171875 disk.sda1.in‚0 disk.sda1.out‚0& disk.rbd0.in‚1.759218604441423e+13 disk.dm-0.in‚53575.35546875 disk.sda3.in‚ 108.921875& disk.nvme0n1.out‚151789.0693359375 disk.sda2.in‚0 disk.rbd1.out‚ 59.3896484375$ memory_percent‚82.89682865142822* proc.memory.percent‚17.113566398620605 network.recv‚ 5.674231e+06* disk./.usagePercent‚37.986576623833265 disk.drbd2165.out‚0) proc.memory.availableMB‚ 2802.18359375# proc.memory.rssMB‚ 2803.88671875 network.sent‚ 2.6607122e+07 disk.rbd0.out‚0 cpu‚25.583626311473612$ disk./.usageGB‚566.0274429321289 disk.sda.out‚ 1383.48828125 disk.sda3.out‚ 1383.48828125 disk.sda.in‚ 108.921875 disk.sda2.out‚0 disk.rbd1.in‚ 0.53564453125cزî4Ù6/ _wandbruntime‚22778  _runtime‚22778uç aÌÚ6Æ ' lr-AdamW/pg2‚0.00015958669148696027 trainer/global_step‚13749 % _timestamp‚1.7843125944827385e+09 _runtime‚22778.699917384 _step‚1130 ' lr-AdamW/pg1‚0.000159586691486960275t|ÑÛ6Ë _runtime‚22778.699917384 _step‚1130 ' lr-AdamW/pg1‚0.00015958669148696027 ' lr-AdamW/pg2‚0.00015958669148696027 trainer/global_step‚13749 % _timestamp‚1.7843125944827385e+09ê>镺4Ü6/ _wandbruntime‚22778  _runtime‚22778˜@Ð(Ý6“P M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 119083528 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1067265673351168 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16678410646528 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12934824870400 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚38957708 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 117325075 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚261 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚74231374 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 % _timestamp‚1.7843125944838214e+09 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12934824870400 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚3951041842032640 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 119083528 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 117325069 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3965683708764672 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12936170259968 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚3952748883894272 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚3952748910067712 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 117325075 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 _step‚1131 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚74231113 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚256 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3952748910067712 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12914035953432 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 236408597 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3965685080327680 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12936170259968 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3952748883894272 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1067266041442304 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 trainer/global_step‚13749Ëð¢–(Þ6˜P MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12914035953432 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 236408597 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3965685080327680 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12936170259968 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚3952748883894272 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1067266041442304 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 trainer/global_step‚13749 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 119083528 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1067265673351168 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16678410646528 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12934824870400 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚38957708 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 117325075 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚261 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚74231374 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 % _timestamp‚1.7843125944838214e+09 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12934824870400 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚3951041842032640 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 119083528 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 117325069 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3965683708764672 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12936170259968 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚3952748883894272 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚3952748910067712 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 117325075 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 _step‚1131 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚74231113 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚256 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3952748910067712 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352ë~ŒO4ß6/ _wandbruntime‚22780  _runtime‚22780ü÷2ð¡à6žN ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 117333275 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 119101993 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3965964006100480 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚74236367 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12937095302656 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 117333275 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚3953026884624384 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3953026910797824 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 236435274 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 117333281 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 % _timestamp‚1.7843125958674562e+09 _step‚1132 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3953026884624384 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1084023163120640 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1084022760500224 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚74236620 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3951319749238784 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚253 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3951319723810816 trainer/global_step‚13749 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 236425281 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1067343159758848 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3964234708658574 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12935749905920 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚35276209 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12935749905920 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 236435274 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3965962634530304 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 119101993 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12937095302656 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3965964006100480 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 117333281 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12914959419790 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚248 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1067343527849984 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352êÉ~ê)'á6£N KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚248 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1067343527849984 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 117333275 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 119101993 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3965964006100480 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚74236367 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12937095302656 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 117333275 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚3953026884624384 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3953026910797824 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 236435274 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 117333281 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 % _timestamp‚1.7843125958674562e+09 _step‚1132 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3953026884624384 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1084023163120640 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1084022760500224 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚74236620 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3951319749238784 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚253 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3951319723810816 trainer/global_step‚13749 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 236425281 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1067343159758848 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3964234708658574 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12935749905920 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚35276209 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12935749905920 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 236435274 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚3965962634530304 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 119101993 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12937095302656 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3965964006100480 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 117333281 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12914959419790ìŒÎõÂ4â6/ _wandbruntime‚22780  _runtime‚227800/wÕã6Ž 1loss/FRt2/assignment_loss‚0.09268885718123747 3loss/SRqqt2/assignment_loss‚0.16282877589614572 2loss/SRqqt1/detection_loss‚0.44035240421908384 1loss/SRqqt2/detection_loss‚0.5324027338433716 _step‚1133 2loss/SRqqt1/assignment_loss‚0.0842099714980168 0loss/FRt1/detection_loss‚0.47125704785521716 /loss/FRt2/detection_loss‚1.2894188728518345 &loss/total_loss‚4.6316614098835505 /loss/FRt1/assignment_loss‚1.026203798595816 3loss/FBt2/assignment_loss‚4.188096638596845e-06 /loss/FBt1/detection_loss‚0.3352417004824632 /loss/FBt2/detection_loss‚0.1970378621477089 epoch‚34 % _timestamp‚1.7843125959546413e+09 4loss/FBt1/assignment_loss‚1.5197216016989683e-05 trainer/global_step‚13749 _runtime‚22780.171539271Y’÷qä6“ 0loss/FRt1/detection_loss‚0.47125704785521716 /loss/FRt2/detection_loss‚1.2894188728518345 &loss/total_loss‚4.6316614098835505 /loss/FRt1/assignment_loss‚1.026203798595816 3loss/FBt2/assignment_loss‚4.188096638596845e-06 /loss/FBt1/detection_loss‚0.3352417004824632 /loss/FBt2/detection_loss‚0.1970378621477089 epoch‚34 % _timestamp‚1.7843125959546413e+09 4loss/FBt1/assignment_loss‚1.5197216016989683e-05 trainer/global_step‚13749 _runtime‚22780.171539271 1loss/FRt2/assignment_loss‚0.09268885718123747 3loss/SRqqt2/assignment_loss‚0.16282877589614572 2loss/SRqqt1/detection_loss‚0.44035240421908384 1loss/SRqqt2/detection_loss‚0.5324027338433716 _step‚1133 2loss/SRqqt1/assignment_loss‚0.0842099714980168ímú›”å6‚(:‰ ˜æéÒöú‚÷ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚40.0 gpu.0.powerWatts‚293.271' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚58.6542 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0òO™¦Aæ6‚(:¶ ˜æéÒ­èÉù network.sent‚ 2.6611506e+07 network.recv‚ 5.677175e+06 disk.dm-1.out‚ 1384.0625 disk.dm-0.in‚53575.77734375' proc.memory.availableMB‚ 2801.953125 disk.rbd1.out‚ 59.3896484375 proc.cpu.threads‚17$ disk.nvme0c0n1.in‚53575.77734375 disk.sda3.out‚ 1384.0625 disk.rbd1.in‚ 0.53564453125& disk.rbd0.in‚1.759218604441423e+13 disk.rbd0.out‚0# disk.dm-0.out‚151799.2451171875 disk.drbd2165.out‚0 disk.rbd3.in‚0$ memory_percent‚82.89823532104492* proc.memory.percent‚17.113566398620605 disk.sda1.in‚0 disk.sda3.in‚ 108.921875( disk.nvme0c0n1.out‚151799.2451171875 disk.sda2.out‚0 disk.dm-1.in‚ 108.921875 cpu‚25.583482322236005 disk.sda2.in‚0 disk.sda.in‚ 108.921875 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.rbd3.out‚ 0.4951171875) disk./.usagePercent‚37.98716492914694$ disk./.usageGB‚566.0362091064453" disk.nvme0n1.in‚53575.77734375# proc.memory.rssMB‚ 2803.88671875& disk.nvme0n1.out‚151799.2451171875 disk.sda.out‚ 1384.0625Bb¤U ç6‚(:• §æéҞتø gpu.0.gpu‚100.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚299.744' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚59.948800000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚03xMè6‚(: §æéÒ–ÏÆù$ disk./.usageGB‚566.0406646728516 disk.sda2.out‚0 disk.sda1.out‚0 disk.sda3.in‚ 108.921875 disk.sda3.out‚ 1384.3515625& disk.rbd0.in‚1.759218604441423e+13* disk./.usagePercent‚37.987463945946935( disk.nvme0c0n1.out‚151811.6279296875) proc.memory.availableMB‚ 2802.40234375# proc.memory.rssMB‚ 2803.88671875 cpu‚25.583372743530546 disk.drbd2165.in‚0 disk.rbd3.out‚ 0.4951171875$ disk.nvme0c0n1.in‚53660.09765625# disk.dm-0.out‚151811.6279296875 proc.cpu.threads‚17 network.sent‚ 2.6674721e+07 disk.drbd2165.out‚0 disk.rbd3.in‚0 disk.rbd1.out‚ 59.4521484375 disk.dm-1.in‚ 108.921875 disk.dm-1.out‚ 1384.3515625 disk.rbd0.out‚0& disk.nvme0n1.out‚151811.6279296875 disk.sda2.in‚0 disk.sda1.in‚0" disk.nvme0n1.in‚53660.09765625 disk.rbd1.in‚ 0.53564453125 disk.sda.in‚ 108.921875 disk.sda.out‚ 1384.3515625* proc.memory.percent‚17.113566398620605 network.recv‚ 5.681741e+06 disk.dm-0.in‚53660.09765625$ memory_percent‚82.89549350738525Þ¥D”é6‚(:‰ ¶æéÒ“í‚÷ gpu.0.gpu‚87.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚39.0 gpu.0.powerWatts‚110.527' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.1054 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚08Þe#Fê6‚(:» ¶æéÒ°­Ñù disk.sda3.in‚ 108.921875 disk.drbd2165.in‚0 disk.sda1.in‚0( disk.nvme0c0n1.out‚151822.7744140625 disk.rbd1.in‚ 0.53564453125 disk.sda.in‚ 108.921875# proc.memory.rssMB‚ 2803.88671875 cpu‚25.583241161870724 proc.cpu.threads‚17 disk.dm-1.out‚ 1384.890625 disk.rbd0.out‚0# disk.nvme0c0n1.in‚ 53712.6171875 disk.dm-1.in‚ 108.921875& disk.rbd0.in‚1.759218604441423e+13 disk.sda1.out‚0 disk.drbd2165.out‚0* disk./.usagePercent‚37.987836948924325 disk.rbd3.out‚ 0.4951171875 disk.sda2.out‚0 disk.rbd1.out‚ 59.4521484375* proc.memory.percent‚17.113566398620605 disk.sda3.out‚ 1384.890625 disk.rbd3.in‚0& disk.nvme0n1.out‚151822.7744140625 disk.sda2.in‚0 disk.sda.out‚ 1384.890625 network.sent‚ 2.6679307e+07! disk.nvme0n1.in‚ 53712.6171875 network.recv‚ 5.685278e+06$ disk./.usageGB‚566.0462226867676 disk.dm-0.in‚ 53712.6171875# disk.dm-0.out‚151822.7744140625# memory_percent‚82.8970193862915) proc.memory.availableMB‚ 2802.15234375YJ¦_Ÿë6‚(:” ÅæéÒº¦‹÷ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚295.399' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚59.079800000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0vt ÚEì6‚(:º ÅæéҀƓù disk.dm-1.in‚ 108.921875 disk.sda1.out‚0 disk.rbd3.out‚ 0.4951171875$ memory_percent‚82.89704322814941$ disk./.usageGB‚566.0532073974609& disk.nvme0n1.out‚151831.8955078125( proc.memory.availableMB‚ 2802.1484375# proc.memory.rssMB‚ 2803.88671875* proc.memory.percent‚17.113566398620605 cpu‚25.583100145596298 disk.sda.in‚ 108.921875 proc.cpu.threads‚17 disk.drbd2165.out‚0 disk.dm-0.in‚ 53713.0390625 disk.sda1.in‚0! disk.nvme0n1.in‚ 53713.0390625 disk.dm-1.out‚ 1385.640625 disk.drbd2165.in‚0 disk.rbd3.in‚0# disk.nvme0c0n1.in‚ 53713.0390625) disk./.usagePercent‚37.98830569871952( disk.nvme0c0n1.out‚151831.8955078125 disk.sda3.in‚ 108.921875 disk.sda3.out‚ 1385.640625 network.recv‚ 5.688556e+06 disk.rbd1.out‚ 59.5302734375 disk.sda.out‚ 1385.640625& disk.rbd0.in‚1.759218604441423e+13 disk.rbd0.out‚0# disk.dm-0.out‚151831.8955078125 network.sent‚ 2.6684939e+07 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd1.in‚ 0.53564453125”)R\”í6‚(:‰ ÔæéÒ÷À…÷ gpu.0.gpu‚99.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚335.402' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚67.0804 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0š+îSFî6‚(:» ÔæéÒÕ•¬ú disk.rbd0.out‚0 network.sent‚ 2.6690131e+07 network.recv‚ 5.692107e+06 disk.drbd2165.in‚0 disk.rbd3.out‚ 0.4951171875 cpu‚25.582977266406658 disk.rbd3.in‚0 disk.rbd1.out‚ 59.5302734375 disk.dm-1.in‚ 108.921875# memory_percent‚82.8892469406128 proc.cpu.threads‚17 disk.sda3.in‚ 108.921875( disk.nvme0c0n1.out‚151848.9912109375 disk.sda.out‚ 1385.984375# disk.dm-0.out‚151848.9912109375) proc.memory.availableMB‚ 2803.42578125$ disk./.usageGB‚566.0576705932617& disk.nvme0n1.out‚151848.9912109375 disk.sda2.out‚0 disk.sda.in‚ 108.921875 disk.dm-1.out‚ 1385.984375# proc.memory.rssMB‚ 2803.88671875* disk./.usagePercent‚37.988605227534585 disk.rbd1.in‚ 0.53564453125& disk.rbd0.in‚1.759218604441423e+13 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda3.out‚ 1385.984375 disk.drbd2165.out‚0 disk.sda2.in‚0 disk.dm-0.in‚ 53713.2578125* proc.memory.percent‚17.113566398620605# disk.nvme0c0n1.in‚ 53713.2578125! disk.nvme0n1.in‚ 53713.2578125Ï\+¼4ï6/ _wandbruntime‚22857  _runtime‚22857;ä¼DÌð6Æ trainer/global_step‚13799 % _timestamp‚1.7843126732953703e+09 _runtime‚22857.512577157 _step‚1134 ' lr-AdamW/pg1‚0.00015725695906145977 ' lr-AdamW/pg2‚0.00015725695906145977ƒPS‹Ññ6Ë ' lr-AdamW/pg1‚0.00015725695906145977 ' lr-AdamW/pg2‚0.00015725695906145977 trainer/global_step‚13799 % _timestamp‚1.7843126732953703e+09 _runtime‚22857.512577157 _step‚1134î´ïº4ò6/ _wandbruntime‚22857  _runtime‚22857ŒÖ3r%ó6”P VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 119509280 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚39095203 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚3964935930941440 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12958928466218 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 117735375 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚257 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3979631432763904 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚3977894859407658 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16738083988480 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 119519253 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 trainer/global_step‚13799 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚32439296 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚3964935956369408 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3966648946573312 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3979631432763904 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12982486190592 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 % _timestamp‚1.7843126732965486e+09 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3979630061200896 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚3964935930941440 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12958928466218 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 117735375 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚257 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚3979631432763904 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚3977894859407658 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16738083988480 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 119519253 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 trainer/global_step‚13799 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚32439296 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚3964935956369408 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3966648946573312 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚3979631432763904 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12982486190592 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 % _timestamp‚1.7843126732965486e+09 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3979630061200896 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 JDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 117743575 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚3966926921129984 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚251 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1087956756364288 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 119517758 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚35403359 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16739276221440 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚246 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚3966926947303424 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚39097925 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 237271326 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3979910358536704 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚74501279 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 119527745 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 119527745 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3979910358536704 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3966926921129984 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12982065836544 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚35403354 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 117743581ß_˜)'÷6£N 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 237271326 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚74501530 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1071217480142848 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚3978175034680482 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 119517758 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 237261333 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 _step‚1136 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 trainer/global_step‚13799 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1087957158984704 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3966926947303424 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1087956756364288 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 119517758 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚35403359 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16739276221440 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚246 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚3966926947303424 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚39097925 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 237271326 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3979910358536704 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚74501279 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 119527745 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 119527745 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚3979910358536704 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3966926921129984 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12982065836544 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚35403354 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 117743581 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 117743575 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚3966926921129984 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚251 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 QDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚35527966 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3993576413637120 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13027456731648 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13006510179648 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13027456731648 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 trainer/global_step‚13849 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3991836555457856 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚3993576413637120 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1091812215037952 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 $ _timestamp‚1.784312751845387e+09 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 119945005 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 _runtime‚22936.06309217 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 238080701 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13028802121216 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚255 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16797900918784 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚3978830045278208 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚260 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 _step‚1139 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚39232721 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 400530432 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚3991835185429566 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚74760942 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13028802121216 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1075014314119168 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13028802121216 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚255 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16797900918784 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚3978830045278208 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚260 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 _step‚1139 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚39232721 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 400530432 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚3991835185429566 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚74760942 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13028802121216 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1075014314119168 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚35527966 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚3993576413637120 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13027456731648 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13006510179648 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13027456731648 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 trainer/global_step‚13849 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚3991836555457856 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚3993576413637120 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1091812215037952 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 $ _timestamp‚1.784312751845387e+09 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 119945005 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 _runtime‚22936.06309217 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 238080701 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 118153881 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 118153875 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚1140 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13006089045572 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚74766019 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16799096829952 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch’k|±_end/segment.small_pool.allocated‚820 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 119943510 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚35530504 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13028381767168 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 238107378 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 118153875 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13029727163904 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 119953497 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3979107901628416 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 119943510 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚3980826957635584 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚253 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16799131359232 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13028381767168 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13007433646006 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚22937.446798541 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 238107378 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚39235768 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1075092168617984 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1075091800526848 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 119953497 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13029727163904 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3993856710972928 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3980826983809024 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 % _timestamp‚1.7843127532294233e+09 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚258 trainer/global_step‚13849 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚74766277 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚35530509 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1091890897356800 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 238107378 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 118153875 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13029727163904 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 119953497 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚3979107901628416 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 119943510 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚3980826957635584 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚253 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16799131359232 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13028381767168 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13007433646006 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚22937.446798541 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 238107378 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚39235768 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1075092168617984 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1075091800526848 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 119953497 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13029727163904 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚3993856710972928 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3980826983809024 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 % _timestamp‚1.7843127532294233e+09 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚258 trainer/global_step‚13849 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚74766277 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚35530509 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1091890897356800 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 118153881 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 118153875 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚1140 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13006089045572 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚74766019 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16799096829952 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 119943510 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚35530504 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13028381767168 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 EDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13073772662272 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 118555975 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 120370757 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚1143 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1095746661132288 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16858026629120 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3994449019584512 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13073772662272 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 120360784 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚4007524137636352 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚4007522766073344 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚35655116 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚4005775511451474[Vh(¢7™P M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 120360784 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚4007524137636352 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚4007522766073344 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚35655116 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚4005775511451474 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13075118051840 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚4005776881479764 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1078889002594304 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚39370854 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16858061165568 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 238926732 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 118555969 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13073772662272 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 118555975 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 120370757 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚1143 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1095746661132288 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16858026629120 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚3994449019584512 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13073772662272 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0÷ö²ñ4£7/ _wandbruntime‚23016  _runtime‚23016»Y$'¤7žN ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13076043094528 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚35657659 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 118564181 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3994727020314624 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚3994727020314624 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 118564175 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13053670759114 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚4007803063409152 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3994726994141184 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚39373528 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16859259633664 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚4006054316695896 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3993002015965184 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 238943430 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 238943430 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 120369262 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚4007803063409152 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚75031427 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚3994726994141184 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚75031182 _runtime‚23016.190322997 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚4007801691838976 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 238933437 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 120369262 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚4006055686724298 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 _step‚1144 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0É°cþG¥7£N TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 _step‚1144 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13076043094528 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚35657659 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 118564181 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚3994727020314624 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚3994727020314624 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 118564175 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13053670759114 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚4007803063409152 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚3994726994141184 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚39373528 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16859259633664 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚4006054316695896 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚3993002015965184 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 238943430 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 238943430 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 120369262 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚4007803063409152 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚75031427 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚3994726994141184 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚75031182 _runtime‚23016.190322997 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚4007801691838976 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 238933437 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 120369262 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚4006055686724298øLÇ•¥4¦7/ _wandbruntime‚23016  _runtime‚23016Ax^ú§7 3loss/SRqqt2/assignment_loss‚0.17195807750107941 /loss/FBt1/detection_loss‚0.2943166459815323 %loss/total_loss‚4.599720468508916 trainer/global_step‚13899 # _timestamp‚1.78431283205792e+09 4loss/FBt1/assignment_loss‚0.00047888551751188146 3loss/FBt2/assignment_loss‚0.0001699655147356688 1loss/SRqqt2/detection_loss‚0.4675811535045109 _runtime‚23016.274826718 0loss/FRt1/assignment_loss‚1.0946026420818185 3loss/SRqqt1/assignment_loss‚0.08133919989834218 /loss/FRt1/detection_loss‚0.4305723579180447 2loss/SRqqt1/detection_loss‚0.44271332304785305 epoch‚34 _step‚1145 1loss/FRt2/assignment_loss‚0.10592569678257673 /loss/FRt2/detection_loss‚1.2910014980757694 0loss/FBt2/detection_loss‚0.21906102268514116œ‘Ù`¨7’ 0loss/FRt1/assignment_loss‚1.0946026420818185 3loss/SRqqt1/assignment_loss‚0.08133919989834218 /loss/FRt1/detection_loss‚0.4305723579180447 2loss/SRqqt1/detection_loss‚0.44271332304785305 epoch‚34 _step‚1145 1loss/FRt2/assignment_loss‚0.10592569678257673 /loss/FRt2/detection_loss‚1.2910014980757694 0loss/FBt2/detection_loss‚0.21906102268514116 3loss/SRqqt2/assignment_loss‚0.17195807750107941 /loss/FBt1/detection_loss‚0.2943166459815323 %loss/total_loss‚4.599720468508916 trainer/global_step‚13899 # _timestamp‚1.78431283205792e+09 4loss/FBt1/assignment_loss‚0.00047888551751188146 3loss/FBt2/assignment_loss‚0.0001699655147356688 1loss/SRqqt2/detection_loss‚0.4675811535045109 _runtime‚23016.274826718ù•/À7‚(:‰ ˆèéÒçÁôö gpu.0.gpu‚89.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚276.276' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.2552 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0žaµFª7‚(:» ˆèéÒçæßù$ disk./.usageGB‚566.1315765380859 disk.rbd1.out‚ 60.0224609375 disk.dm-1.in‚ 108.921875 disk.dm-1.out‚ 1389.546875 disk.dm-0.in‚53914.26953125& proc.memory.availableMB‚ 2802.40625 network.recv‚ 5.734997e+06( disk.nvme0c0n1.out‚151987.2001953125* proc.memory.percent‚17.113566398620605" disk.nvme0n1.in‚53914.26953125# disk.dm-0.out‚151987.2001953125 disk.sda3.in‚ 108.921875 disk.drbd2165.out‚0 proc.cpu.threads‚17 disk.sda.out‚ 1389.546875 disk.rbd0.out‚0 disk.rbd3.out‚ 0.4951171875 network.sent‚ 2.6867537e+07& disk.nvme0n1.out‚151987.2001953125 disk.rbd1.in‚ 0.53564453125 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda3.out‚ 1389.546875) disk./.usagePercent‚37.99356511750295$ disk.nvme0c0n1.in‚53914.26953125 disk.sda2.out‚0 disk.sda.in‚ 108.921875 disk.rbd3.in‚0 cpu‚25.581408628278943 disk.sda2.in‚0& disk.rbd0.in‚1.759218604441423e+13 disk.drbd2165.in‚0$ memory_percent‚82.89546966552734# proc.memory.rssMB‚ 2803.88671875Wl¹rŸ«7‚(:” —èéÒŽôø gpu.0.gpu‚65.0 gpu.0.memory‚20, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚40.0 gpu.0.powerWatts‚273.894' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚54.778800000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0œèsM¬7‚(: —èéÒ¥¦ú disk.sda.in‚ 108.921875 disk.drbd2165.in‚0 disk.rbd3.out‚ 0.4951171875$ disk.nvme0c0n1.in‚53914.48828125& proc.memory.availableMB‚ 2802.21875 disk.rbd1.out‚ 60.0849609375 network.recv‚ 5.739698e+06 disk.dm-1.in‚ 108.921875& disk.rbd0.in‚1.759218604441423e+13 disk.sda3.out‚ 1389.78515625 cpu‚25.581267706530202 disk.dm-0.in‚53914.48828125 disk.sda1.in‚0 disk.sda1.out‚0" disk.nvme0n1.in‚53914.48828125$ disk./.usageGB‚566.1360054016113 disk.sda2.in‚0 disk.sda2.out‚0$ disk.dm-0.out‚151995.43310546875) disk.nvme0c0n1.out‚151995.43310546875 proc.cpu.threads‚17 disk.rbd0.out‚0 disk.sda3.in‚ 108.921875 disk.drbd2165.out‚0 disk.rbd1.in‚ 0.53564453125# proc.memory.rssMB‚ 2803.88671875* proc.memory.percent‚17.113566398620605( disk./.usagePercent‚37.9938623422502 disk.sda.out‚ 1389.78515625 disk.rbd3.in‚0' disk.nvme0n1.out‚151995.43310546875 network.sent‚ 2.693119e+07 disk.dm-1.out‚ 1389.78515625$ memory_percent‚82.896614074707030€Z¤ž­7‚(:“ ¦èéÒÛìþö gpu.0.gpu‚90.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚281.143' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚56.22859999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ódªAC®7‚(:¸ ¦èéÒ»ˆ³ù' disk./.usagePercent‚37.994462679918$ disk./.usageGB‚566.1449508666992 disk.nvme0n1.in‚ 53981.65625 proc.cpu.threads‚17 network.sent‚ 2.6936307e+07 disk.rbd1.in‚ 0.53564453125 disk.drbd2165.in‚0! disk.nvme0c0n1.in‚ 53981.65625$ memory_percent‚82.89659023284912 disk.sda2.out‚0$ disk.dm-0.out‚152006.45361328125 disk.sda1.in‚0 cpu‚25.58115195791066 disk.dm-0.in‚ 53981.65625) disk.nvme0c0n1.out‚152006.45361328125' disk.nvme0n1.out‚152006.45361328125 disk.drbd2165.out‚0 network.recv‚ 5.742899e+06 disk.rbd3.in‚0* proc.memory.percent‚17.113566398620605 disk.sda.out‚ 1389.9765625& disk.rbd0.in‚1.759218604441423e+13 disk.rbd0.out‚0 disk.sda3.in‚ 108.921875 disk.sda3.out‚ 1389.9765625# proc.memory.rssMB‚ 2803.88671875 disk.sda2.in‚0 disk.rbd1.out‚ 60.0849609375 disk.sda.in‚ 108.921875 disk.dm-1.in‚ 108.921875 disk.dm-1.out‚ 1389.9765625 disk.sda1.out‚0 disk.rbd3.out‚ 0.4951171875) proc.memory.availableMB‚ 2802.22265625ÑÙ9£“¯7‚(:ˆ µèéҙΞø gpu.0.gpu‚14.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚332.701' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚66.5402 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0`m7B°7‚(:· µèéÒÏ£¸ø disk.rbd1.out‚ 60.1943359375 disk.sda.in‚ 108.921875" disk.dm-0.out‚152018.923828125 disk.rbd3.out‚ 0.4951171875 proc.cpu.threads‚17 disk.nvme0n1.in‚ 53981.859375% disk.nvme0n1.out‚152018.923828125 disk.sda1.out‚0 disk.sda3.in‚ 108.921875 disk.drbd2165.out‚0$ memory_percent‚82.89661407470703& proc.memory.availableMB‚ 2802.21875# proc.memory.rssMB‚ 2803.88671875 network.sent‚ 2.6941404e+07" disk.nvme0c0n1.in‚ 53981.859375* proc.memory.percent‚17.113566398620605 disk.sda2.in‚0 disk.dm-0.in‚ 53981.859375 disk.sda1.in‚0 disk.drbd2165.in‚0 disk.rbd3.in‚0 disk.rbd0.out‚0# disk./.usageGB‚566.149055480957& disk.rbd0.in‚1.759218604441423e+13 network.recv‚ 5.746312e+06) disk./.usagePercent‚37.99473814402484 disk.sda2.out‚0 disk.dm-1.out‚ 1390.30078125 disk.sda3.out‚ 1390.30078125 cpu‚25.581036557012123' disk.nvme0c0n1.out‚152018.923828125 disk.rbd1.in‚ 0.53564453125 disk.sda.out‚ 1390.30078125 disk.dm-1.in‚ 108.921875ÚÜäw”±7‚(:‰ ÄèéÒ¯üö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚276.839' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.3678 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0uÅ°H²7‚(:½ ÄèéÒ“¿Éù disk.sda.in‚ 108.921875 disk.sda2.out‚0# disk.dm-0.out‚152029.4150390625 disk.sda3.in‚ 108.921875) proc.memory.availableMB‚ 2802.21484375 disk.dm-1.out‚ 1390.41796875 disk.rbd0.out‚0 disk.sda1.out‚0* disk./.usagePercent‚37.995199725609076 disk.dm-1.in‚ 108.921875 disk.drbd2165.out‚0* proc.memory.percent‚17.113566398620605 disk.sda3.out‚ 1390.41796875# disk./.usageGB‚566.155933380127( disk.nvme0c0n1.out‚152029.4150390625& disk.nvme0n1.out‚152029.4150390625 disk.rbd1.out‚ 60.1943359375& disk.rbd0.in‚1.759218604441423e+13 disk.dm-0.in‚ 53982.078125 network.sent‚ 2.6946533e+07" disk.nvme0c0n1.in‚ 53982.078125 disk.rbd1.in‚ 0.53564453125# proc.memory.rssMB‚ 2803.88671875 network.recv‚ 5.749796e+06 disk.sda.out‚ 1390.41796875 disk.sda1.in‚0 disk.drbd2165.in‚0 disk.rbd3.in‚0 cpu‚25.58089473488739 disk.nvme0n1.in‚ 53982.078125 disk.rbd3.out‚ 0.4951171875$ memory_percent‚82.89663791656494 proc.cpu.threads‚17 disk.sda2.in‚0H(•h4³7/ _wandbruntime‚23093  _runtime‚23093ÎU¶²Ì´7Æ ' lr-AdamW/pg1‚0.00015034106289732666 ' lr-AdamW/pg2‚0.00015034106289732666 trainer/global_step‚13949 % _timestamp‚1.7843129091513085e+09 _runtime‚23093.368435581 _step‚1146…@”Éѵ7Ë % _timestamp‚1.7843129091513085e+09 _runtime‚23093.368435581 _step‚1146 ' lr-AdamW/pg1‚0.00015034106289732666 ' lr-AdamW/pg2‚0.00015034106289732666 trainer/global_step‚13949ú²<4¶7/ _wandbruntime‚23093  _runtime‚23093¬,­:(·7”P S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚4021469118509568 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚39508400 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 118966269 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 239752805 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 120786536 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚252 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚4006618223095808 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 120796509 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚4019715837473382 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚4008349056090112 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 118966269 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚4019717207501672 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13120088592896 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚4021470490072576 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 120796509 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 239762784 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16917930924032 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 118966275 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1082762954887168 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚4021469118509568 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚39508400 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 118966269 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 239752805 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 120786536 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚252 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚4006618223095808 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 120796509 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚4019715837473382 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚4008349056090112 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 118966269 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚4019717207501672 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13120088592896 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚4021470490072576 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 120796509 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 239762784 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16917930924032 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 118966275 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1082762954887168 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 JDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚243 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚32432128 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13122359025152 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1099759940072448 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13121013628416 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚4019994642717804 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 118974481 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 120795014 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 239779482 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚4008627056820224 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚4006896079446016 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 118974475 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚35784809 _step‚1148 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚4019996012746206 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚4008627030646784 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚39511144 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚75295948 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 120805001 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1082840809385984 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 400523264 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚4006896104873984 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 120805001 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚4021748044275200 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13121013628416 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 239769489 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚4008627056820224 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 239779482 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 118974481 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚13949 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 120795014 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚35784804 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 _runtime‚23094.753355297 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚238 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1099759539549184 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚4021749415845376 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13122359025152 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820d/@x)'»7£N QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚4019996012746206 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚4008627030646784 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚39511144 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚75295948 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 120805001 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1082840809385984 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 400523264 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚4006896104873984 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 120805001 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚4021748044275200 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13121013628416 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 239769489 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚4008627056820224 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 239779482 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 118974481 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚13949 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 120795014 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚35784804 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 _runtime‚23094.753355297 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚238 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1099759539549184 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚4021749415845376 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13122359025152 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1082840441294848 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13098563271788 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚243 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚32432128 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13122359025152 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1099759940072448 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13121013628416 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚4019994642717804 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 118974481 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 120795014 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 239779482 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚4008627056820224 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚4006896079446016 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 118974475 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚35784809 _step‚1148 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736üThc4¼7/ _wandbruntime‚23094  _runtime‚23094¶Ô¬½7Œ 0loss/FBt2/detection_loss‚0.21469097256065323 epoch‚34 3loss/FBt1/assignment_loss‚0.0008489088288288014 trainer/global_step‚13949 _runtime‚23094.83916583 _step‚1149 3loss/SRqqt1/assignment_loss‚0.14219431870701446 %loss/total_loss‚5.027283284279667 0loss/FRt1/assignment_loss‚1.0047505262020082 2loss/SRqqt2/assignment_loss‚0.1630056408060246 /loss/FRt2/detection_loss‚1.2652830593491091 1loss/SRqqt1/detection_loss‚0.5203736194532524 2loss/SRqqt2/detection_loss‚0.44241121992427207 /loss/FBt1/detection_loss‚0.7206861625752936 % _timestamp‚1.7843129106222236e+09 1loss/FRt2/assignment_loss‚0.13628359958317204 3loss/FBt2/assignment_loss‚4.170899495150033e-05 /loss/FRt1/detection_loss‚0.4167135472950869ENVÚ¾7‘ 1loss/FRt2/assignment_loss‚0.13628359958317204 3loss/FBt2/assignment_loss‚4.170899495150033e-05 /loss/FRt1/detection_loss‚0.4167135472950869 0loss/FBt2/detection_loss‚0.21469097256065323 epoch‚34 3loss/FBt1/assignment_loss‚0.0008489088288288014 trainer/global_step‚13949 _runtime‚23094.83916583 _step‚1149 3loss/SRqqt1/assignment_loss‚0.14219431870701446 %loss/total_loss‚5.027283284279667 0loss/FRt1/assignment_loss‚1.0047505262020082 2loss/SRqqt2/assignment_loss‚0.1630056408060246 /loss/FRt2/detection_loss‚1.2652830593491091 1loss/SRqqt1/detection_loss‚0.5203736194532524 2loss/SRqqt2/detection_loss‚0.44241121992427207 /loss/FBt1/detection_loss‚0.7206861625752936 % _timestamp‚1.7843129106222236e+09ý…«VÑ”¿7‚(:‰ ÓèéÒÉšÌö gpu.0.gpu‚93.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚339.701' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚67.9402 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0S=±Ù7À7‚(:¬ ÓèéÒßÐßú% proc.memory.availableMB‚ 2801.9375 disk.rbd1.out‚ 60.2568359375 disk.rbd0.out‚0$ disk.dm-0.out‚152042.92333984375 disk.drbd2165.out‚0 disk.sda2.out‚0* proc.memory.percent‚17.113566398620605$ disk./.usageGB‚566.1627540588379 disk.sda.in‚ 108.921875 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd2165.in‚0$ memory_percent‚82.89833068847656# proc.memory.rssMB‚ 2803.88671875 disk.rbd1.in‚ 0.53564453125& disk.rbd0.in‚1.759218604441423e+13) disk.nvme0c0n1.out‚152042.92333984375 proc.cpu.threads‚17( disk./.usagePercent‚37.9956574670803' disk.nvme0n1.out‚152042.92333984375 disk.sda2.in‚0 disk.dm-0.in‚53982.5 disk.nvme0c0n1.in‚53982.5 disk.dm-1.out‚ 1390.74609375 disk.sda3.out‚ 1390.74609375 cpu‚25.58076389830868 network.sent‚ 2.6952141e+07 network.recv‚ 5.753256e+06 disk.sda3.in‚ 108.921875 disk.sda.out‚ 1390.74609375 disk.dm-1.in‚ 108.921875 disk.rbd3.in‚0 disk.rbd3.out‚ 0.4951171875 disk.nvme0n1.in‚53982.5îLÇ”Á7‚(:‰ âèéÒ¯óùö gpu.0.gpu‚98.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚266.062' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.2124 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0§â JÂ7‚(:¿ âèéÒ¸¢Õù disk.rbd0.out‚0 disk.dm-0.in‚54047.01171875 disk.sda1.out‚0' proc.memory.availableMB‚ 2802.046875) disk.nvme0c0n1.out‚152053.42822265625" disk.nvme0n1.in‚54047.01171875& disk.rbd0.in‚1.759218604441423e+13* proc.memory.percent‚17.113566398620605 cpu‚25.580627566549897 disk.drbd2165.in‚0$ disk.nvme0c0n1.in‚54047.01171875 disk.rbd3.in‚0 disk.sda2.in‚0 disk.rbd1.in‚ 0.53564453125 disk.rbd1.out‚ 60.2568359375# proc.memory.rssMB‚ 2803.88671875 disk.sda2.out‚0 disk.sda.in‚ 108.921875 disk.sda1.in‚0 disk.sda3.in‚ 108.921875) disk./.usagePercent‚37.99593190715701 disk.drbd2165.out‚0 disk.rbd3.out‚ 0.4951171875' disk.nvme0n1.out‚152053.42822265625 disk.dm-1.in‚ 108.921875$ memory_percent‚82.89766311645508 network.sent‚ 2.7015489e+07 disk.sda.out‚ 1390.890625$ disk.dm-0.out‚152053.42822265625 disk.sda3.out‚ 1390.890625 proc.cpu.threads‚17 network.recv‚ 5.758203e+06$ disk./.usageGB‚566.1668434143066 disk.dm-1.out‚ 1390.890625ñ]ïgŸÃ7‚(:” ñèéÒÛ©Öö gpu.0.gpu‚100.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚46.0 gpu.0.powerWatts‚339.941' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚67.98819999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Çx—ŽKÄ7‚(:À ñèéÒû±¤ù disk.sda.in‚ 108.921875 disk.dm-0.in‚54047.43359375 disk.sda3.out‚ 1391.1015625" disk.nvme0n1.in‚54047.43359375 disk.rbd1.in‚ 0.53564453125$ memory_percent‚82.89468288421631 disk.sda3.in‚ 108.921875 disk.drbd2165.out‚0$ disk./.usageGB‚566.1760673522949 disk.sda.out‚ 1391.1015625 disk.dm-1.in‚ 108.921875 disk.dm-1.out‚ 1391.1015625 disk.rbd1.out‚ 60.3349609375# proc.memory.rssMB‚ 2803.88671875 cpu‚25.58051149633808 proc.cpu.threads‚17 network.sent‚ 2.7019509e+07 disk.sda2.out‚0& disk.rbd0.in‚1.759218604441423e+13 disk.rbd0.out‚0# disk.dm-0.out‚152062.7373046875 disk.sda1.out‚0 disk.drbd2165.in‚0$ disk.nvme0c0n1.in‚54047.43359375 disk.sda2.in‚0* proc.memory.percent‚17.113566398620605 network.recv‚ 5.760044e+06 disk.rbd3.in‚0 disk.rbd3.out‚ 0.4951171875& disk.nvme0n1.out‚152062.7373046875) proc.memory.availableMB‚ 2802.53515625) disk./.usagePercent‚37.99655093337479 disk.sda1.in‚0( disk.nvme0c0n1.out‚152062.7373046875ŒR¥÷”Å7‚(:‰ €ééÒšé³ù gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚260.476' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.0952 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0;+ÛlHÆ7‚(:½ €ééÒå×Äú disk.rbd1.out‚ 60.3349609375& disk.rbd0.in‚1.759218604441423e+13 disk.sda2.out‚0$ memory_percent‚82.89632797241211 cpu‚25.580370152288626 disk.dm-1.out‚ 1391.234375 disk.sda3.in‚ 108.921875" disk.nvme0n1.in‚54047.65234375 disk.sda2.in‚0 disk.sda.in‚ 108.921875$ disk.dm-0.out‚152079.70556640625 disk.sda1.in‚0) disk.nvme0c0n1.out‚152079.70556640625* proc.memory.percent‚17.113566398620605 disk.sda1.out‚0 disk.sda.out‚ 1391.234375 disk.rbd0.out‚0 disk.sda3.out‚ 1391.234375 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.rbd3.out‚ 0.4951171875' proc.memory.availableMB‚ 2802.265625 network.recv‚ 5.76481e+06( disk./.usagePercent‚37.9968432939789$ disk./.usageGB‚566.1804237365723 disk.dm-1.in‚ 108.921875 disk.dm-0.in‚54047.65234375' disk.nvme0n1.out‚152079.70556640625 network.sent‚ 2.7026374e+07 disk.rbd3.in‚0$ disk.nvme0c0n1.in‚54047.65234375 disk.rbd1.in‚ 0.53564453125# proc.memory.rssMB‚ 2803.88671875 proc.cpu.threads‚17¿¢å•Ç7‚(:Š ééÒ¬è‹÷ gpu.0.gpu‚100.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚46.0 gpu.0.powerWatts‚343.304' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚68.6608 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Çf„Ó6È7‚(:« ééÒ‰Ýïù$ disk./.usageGB‚566.1872520446777 disk.sda1.in‚0 network.sent‚ 2.7030192e+07 disk.sda.out‚ 1391.53125 disk.dm-0.out‚ 152087.28125 disk.sda2.in‚0# proc.memory.rssMB‚ 2803.88671875* proc.memory.percent‚17.113566398620605 disk.dm-0.in‚54047.85546875 disk.drbd2165.out‚0 disk.rbd3.out‚ 0.4951171875 proc.cpu.threads‚17) disk./.usagePercent‚37.99730154746519 disk.sda.in‚ 108.921875 disk.rbd3.in‚0# disk.nvme0c0n1.out‚ 152087.28125 disk.rbd1.out‚ 60.3974609375( proc.memory.availableMB‚ 2802.2734375 cpu‚25.580351974879147" disk.nvme0n1.in‚54047.85546875 disk.dm-1.in‚ 108.921875 disk.dm-1.out‚ 1391.53125& disk.rbd0.in‚1.759218604441423e+13 disk.rbd0.out‚0 disk.sda3.in‚ 108.921875 disk.drbd2165.in‚0 disk.sda3.out‚ 1391.53125 disk.sda2.out‚0 network.recv‚ 5.768018e+06$ disk.nvme0c0n1.in‚54047.85546875! disk.nvme0n1.out‚ 152087.28125$ memory_percent‚82.89628028869629 disk.sda1.out‚0 disk.rbd1.in‚ 0.53564453125ö-JŽ4É7/ _wandbruntime‚23172  _runtime‚23172|§ÉÊ7à & lr-AdamW/pg1‚0.0001480606914312254 & lr-AdamW/pg2‚0.0001480606914312254 trainer/global_step‚13999 $ _timestamp‚1.784312988117055e+09 _runtime‚23172.334529148 _step‚1150Gy‘·ÎË7È & lr-AdamW/pg1‚0.0001480606914312254 & lr-AdamW/pg2‚0.0001480606914312254 trainer/global_step‚13999 $ _timestamp‚1.784312988117055e+09 _runtime‚23172.334529148 _step‚1150þ«mÚ4Ì7/ _wandbruntime‚23172  _runtime‚23172ì\tì Í7“P WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16977420478464 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36633600 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13167749913088 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚4022249066422272 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13166404523520 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 240598836 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚75555591 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13143876918650 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 119376569 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚4035416842508800 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚4022249092595712 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 ž°Å-UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 121222261 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13167749913088 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚39646175 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1103615063840768 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚35909411 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1086637275271168 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16977383844864 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚4020512312004608 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚4020512286576640 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 121222261 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 $ _timestamp‚1.784312988118446e+09 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚282 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚35909416 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚4035415470945792 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404724736 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1103614659116032&ö—ú(Î7˜P MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚4020512286576640 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 121222261 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 $ _timestamp‚1.784312988118446e+09 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚282 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚35909416 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚4035415470945792 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404724736 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1103614659116032 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16977420478464 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36633600 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13167749913088 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚4022249066422272 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13166404523520 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 240598836 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚75555591 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13143876918650 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 119376569 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚4035416842508800 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚4022249092595712 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 121222261 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13167749913088 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚39646175 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1103615063840768 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚35909411 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1086637275271168 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16977383844864 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚4020512312004608ÿW½‚·4Ï7/ _wandbruntime‚23173  _runtime‚23173 Ó"˜#'Ð7N H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 240605541 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚4033934968739712 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 119384781 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚4035694396711424 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1086715129769984 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 240615534 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 119384775 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚264 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚75560593 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚4035695768281600 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1086714761678848 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚4020790168354816 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚35911954 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚39648898 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16978567689216 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚4022527093325824 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36626432 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404717568 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚35911959 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 240615534 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13167329559040 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 trainer/global_step‚13999 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16978604315648 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1103693734085632 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 240605541 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚4035695768281600 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136½ï#(;Ñ7¢N YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 trainer/global_step‚13999 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16978604315648 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1103693734085632 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 240605541 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚4035695768281600 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 240605541 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚4033934968739712 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 119384781 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚4035694396711424 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1086715129769984 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 240615534 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 119384775 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚264 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚75560593 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚4035695768281600 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YIDNÝeíeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1086714761678848 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚4020790168354816 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚35911954 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚39648898 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16978567689216 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚4022527093325824 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36626432 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404717568 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚35911959 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 240615534 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13167329559040 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154€ ,4Ò7/ _wandbruntime‚23173  _runtime‚23173±¼…hÓ7Ž 0loss/FRt1/assignment_loss‚0.9918880409689617 1loss/FRt2/assignment_loss‚0.21100409964258357 3loss/SRqqt1/assignment_loss‚0.08457686293258682 0loss/FBt2/detection_loss‚0.17862168877444354 epoch‚34 trainer/global_step‚13999 .loss/FRt2/detection_loss‚1.322803518980055 1loss/SRqqt1/detection_loss‚0.4912026278592474 0loss/FBt1/detection_loss‚0.41310658850461157 _runtime‚23173.807480996 _step‚1153 4loss/FBt1/assignment_loss‚1.2257643385715296e-06 3loss/FBt2/assignment_loss‚0.0008840920085809213 /loss/FRt1/detection_loss‚0.4326384741052091 %loss/total_loss‚4.748436394511762 % _timestamp‚1.7843129895905828e+09 3loss/SRqqt2/assignment_loss‚0.17249871630842747 1loss/SRqqt2/detection_loss‚0.4492104586627169,%M+Ô7“ _step‚1153 4loss/FBt1/assignment_loss‚1.2257643385715296e-06 3loss/FBt2/assignment_loss‚0.0008840920085809213 /loss/FRt1/detection_loss‚0.4326384741052091 %loss/total_loss‚4.748436394511762 % _timestamp‚1.7843129895905828e+09 3loss/SRqqt2/assignment_loss‚0.17249871630842747 1loss/SRqqt2/detection_loss‚0.4492104586627169 0loss/FRt1/assignment_loss‚0.9918880409689617 1loss/FRt2/assignment_loss‚0.21100409964258357 3loss/SRqqt1/assignment_loss‚0.08457686293258682 0loss/FBt2/detection_loss‚0.17862168877444354 epoch‚34 trainer/global_step‚13999 .loss/FRt2/detection_loss‚1.322803518980055 1loss/SRqqt1/detection_loss‚0.4912026278592474 0loss/FBt1/detection_loss‚0.41310658850461157 _runtime‚23173.807480996 ÕÖŠ’Õ7‚(:‡ žééÒñâƒ÷ gpu.0.gpu‚31.0 gpu.0.memory‚9, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚39.0 gpu.0.powerWatts‚110.125' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.025 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0y0ÀÙJÖ7‚(:¿ žééÒù¼Áù" disk.nvme0c0n1.in‚ 54110.609375' disk.nvme0n1.out‚152099.19287109375 disk.rbd0.out‚0 network.recv‚ 5.77134e+06 disk.sda.out‚ 1391.71484375& disk.rbd0.in‚1.759218604441423e+13 disk.sda1.out‚0 disk.sda2.in‚0 disk.rbd1.out‚ 60.5693359375 disk.dm-1.in‚ 108.921875 disk.dm-0.in‚ 54110.609375 disk.sda3.in‚ 108.921875 disk.nvme0n1.in‚ 54110.609375 network.sent‚ 2.7035275e+07 disk.rbd3.out‚ 0.4951171875 disk.sda1.in‚0) disk.nvme0c0n1.out‚152099.19287109375# proc.memory.rssMB‚ 2855.25390625) disk./.usagePercent‚37.99659599070082 disk.sda2.out‚0 disk.sda3.out‚ 1391.71484375) proc.memory.availableMB‚ 2750.80078125 disk.rbd1.in‚ 0.53564453125$ disk.dm-0.out‚152099.19287109375 disk.drbd2165.in‚0 disk.drbd2165.out‚0 cpu‚25.580559489059276 proc.cpu.threads‚17$ disk./.usageGB‚566.1767387390137 disk.sda.in‚ 108.921875 disk.rbd3.in‚0$ memory_percent‚83.21044445037842) proc.memory.percent‚17.42708683013916 disk.dm-1.out‚ 1391.71484375¶ÔѪ”×7‚(:‰ ­ééÒÒ° ÷ gpu.0.gpu‚70.0 gpu.0.memory‚22, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚342.642' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚68.5284 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 (‚µIØ7‚(:¾ ­ééÒ䑾ù disk.sda3.in‚ 108.921875 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.rbd1.out‚ 60.6318359375* disk./.usagePercent‚37.996892959440544 disk.dm-1.out‚ 1392.0390625 disk.dm-0.in‚ 54110.828125 disk.sda1.out‚0) disk.nvme0c0n1.out‚152111.51123046875' disk.nvme0n1.out‚152111.51123046875* proc.memory.percent‚17.113566398620605 cpu‚25.582187574257627 network.sent‚ 2.7099964e+07 disk.dm-1.in‚ 108.921875" disk.nvme0c0n1.in‚ 54110.828125$ disk.dm-0.out‚152111.51123046875 disk.sda2.in‚0 disk.rbd1.in‚ 0.53564453125 disk.sda.in‚ 108.921875 disk.sda.out‚ 1392.0390625& disk.rbd0.in‚1.759218604441423e+13 disk.rbd0.out‚0 disk.rbd3.out‚ 0.4951171875 disk.sda2.out‚0( proc.memory.availableMB‚ 2802.3203125 proc.cpu.threads‚17 disk.rbd3.in‚0# proc.memory.rssMB‚ 2803.88671875 network.recv‚ 5.776175e+06$ disk./.usageGB‚566.1811637878418 disk.sda3.out‚ 1392.0390625 disk.sda1.in‚0 disk.nvme0n1.in‚ 54110.828125$ memory_percent‚82.89599418640137·Ð’'4Ù7/ _wandbruntime‚23201  _runtime‚23201•L‹ê >Ú7…| 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9138282537460327 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.14611512422561646 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9983765482902527 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7426793575286865 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.39814814925193787 +jet/accuracy_3_of_3‚0.05195912718772888 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6360337138175964 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.8564814925193787 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9961451292037964 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.1606871336698532 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.16062091290950775 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4296875 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.2239583283662796 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9937976002693176 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.13264775276184082 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8276357054710388 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6702382564544678 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9921892881393433 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.12145866453647614 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9900473356246948 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5717774629592896 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9711654186248779 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5852314233779907 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.15138019621372223 jet/accuracy_4_of_5‚0 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9784842729568481 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.34944093227386475 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.014176989905536175 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.625 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7857142686843872 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.1480766236782074 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 8"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.636603832244873 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7309752702713013 ,jet/accuracy_3_of_4‚0.061335913836956024 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7609222531318665 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.24074074625968933 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.45370373129844666 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.11328248679637909 *particle/specificity‚0.901326060295105 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 particle/accuracy_2_of_2‚1 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.025279272347688675 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6008971929550171 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5852314233779907 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 particle/accuracy_2_of_4‚1 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6636413335800171 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7597506642341614 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.1198863536119461 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.8999999761581421 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.7451815009117126 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 epoch‚34 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9078207612037659 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.7523055076599121 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6209990978240967 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 /particle/accuracy_3_of_4‚0.9965986609458923 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8208665251731873 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8208665251731873 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 6 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.949999988079071 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.24074074625968933 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7154042720794678 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6341509819030762 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.6704082489013672 & Purity/1FRt1SRqqt2FBt/FRt_purity‚1 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 5validation_average_jet_accuracy‚0.325359970331192 8"Purity/*FRt0SRqqt1FBt/event_purity‚0.832579493522644 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7083333134651184 particle/accuracy_1_of_3‚1 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.8846153616905212 ) Purity/2FRt*SRqqt2FBt/FRt_purity‚0.75 jet/accuracy_1_of_5‚1 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.14228542149066925 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 *jet/accuracy_2_of_4‚0.7214334607124329 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.15900716185569763 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6468342542648315 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 - Purity/0FRt2SRqqt1FBt/FBt_purity‚0.890625 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.03289192542433739 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.22955898940563202 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5883687138557434 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.02380097284913063 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 *jet/accuracy_2_of_2‚0.1300031840801239 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.1848047375679016 *jet/accuracy_2_of_3‚0.5021536946296692 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.5754117369651794 trainer/global_step‚13999 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7083333134651184 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.028209952637553215 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7142857313156128 particle/accuracy_5_of_5‚1 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6498146057128906 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.23076923191547394 - Purity/0FRt2SRqqt*FBt/FBt_purity‚0.890625 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.45572730898857117 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.8796296119689941 9"Purity/*FRt2SRqqt*FBt/event_purity‚0.1997852772474289 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7208428978919983 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.10582888871431351 particle/accuracy_1_of_4‚1 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9777833223342896 6 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.134664386510849 *jet/accuracy_1_of_4‚0.9603413343429565 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.42664867639541626 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.11699340492486954 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8888888955116272 particle/accuracy_4_of_5‚1 8"Purity/*FRt*SRqqt2FBt/event_purity‚0.848046064376831 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7177857160568237 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.2222222238779068 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.1480766236782074 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7392290830612183 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6467120051383972 _step‚1154 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.14732106029987335 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6019848585128784 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9888888597488403 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.007391839288175106 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.5598958730697632 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.4425348937511444 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9872841835021973 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.13329851627349854 'particle/accuracy‚0.835860013961792 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6115744113922119 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6856135129928589 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6712857484817505 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 particle/accuracy_3_of_3‚1 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9792529344558716 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.026258230209350586 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.8999999761581421 *jet/accuracy_1_of_3‚0.8953294157981873 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6601606011390686 particle/accuracy_1_of_1‚1 particle/accuracy_3_of_5‚1 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2647031247615814 8"Purity/1FRt1SRqqt1FBt/event_purity‚0.175817608833313 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.024101756513118744 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.014697698876261711 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.13342472910881042 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9977789521217346 6 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.985582709312439 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5228040814399719 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 particle/accuracy_1_of_2‚1 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9869948625564575 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7336912751197815 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.1996387094259262 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.23076923191547394 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9814945459365845 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.5958849787712097 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.10499999672174454 jet/accuracy_5_of_5‚0 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.5842701196670532 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.34287962317466736 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9882744550704956 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.13803236186504364 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.02380097284913063 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9877681732177734 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6463915705680847 *loss/val_total_loss‚10.318492965479564 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5126581192016602 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9924572706222534 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6596264839172363 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 'particle/f_score‚0.6095198392868042 jet/accuracy_3_of_5‚1 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 /particle/accuracy_4_of_4‚0.9368542432785034 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.15546643733978271 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.2222222238779068 ,jet/accuracy_4_of_4‚0.013899499550461769 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.3234403431415558 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 _runtime‚23201.085320464 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9937319755554199 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7476180195808411 particle/accuracy_1_of_5‚1 particle/accuracy_2_of_5‚1 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9898399114608765 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4000000059604645 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.27441221475601196 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 *jet/accuracy_1_of_2‚0.7165024876594543 +particle/sensitivity‚0.5965238809585571 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9948286414146423 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.20359772443771362 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6702874898910522 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6146131753921509 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4358455538749695 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.02506435476243496 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 7"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.74276202917099 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.6622965931892395 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4350285232067108 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 $ _timestamp‚1.784313016867504e+09 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.8564814925193787 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9935952425003052 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.47958335280418396 *jet/accuracy_1_of_1‚0.2820086181163788 particle/accuracy_2_of_3‚1 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9977789521217346 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.024346530437469482 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.14666588604450226 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 jet/accuracy_2_of_5‚1 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9933884739875793 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.21384704113006592 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.34119993448257446(õÁgmÛ7Š| 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9900473356246948 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5717774629592896 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9711654186248779 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5852314233779907 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.15138019621372223 jet/accuracy_4_of_5‚0 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9784842729568481 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.34944093227386475 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.014176989905536175 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.625 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7857142686843872 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.1480766236782074 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 8"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.636603832244873 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7309752702713013 ,jet/accuracy_3_of_4‚0.061335913836956024 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7609222531318665 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.24074074625968933 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.45370373129844666 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.11328248679637909 *particle/specificity‚0.901326060295105 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 particle/accuracy_2_of_2‚1 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.025279272347688675 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6008971929550171 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5852314233779907 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 particle/accuracy_2_of_4‚1 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6636413335800171 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7597506642341614 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.1198863536119461 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.8999999761581421 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.7451815009117126 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 epoch‚34 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9078207612037659 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.7523055076599121 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6209990978240967 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 /particle/accuracy_3_of_4‚0.9965986609458923 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8208665251731873 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8208665251731873 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 6 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.949999988079071 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.24074074625968933 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7154042720794678 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6341509819030762 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.6704082489013672 & Purity/1FRt1SRqqt2FBt/FRt_purity‚1 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 5validation_average_jet_accuracy‚0.325359970331192 8"Purity/*FRt0SRqqt1FBt/event_purity‚0.832579493522644 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7083333134651184 particle/accuracy_1_of_3‚1 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.8846153616905212 ) Purity/2FRt*SRqqt2FBt/FRt_purity‚0.75 jet/accuracy_1_of_5‚1 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.14228542149066925 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 *jet/accuracy_2_of_4‚0.7214334607124329 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.15900716185569763 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6468342542648315 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 - Purity/0FRt2SRqqt1FBt/FBt_purity‚0.890625 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.03289192542433739 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.22955898940563202 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5883687138557434 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.02380097284913063 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 *jet/accuracy_2_of_2‚0.1300031840801239 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.1848047375679016 *jet/accuracy_2_of_3‚0.5021536946296692 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.5754117369651794 trainer/global_step‚13999 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7083333134651184 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.028209952637553215 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7142857313156128 particle/accuracy_5_of_5‚1 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6498146057128906 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.23076923191547394 - Purity/0FRt2SRqqt*FBt/FBt_purity‚0.890625 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.45572730898857117 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.8796296119689941 9"Purity/*FRt2SRqqt*FBt/event_purity‚0.1997852772474289 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7208428978919983 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.10582888871431351 particle/accuracy_1_of_4‚1 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9777833223342896 6 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.134664386510849 *jet/accuracy_1_of_4‚0.9603413343429565 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.42664867639541626 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 & Purity/2FRt*SRqq"¯ßl£"t2FBt/FBt_purity‚1 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.11699340492486954 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8888888955116272 particle/accuracy_4_of_5‚1 8"Purity/*FRt*SRqqt2FBt/event_purity‚0.848046064376831 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7177857160568237 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.2222222238779068 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.1480766236782074 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7392290830612183 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6467120051383972 _step‚1154 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.14732106029987335 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6019848585128784 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9888888597488403 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.007391839288175106 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.5598958730697632 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.4425348937511444 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9872841835021973 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.13329851627349854 'particle/accuracy‚0.835860013961792 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6115744113922119 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6856135129928589 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6712857484817505 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 particle/accuracy_3_of_3‚1 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9792529344558716 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.026258230209350586 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.8999999761581421 *jet/accuracy_1_of_3‚0.8953294157981873 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6601606011390686 particle/accuracy_1_of_1‚1 particle/accuracy_3_of_5‚1 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2647031247615814 8"Purity/1FRt1SRqqt1FBt/event_purity‚0.175817608833313 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.024101756513118744 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.014697698876261711 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.13342472910881042 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9977789521217346 6 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.985582709312439 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5228040814399719 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 particle/accuracy_1_of_2‚1 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9869948625564575 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7336912751197815 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.1996387094259262 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.23076923191547394 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9814945459365845 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.5958849787712097 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.10499999672174454 jet/accuracy_5_of_5‚0 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.5842701196670532 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.34287962317466736 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9882744550704956 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.13803236186504364 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.02380097284913063 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9877681732177734 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6463915705680847 *loss/val_total_loss‚10.318492965479564 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5126581192016602 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9924572706222534 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6596264839172363 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 'particle/f_score‚0.6095198392868042 jet/accuracy_3_of_5‚1 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 /particle/accuracy_4_of_4‚0.9368542432785034 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.15546643733978271 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.2222222238779068 ,jet/accuracy_4_of_4‚0.013899499550461769 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.3234403431415558 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 _runtime‚23201.085320464 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9937319755554199 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7476180195808411 particle/accuracy_1_of_5‚1 particle/accuracy_2_of_5‚1 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9898399114608765 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4000000059604645 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.27441221475601196 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 *jet/accuracy_1_of_2‚0.7165024876594543 +particle/sensitivity‚0.5965238809585571 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9948286414146423 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.20359772443771362 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6702874898910522 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6146131753921509 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4358455538749695 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.02506435476243496 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 7"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.74276202917099 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.6622965931892395 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4350285232067108 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 $ _timestamp‚1.784313016867504e+09 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.8564814925193787 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9935952425003052 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.47958335280418396 *jet/accuracy_1_of_1‚0.2820086181163788 particle/accuracy_2_of_3‚1 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9977789521217346 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.024346530437469482 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.14666588604450226 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 jet/accuracy_2_of_5‚1 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9933884739875793 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.21384704113006592 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.34119993448257446 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9138282537460327 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.14611512422561646 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9983765482902527 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7426793575286865 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.39814814925193787 +jet/accuracy_3_of_3‚0.05195912718772888 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6360337138175964 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.8564814925193787 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9961451292037964 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.1606871336698532 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.16062091290950775 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4296875 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.2239583283662796 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9937976002693176 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.13264775276184082 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8276357054710388 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6702382564544678 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9921892881393433 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.12145866453647614 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1‚ ‰œ”Ü7‚(:‰ ¼ééÒÄâÝö gpu.0.gpu‚39.0 gpu.0.memory‚13, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚336.753' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚67.3506 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0GþTBÝ7‚(:· ¼ééÒ¿ÿèø$ disk./.usageGB‚566.1884727478027 disk.rbd1.in‚ 0.53564453125 disk.dm-1.in‚ 108.921875 disk.sda3.out‚ 1392.4296875 disk.rbd3.in‚0 disk.dm-1.out‚ 1392.4296875 disk.dm-0.in‚ 54111.03125 disk.drbd2165.in‚0 cpu‚25.583502901658985 proc.cpu.threads‚17 disk.sda2.out‚0 disk.rbd1.out‚ 60.6318359375 disk.sda.out‚ 1392.4296875 disk.nvme0n1.in‚ 54111.03125& disk.nvme0n1.out‚152120.8310546875! disk.nvme0c0n1.in‚ 54111.03125( disk.nvme0c0n1.out‚152120.8310546875) disk./.usagePercent‚37.99738346987615 disk.sda.in‚ 108.921875 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda3.in‚ 108.921875& disk.rbd0.in‚1.759218604441423e+13 disk.rbd3.out‚ 0.4951171875( proc.memory.availableMB‚ 2802.0546875* proc.memory.percent‚17.113566398620605 network.sent‚ 2.7105044e+07 disk.sda2.in‚0# proc.memory.rssMB‚ 2803.88671875 disk.rbd0.out‚0# disk.dm-0.out‚152120.8310546875 disk.drbd2165.out‚0$ memory_percent‚82.89761543273926 network.recv‚ 5.779567e+06Xå/½ŸÞ7‚(:” ËééÒŶ¢÷ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚257.462' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚51.492399999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0$«’^Dß7‚(:¹ ËééÒ³Åðø( disk./.usagePercent‚37.9990265262309$ disk./.usageGB‚566.2129554748535& disk.rbd0.in‚1.759218604441423e+13 disk.nvme0n1.in‚ 54111.453125 network.recv‚ 5.784043e+06 disk.dm-1.in‚ 108.921875( disk.nvme0c0n1.out‚152134.4716796875 disk.sda.in‚ 108.921875 disk.sda1.out‚0 disk.sda3.in‚ 108.921875 disk.sda2.out‚0 disk.rbd1.out‚ 60.6943359375$ memory_percent‚82.89618492126465( proc.memory.availableMB‚ 2802.2890625 cpu‚25.583588696810722 disk.dm-0.in‚ 54111.453125 disk.sda1.in‚0& disk.nvme0n1.out‚152134.4716796875 disk.sda2.in‚0 proc.cpu.threads‚17 network.sent‚ 2.7162955e+07 disk.drbd2165.out‚0 disk.sda.out‚ 1392.8515625# disk.dm-0.out‚152134.4716796875 disk.rbd3.in‚0 disk.rbd3.out‚ 0.4951171875 disk.rbd0.out‚0 disk.sda3.out‚ 1392.8515625" disk.nvme0c0n1.in‚ 54111.453125# proc.memory.rssMB‚ 2803.88671875* proc.memory.percent‚17.113566398620605 disk.dm-1.out‚ 1392.8515625 disk.drbd2165.in‚0 disk.rbd1.in‚ 0.53564453125ÝÏS’à7‚(:‡ ÚééÒ¦°ý÷ gpu.0.gpu‚9.0 gpu.0.memory‚4, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚318.173' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚63.6346 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚06=û7Oá7‚(:Ä Úééҳнù disk.sda.in‚ 108.921875 disk.dm-0.in‚54210.60546875 disk.sda2.in‚0 disk.rbd1.out‚ 60.6943359375 disk.drbd2165.out‚0" disk.nvme0n1.in‚54210.60546875* proc.memory.percent‚17.113566398620605$ disk./.usageGB‚566.2202110290527# disk.dm-0.out‚152147.7548828125 disk.rbd3.out‚ 0.4951171875( disk.nvme0c0n1.out‚152147.7548828125 disk.sda2.out‚0 disk.rbd1.in‚ 0.53564453125 proc.cpu.threads‚17* disk./.usagePercent‚37.999513452561025 disk.sda.out‚ 1393.04296875& disk.rbd0.in‚1.759218604441423e+13 cpu‚25.58371615187778 network.recv‚ 5.787583e+06 disk.rbd0.out‚0 disk.drbd2165.in‚0 disk.rbd3.in‚0& disk.nvme0n1.out‚152147.7548828125) proc.memory.availableMB‚ 2802.39453125# proc.memory.rssMB‚ 2803.88671875 disk.sda1.out‚0 network.sent‚ 2.7168082e+07 disk.dm-1.in‚ 108.921875 disk.sda1.in‚0 disk.sda3.in‚ 108.921875 disk.sda3.out‚ 1393.04296875$ memory_percent‚82.89554119110107 disk.dm-1.out‚ 1393.04296875$ disk.nvme0c0n1.in‚54210.60546875³¸&¾”â7‚(:‰ éééÒ’üÂ÷ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚259.116' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚51.8232 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0­+ÙvKã7‚(:À éééÒ‰×ëú disk.sda2.in‚0* proc.memory.percent‚17.113566398620605 cpu‚25.583777220930013 proc.cpu.threads‚17 disk.dm-1.in‚ 108.921875 disk.dm-0.in‚54211.02734375 disk.rbd3.in‚0 disk.rbd1.out‚ 60.7724609375# disk.dm-0.out‚152156.5986328125$ disk.nvme0c0n1.in‚54211.02734375 disk.sda.in‚ 108.921875 network.sent‚ 2.7173165e+07$ disk./.usageGB‚566.2272148132324& disk.rbd0.in‚1.759218604441423e+13 disk.rbd3.out‚ 0.4951171875 disk.sda.out‚ 1393.3671875 disk.rbd1.in‚ 0.53564453125# memory_percent‚82.8970193862915# proc.memory.rssMB‚ 2803.88671875 disk.dm-1.out‚ 1393.3671875 disk.drbd2165.in‚0" disk.nvme0n1.in‚54211.02734375 network.recv‚ 5.790791e+06) disk./.usagePercent‚37.99998348239389 disk.sda3.in‚ 108.921875 disk.drbd2165.out‚0& disk.nvme0n1.out‚152156.5986328125) proc.memory.availableMB‚ 2802.15234375 disk.sda1.out‚0 disk.sda3.out‚ 1393.3671875( disk.nvme0c0n1.out‚152156.5986328125 disk.sda2.out‚0 disk.rbd0.out‚0 disk.sda1.in‚0£<ï”ä7‚(:‰ øééÒ¬€æö gpu.0.gpu‚57.0 gpu.0.memory‚17, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚332.701' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚66.5402 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0—SЮNå7‚(:à øééÒ¨÷ëø& disk.rbd0.in‚1.759218604441423e+13 disk.dm-0.in‚54211.23046875 disk.drbd2165.in‚0$ disk.nvme0c0n1.in‚54211.23046875) disk.nvme0c0n1.out‚152172.92919921875 disk.rbd1.out‚ 60.7724609375) disk./.usagePercent‚38.00026816277197 disk.sda3.in‚ 108.921875 disk.sda2.in‚0 disk.rbd1.in‚ 0.53564453125( proc.memory.availableMB‚ 2802.8046875# proc.memory.rssMB‚ 2803.88671875$ disk./.usageGB‚566.2314567565918 disk.rbd0.out‚0$ disk.dm-0.out‚152172.92919921875 cpu‚25.583910340063564' disk.nvme0n1.out‚152172.92919921875 disk.sda.in‚ 108.921875 disk.sda.out‚ 1393.6796875 disk.dm-1.out‚ 1393.6796875 disk.rbd3.out‚ 0.4951171875 disk.sda3.out‚ 1393.6796875 disk.rbd3.in‚0 network.recv‚ 5.794488e+06 disk.sda2.out‚0 proc.cpu.threads‚17 network.sent‚ 2.7176983e+07* proc.memory.percent‚17.113566398620605" disk.nvme0n1.in‚54211.23046875 disk.dm-1.in‚ 108.921875 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd2165.out‚0$ memory_percent‚82.89303779602051ž¨Vä4æ7/ _wandbruntime‚23279  _runtime‚23279i€±ŽËç7Å _runtime‚23279.10002869 _step‚1155 ' lr-AdamW/pg1‚0.00014579302837020762 ' lr-AdamW/pg2‚0.00014579302837020762 trainer/global_step‚14049 % _timestamp‚1.7843130948827963e+09JFÉ=Ðè7Ê _step‚1155 ' lr-AdamW/pg1‚0.00014579302837020762 ' lr-AdamW/pg2‚0.00014579302837020762 trainer/global_step‚14049 % _timestamp‚1.7843130948827963e+09 _runtime‚23279.10002869ƒ yû?4é7/ _wandbruntime‚23279  _runtime‚23279b´B(ê7”P A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚4053197018991616 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 241625876 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 _step‚1156 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚4051428227581243 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚4039977110536192 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚4053195647428608 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚14049 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 119918875 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 121706995 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 121697022 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚4039977110536192 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚17044815674368 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1107604489039872 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚4053197018991616 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚17044781137920 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 121697022 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1090560075993088 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚4051429597609533 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚4038232309846016 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 % _timestamp‚1.7843130948839498e+09 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 119918875 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚75865423 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚4038232284418048 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 121706995 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 119918881 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 119918881 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13219908455424 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13195943163195 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1090559707901952 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13219908455424 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820ÊW¿önë7™P G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 _runtime‚23279.101312053 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚4053195647428608 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚14049 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 119918875 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 121706995 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 121697022 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚4039977110536192 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚17044815674368 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1107604489039872 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚4053197018991616 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚17044781137920 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 121697022 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1090560075993088 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚4051429597609533 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚4038232309846016 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 % _timestamp‚1.7843130948839498e+09 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 119918875 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚75865423 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚4038232284418048 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 121706995 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 119918881 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 119918881 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13219908455424 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13195943163195 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1090559707901952 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13219908455424 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚8¬~îè±20 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚4053197018991616 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 241625876 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 _step‚1156 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚4051428227581243 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚4039977110536192 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 JDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 241642574 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17045980096512 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚4051707032825665 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚4053475944764416 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13220833498112 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 241632581 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚36048022 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚4040255111266304 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚238 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚23280.487597904 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13198211229875 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚17046014625792 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 119927087 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚39822706 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚4051708402854067 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13196866629441 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚4040255111266304 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13219488101376 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚39822468 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚4053475944764416 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚75870490 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 119927081 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1107683174406144 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 121705500 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1107683577026560 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚4038510166196224 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚4040255085092864 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 121705500 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚4053474573194240 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 241632581 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 121715487 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1090637194309632 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008±<~ )'ð7£N PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1090637194309632 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 119927087 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 241642574 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17045980096512 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚4051707032825665 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚4053475944764416 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13220833498112 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 241632581 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚36048022 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚4040255111266304 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚238 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚23280.487597904 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13198211229875 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚17046014625792 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 119927087 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚39822706 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚4051708402854067 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13196866629441 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚4040255111266304 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13219488101376 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚39822468 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚4053475944764416 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚75870490 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 119927081 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1107683174406144 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 121705500 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1107683577026560 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚4038510166196224 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚4040255085092864 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 121705500 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚4053474573194240 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 241632581 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 121715487… œOr4ñ7/ _wandbruntime‚23280  _runtime‚23280 Q-ò7 3loss/SRqqt2/assignment_loss‚0.13841365106944659 2loss/FBt1/assignment_loss‚2.03485948618135e-05 1loss/SRqqt1/detection_loss‚0.4434929553644483 2loss/SRqqt2/detection_loss‚0.48368399016460933 epoch‚35 0loss/FRt1/assignment_loss‚0.9157057231274721 1loss/FRt2/assignment_loss‚0.13938443902376535 3loss/SRqqt1/assignment_loss‚0.06940881624531885 4loss/FBt2/assignment_loss‚3.1019854004451993e-06 _runtime‚23280.573862933 0loss/FRt1/detection_loss‚0.43551526085946357 0loss/FBt1/detection_loss‚0.47341327590487814 /loss/FBt2/detection_loss‚0.2761921964723318 trainer/global_step‚14049 _step‚1158 /loss/FRt2/detection_loss‚1.4469287048054644 %loss/total_loss‚4.822162463617461 % _timestamp‚1.7843130963569207e+09P›6ó7” epoch‚35 0loss/FRt1/assignment_loss‚0.9157057231274721 1loss/FRt2/assignment_loss‚0.13938443902376535 3loss/SRqqt1/assignment_loss‚0.06940881624531885 4loss/FBt2/assignment_loss‚3.1019854004451993e-06 _runtime‚23280.573862933 0loss/FRt1/detection_loss‚0.43551526085946357 0loss/FBt1/detection_loss‚0.47341327590487814 /loss/FBt2/detection_loss‚0.2761921964723318 trainer/global_step‚14049 _step‚1158 /loss/FRt2/detection_loss‚1.4469287048054644 %loss/total_loss‚4.822162463617461 % _timestamp‚1.7843130963569207e+09 3loss/SRqqt2/assignment_loss‚0.13841365106944659 2loss/FBt1/assignment_loss‚2.03485948618135e-05 1loss/SRqqt1/detection_loss‚0.4434929553644483 2loss/SRqqt2/detection_loss‚0.48368399016460933† –ïŸô7‚(:” –êéÒÀèçö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚161.348' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚32.269600000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0b“½Kõ7‚(:À –êéÒ‡¦Õù& disk.nvme0n1.out‚152195.8115234375$ disk./.usageGB‚566.2434959411621 disk.drbd2165.out‚0* proc.memory.percent‚17.113566398620605* disk./.usagePercent‚38.001076122550025 disk.sda3.out‚ 1394.16796875( disk.nvme0c0n1.out‚152195.8115234375 disk.dm-1.in‚ 108.921875 disk.dm-1.out‚ 1394.16796875# disk.dm-0.out‚152195.8115234375# proc.memory.rssMB‚ 2803.88671875 network.sent‚ 2.7234527e+07 disk.sda3.in‚ 108.921875 disk.sda.in‚ 108.921875 disk.sda1.in‚0 disk.rbd3.out‚ 0.4951171875 disk.rbd0.out‚0' proc.memory.availableMB‚ 2802.203125& disk.rbd0.in‚1.759218604441423e+13 network.recv‚ 5.801916e+06 disk.drbd2165.in‚0# disk.nvme0c0n1.in‚ 54322.9609375 disk.sda2.out‚0 disk.rbd1.out‚ 60.8349609375 cpu‚25.584111414939933 proc.cpu.threads‚17 disk.rbd1.in‚ 0.53564453125 disk.dm-0.in‚ 54322.9609375 disk.rbd3.in‚0 disk.sda2.in‚0! disk.nvme0n1.in‚ 54322.9609375$ memory_percent‚82.89670944213867 disk.sda.out‚ 1394.16796875 disk.sda1.out‚0“ïøÛ”ö7‚(:‰ ¥êéÒ¯óùö gpu.0.gpu‚90.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚267.733' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.5466 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0!w¶þI÷7‚(:¾ ¥êéÒ¤ßõù disk.rbd3.out‚ 0.4951171875# disk.nvme0c0n1.in‚ 54323.1640625( disk.nvme0c0n1.out‚152203.2587890625 disk.sda1.in‚0 disk.sda2.in‚0$ disk./.usageGB‚566.2501525878906 disk.sda3.in‚ 108.921875 disk.drbd2165.out‚0# proc.memory.rssMB‚ 2803.88671875 disk.rbd1.out‚ 60.9130859375 disk.dm-1.out‚ 1394.50390625 proc.cpu.threads‚17 disk.sda2.out‚0 disk.sda.in‚ 108.921875 disk.sda.out‚ 1394.50390625& disk.rbd0.in‚1.759218604441423e+13) disk./.usagePercent‚38.00152285569728 disk.rbd3.in‚0 network.sent‚ 2.728722e+07 cpu‚25.584185349518062 network.recv‚ 5.806206e+06 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.dm-1.in‚ 108.921875$ memory_percent‚82.90033340454102* proc.memory.percent‚17.113566398620605 disk.sda3.out‚ 1394.50390625 disk.rbd0.out‚0' proc.memory.availableMB‚ 2801.609375! disk.nvme0n1.in‚ 54323.1640625& disk.nvme0n1.out‚152203.2587890625 disk.rbd1.in‚ 0.53564453125 disk.dm-0.in‚ 54323.1640625# disk.dm-0.out‚152203.2587890625áiVAŸø7‚(:” ´êéÒÚÞˆ÷ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚41.0 gpu.0.powerWatts‚288.103' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚57.620599999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0>,¼Lù7‚(:Á ´êéÒ‹ƒúù# proc.memory.rssMB‚ 2803.88671875# disk.dm-0.out‚152213.8818359375 disk.sda3.out‚ 1394.83203125* disk./.usagePercent‚38.001998261688364! disk.nvme0n1.in‚ 54323.5859375 disk.sda.in‚ 108.921875* proc.memory.percent‚17.113566398620605 proc.cpu.threads‚17 network.recv‚ 5.809632e+06& disk.rbd0.in‚1.759218604441423e+13 disk.drbd2165.in‚0 disk.drbd2165.out‚0( disk.nvme0c0n1.out‚152213.8818359375& disk.nvme0n1.out‚152213.8818359375 disk.sda2.out‚0 disk.rbd1.in‚ 0.53564453125 disk.rbd0.out‚0 disk.sda1.in‚0 disk.rbd3.out‚ 0.4951171875) proc.memory.availableMB‚ 2801.86328125 disk.rbd3.in‚0$ memory_percent‚82.89878368377686 cpu‚25.58430248798131 network.sent‚ 2.7292305e+07 disk.rbd1.out‚ 60.9130859375 disk.dm-0.in‚ 54323.5859375 disk.sda1.out‚0 disk.sda3.in‚ 108.921875$ disk./.usageGB‚566.2572364807129# disk.nvme0c0n1.in‚ 54323.5859375 disk.sda2.in‚0 disk.sda.out‚ 1394.83203125 disk.dm-1.in‚ 108.921875 disk.dm-1.out‚ 1394.83203125ó¼ïgú7‚(:” ÃêéÒÏ”øö gpu.0.gpu‚90.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.761306p "98762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚285.446' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚57.089200000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0̨‘jCû7‚(:¸ ÃêéÒ“†Ëù disk.rbd1.out‚ 61.0380859375 disk.dm-1.out‚ 1395.21875 disk.sda.in‚ 108.921875& disk.rbd0.in‚1.759218604441423e+13$ disk.dm-0.out‚152225.81201171875 disk.rbd3.in‚0 disk.sda2.in‚0 disk.dm-1.in‚ 108.921875 disk.rbd0.out‚0 disk.drbd2165.in‚0! disk.nvme0n1.in‚ 54323.8046875 disk.sda1.in‚0 disk.sda3.out‚ 1395.21875 disk.drbd2165.out‚0' disk.nvme0n1.out‚152225.81201171875# memory_percent‚82.8974962234497 network.sent‚ 2.7297437e+07 disk.rbd3.out‚ 0.4951171875) disk.nvme0c0n1.out‚152225.81201171875) proc.memory.availableMB‚ 2802.07421875 cpu‚25.58434550523696 disk.sda1.out‚0 disk.sda3.in‚ 108.921875 proc.cpu.threads‚17 disk.sda.out‚ 1395.21875 disk.dm-0.in‚ 54323.8046875* proc.memory.percent‚17.113566398620605) disk./.usagePercent‚38.00231801509863 disk.sda2.out‚0# disk.nvme0c0n1.in‚ 54323.8046875# proc.memory.rssMB‚ 2803.88671875 network.recv‚ 5.81251e+06$ disk./.usageGB‚566.2620010375977 disk.rbd1.in‚ 0.53564453125*=Ÿ”ü7‚(:‰ ÒêéÒ °—÷ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚40.0 gpu.0.powerWatts‚296.693' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚59.3386 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0~§k4Lý7‚(:Á ÒêéÒ‘ñ²ú disk.sda3.in‚ 108.921875 disk.drbd2165.out‚0 disk.sda1.in‚0) disk.nvme0c0n1.out‚152234.78857421875 disk.dm-1.out‚ 1395.5234375 disk.rbd0.out‚0 network.sent‚ 2.7302636e+07 disk.rbd3.out‚ 0.4951171875$ disk.nvme0c0n1.in‚54454.71484375" disk.nvme0n1.in‚54454.71484375' disk.nvme0n1.out‚152234.78857421875 disk.sda2.out‚0 disk.rbd1.out‚ 61.0380859375 disk.dm-1.in‚ 108.921875 disk.sda3.out‚ 1395.5234375 disk.sda.out‚ 1395.5234375$ memory_percent‚82.89918899536133* proc.memory.percent‚17.113566398620605 proc.cpu.threads‚17 disk.drbd2165.in‚0' proc.memory.availableMB‚ 2801.796875 network.recv‚ 5.816122e+06) disk./.usagePercent‚38.00289838417875$ disk./.usageGB‚566.2706489562988 disk.sda2.in‚0 disk.sda.in‚ 108.921875 disk.dm-0.in‚54454.71484375$ disk.dm-0.out‚152234.78857421875 disk.sda1.out‚0 disk.rbd3.in‚0& disk.rbd0.in‚1.759218604441423e+13# proc.memory.rssMB‚ 2803.88671875 disk.rbd1.in‚ 0.53564453125 cpu‚25.58439652653512Q²êx4þ7/ _wandbruntime‚23358  _runtime‚23358õÖZÌÿ7Æ _step‚1159 ' lr-AdamW/pg1‚0.00014353821936203442 ' lr-AdamW/pg2‚0.00014353821936203442 trainer/global_step‚14099 % _timestamp‚1.7843131741676972e+09 _runtime‚23358.3848626851%êÑ€8Ë % _timestamp‚1.7843131741676972e+09 _runtime‚23358.384862685 _step‚1159 ' lr-AdamW/pg1‚0.00014353821936203442 ' lr-AdamW/pg2‚0.00014353821936203442 trainer/global_step‚14099‡ ”=48/ _wandbruntime‚23358  _runtime‚23358V ‹(‚8”P P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1094434396377088 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 % _timestamp‚1.7843131741685507e+09 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚4067141999864832 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 122122774 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚76130806 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 120329175 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1094434028285952 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 120329175 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚17104911962112 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚4067143371427840 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚39958437 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚265 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 242451949 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚4053877147041792 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚76131071 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚4067141999864832 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚17104946498560 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 120329181 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 122132747 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚260 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚4067143371427840 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 trainer/global_step‚14099 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚4052126373326848 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1111539342875648 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1111538940248064 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚36172634 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚4053877147041792 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚4065369923631441 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 _runtime‚23358.388982594 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚36172629 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 122132747 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1111539342875648 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1111538940248064 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚36172634 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚4053877147041792 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚4065369923631441 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 _runtime‚23358.388982594 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚36172629 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 122132747 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 122122774 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚76130806 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 120329175 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1094434028285952 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 120329175 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚17104911962112 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚4067143371427840 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚39958437 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚265 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 242451949 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚4053877147041792 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚76131071 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚4067141999864832 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚17104946498560 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640ˆ Yýn4„8/ _wandbruntime‚23359  _runtime‚23359h¯õ&ß!…8žN M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13267149428736 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1094511514693632 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚4067422297200640 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚39960858 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 122131252 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 120337381 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 242478626 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚4065648728875975 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13244448342983 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13265804032000 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17106109773824 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _runtime‚23359.772019657 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 122141239 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 242468633 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 242478626 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 120337387 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚4054155147771904 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _step‚1161 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚76136030 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640œ»ßô)'†8£N VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚4067422297200640 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚39960858 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 122131252 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 120337381 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 242478626 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚4065648728875975 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13244448342983 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13265804032000 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17106109773824 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _runtime‚23359.772019657 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 122141239 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 242468633 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 242478626 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 120337387 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚4054155147771904 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _step‚1161 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚76136030 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13267149428736 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1094511514693632 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 EDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚4081088352301056 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13312540316672 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 120739475 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36633600 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 122548526 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚17164426621952 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13311194927104 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 243288001 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚76395995 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚4079308879625059 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 120739475 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚36299784 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13289761989733 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚76395721 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚40096211 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚269 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚4066020462235648 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚4067777183547392 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚4066020487663616 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 243297980 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 120739481 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13311194927104 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚4081089723864064 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚4081089723864064 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚4067777183547392 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 _step‚1164 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 122558499 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404724736 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 % _timestamp‚1.7843132533458726e+09 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚4079310249653349 trainer/global_step‚14149 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 120739481 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚4081088352301056 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚4067777157373952 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚36299779 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚40095942 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1115472738658304 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 _runtime‚23437.563227008ª£­ƒ/™8™P G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1115472738658304 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 _runtime‚23437.563227008 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 122558499 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13312540316672 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 122548526 JÉç¶ð"BDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚274 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1115473143383040 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚4081088352301056 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13312540316672 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 120739475 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36633600 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 122548526 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚17164426621952 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13311194927104 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 243288001 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚76395995 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚4079308879625059 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 120739475 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚36299784 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13289761989733 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚76395721 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚40096211 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚269 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚4066020462235648 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚4067777183547392 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚4066020487663616 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 243297980 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 120739481 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13311194927104 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚4081089723864064 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚4081089723864064 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚4067777183547392 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 _step‚1164 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 122558499 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404724736 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 % _timestamp‚1.7843132533458726e+09 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚4079310249653349 trainer/global_step‚14149 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 120739481 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚4081088352301056 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚4067777157373952 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚36299779 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚40095942 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926Œ óœM4š8/ _wandbruntime‚23438  _runtime‚23438c Âá#'›8N TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚40098948 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17165574765568 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚4079589054897883 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚17165611392000 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚14149 $ _timestamp‚1.784313254730144e+09 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13313465359360 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚4081368649636864 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13289340855657 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚261 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚256 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚40098692 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13290685456091 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚76401014 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13313465359360 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 120747681 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚4066298344013824 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 243314678 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚4081367278066688 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚4079587684869481§NY"('œ8¢N F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚17165611392000 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚14149 $ _timestamp‚1.784313254730144e+09 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13313465359360 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚4081368649636864 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13289340855657 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚261 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚256 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚40098692 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13290685456091 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚76401014 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13313465359360 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 120747681 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚4066298344013824 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 243314678 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚4081367278066688 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚4079587684869481 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚40098948 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17165574765568 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚4079589054897883 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 Áÿ»î48/ _wandbruntime‚23439  _runtime‚23439É›HËž8Œ 2loss/FBt2/assignment_loss‚0.006718325208327577 2loss/SRqqt2/detection_loss‚0.44779283840467193 /loss/FBt1/detection_loss‚0.3950636022918843 /loss/FBt2/detection_loss‚0.2522205598819695 0loss/FRt1/detection_loss‚0.42875482037765505 1loss/SRqqt1/detection_loss‚0.5533619767383702 %loss/total_loss‚4.627827917765217 epoch‚35 _step‚1166 trainer/global_step‚14149 $ _timestamp‚1.784313254816818e+09 _runtime‚23439.033709243 1loss/FRt2/assignment_loss‚0.09893672185461735 3loss/SRqqt1/assignment_loss‚0.09661534945294341 4loss/FBt1/assignment_loss‚6.8317054324751325e-06 .loss/FRt2/detection_loss‚1.219576273016062 0loss/FRt1/assignment_loss‚0.9767965314926044 3loss/SRqqt2/assignment_loss‚0.15198408734067867ðÜîŸ8‘ 0loss/FRt1/assignment_loss‚0.9767965314926044 3loss/SRqqt2/assignment_loss‚0.15198408734067867 2loss/FBt2/assignment_loss‚0.006718325208327577 2loss/SRqqt2/detection_loss‚0.44779283840467193 /loss/FBt1/detection_loss‚0.3950636022918843 /loss/FBt2/detection_loss‚0.2522205598819695 0loss/FRt1/detection_loss‚0.42875482037765505 1loss/SRqqt1/detection_loss‚0.5533619767383702 %loss/total_loss‚4.627827917765217 epoch‚35 _step‚1166 trainer/global_step‚14149 $ _timestamp‚1.784313254816818e+09 _runtime‚23439.033709243 1loss/FRt2/assignment_loss‚0.09893672185461735 3loss/SRqqt1/assignment_loss‚0.09661534945294341 4loss/FBt1/assignment_loss‚6.8317054324751325e-06 .loss/FRt2/detection_loss‚1.219576273016062Ž µ-cŒ” 8‚(:‰ ¬ëéÒäÕÌö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚250.509' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚50.1018 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ZùÏL¡8‚(:Á ¬ëéÒ´•‰ú disk.rbd1.in‚ 0.53564453125 disk.sda.in‚ 108.921875 disk.drbd2165.in‚0 disk.rbd3.out‚ 0.4951171875 disk.dm-1.in‚ 108.921875 disk.dm-1.out‚ 1396.78515625 disk.dm-0.in‚54599.68359375# disk.dm-0.out‚152300.4501953125 network.sent‚ 2.7391719e+07) disk./.usagePercent‚38.00533224780677 disk.sda.out‚ 1396.78515625& disk.rbd0.in‚1.759218604441423e+13 disk.sda1.out‚0 disk.sda3.out‚ 1396.78515625$ disk.nvme0c0n1.in‚54599.68359375$ disk./.usageGB‚566.3069152832031 disk.sda2.out‚0 disk.sda1.in‚0 network.recv‚ 5.838329e+06 disk.rbd0.out‚0' proc.memory.availableMB‚ 2802.109375" disk.nvme0n1.in‚54599.68359375& disk.nvme0n1.out‚152300.4501953125 disk.rbd1.out‚ 61.2568359375( disk.nvme0c0n1.out‚152300.4501953125$ memory_percent‚82.89728164672852 disk.sda2.in‚0 disk.sda3.in‚ 108.921875* proc.memory.percent‚17.113566398620605 disk.drbd2165.out‚0 disk.rbd3.in‚0# proc.memory.rssMB‚ 2803.88671875 cpu‚25.58492578591757 proc.cpu.threads‚17'ßH”¢8‚(:‰ »ëéÒíÅðö gpu.0.gpu‚88.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚334.052' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚66.8104 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0qôs£8‚(:µ »ëéÒ’œØù network.recv‚ 5.842129e+06) disk./.usagePercent‚38.00577590886361$ disk./.usageG—gÍGB‚566.3135261535645 disk.rbd1.in‚ 0.53564453125& disk.rbd0.in‚1.759218604441423e+13 disk.drbd2165.out‚0' disk.nvme0n1.out‚152313.86865234375$ disk.dm-0.out‚152313.86865234375 disk.sda.in‚ 108.921875 disk.rbd0.out‚0 disk.rbd3.in‚0 disk.dm-0.in‚ 54600.09375# proc.memory.rssMB‚ 2803.88671875 cpu‚25.585053372154643) disk.nvme0c0n1.out‚152313.86865234375 disk.nvme0n1.in‚ 54600.09375) proc.memory.availableMB‚ 2802.35546875* proc.memory.percent‚17.113566398620605! disk.nvme0c0n1.in‚ 54600.09375 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda.out‚ 1397.21875 disk.dm-1.out‚ 1397.21875 disk.sda1.out‚0$ memory_percent‚82.89577960968018 network.sent‚ 2.7454145e+07 disk.rbd3.out‚ 0.4951171875 disk.dm-1.in‚ 108.921875 disk.sda1.in‚0 disk.sda3.out‚ 1397.21875 disk.drbd2165.in‚0 disk.rbd1.out‚ 61.3193359375 disk.sda3.in‚ 108.921875 proc.cpu.threads‚17 Ä®¤8‚(:’ ÊëéÒîèø gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚305.64' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚61.12799999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚08ë`?¥8‚(:´ ÊëéÒÜ•þø network.sent‚ 2.7459329e+07) disk./.usagePercent‚38.00607620570128% disk.nvme0n1.out‚152325.798828125 disk.sda2.out‚0 disk.rbd1.out‚ 61.3193359375* proc.memory.percent‚17.113566398620605 disk.rbd0.out‚0 disk.sda3.in‚ 108.921875 disk.dm-1.in‚ 108.921875 disk.dm-1.out‚ 1397.46484375) proc.memory.availableMB‚ 2802.32421875# proc.memory.rssMB‚ 2803.88671875# disk./.usageGB‚566.318000793457& disk.rbd0.in‚1.759218604441423e+13" disk.dm-0.out‚152325.798828125 disk.rbd1.in‚ 0.53564453125$ memory_percent‚82.89597034454346 cpu‚25.585128355922922 disk.dm-0.in‚ 54600.3125 disk.drbd2165.out‚0 disk.rbd3.in‚0 disk.nvme0c0n1.in‚ 54600.3125' disk.nvme0c0n1.out‚152325.798828125 disk.nvme0n1.in‚ 54600.3125 disk.sda2.in‚0 proc.cpu.threads‚17 disk.sda1.out‚0 disk.sda3.out‚ 1397.46484375 disk.drbd2165.in‚0 disk.sda.in‚ 108.921875 disk.sda.out‚ 1397.46484375 network.recv‚ 5.845729e+06 disk.sda1.in‚0 disk.rbd3.out‚ 0.4951171875:ûgÖ”¦8‚(:‰ ÙëéÒÊ›õö gpu.0.gpu‚92.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚324.638' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚64.9276 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0V¶ÂE§8‚(:º ÙëéÒú·äú disk.sda2.in‚0 disk.dm-1.in‚ 108.921875& disk.rbd0.in‚1.759218604441423e+13 disk.rbd3.in‚0 disk.rbd1.out‚ 61.4599609375 disk.sda.in‚ 108.921875 disk.sda3.in‚ 108.921875( proc.memory.availableMB‚ 2802.5703125 disk.sda2.out‚0 network.recv‚ 5.849315e+06) disk./.usagePercent‚38.00669523191907$ disk./.usageGB‚566.3272247314453 disk.dm-1.out‚ 1397.7109375 disk.sda1.out‚0$ memory_percent‚82.89446830749512 disk.rbd1.in‚ 0.53564453125 disk.dm-0.in‚ 54734.28125 disk.drbd2165.out‚0' disk.nvme0n1.out‚152333.96044921875 disk.sda.out‚ 1397.7109375* proc.memory.percent‚17.113566398620605 network.sent‚ 2.7465832e+07! disk.nvme0c0n1.in‚ 54734.28125) disk.nvme0c0n1.out‚152333.96044921875 disk.rbd0.out‚0$ disk.dm-0.out‚152333.96044921875# proc.memory.rssMB‚ 2803.88671875 disk.drbd2165.in‚0 disk.sda1.in‚0 disk.sda3.out‚ 1397.7109375 cpu‚25.585133759336408 proc.cpu.threads‚17 disk.rbd3.out‚ 0.4951171875 disk.nvme0n1.in‚ 54734.28125d#ˆŸ¨8‚(:” èëéÒäè÷ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚285.735' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚57.147000000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Š\ï—F©8‚(:» èëéÒÑý³ù disk.rbd3.out‚ 0.4951171875 disk.rbd1.in‚ 0.53564453125) proc.memory.availableMB‚ 2802.31640625 disk.sda3.out‚ 1398.08984375 disk.rbd1.out‚ 61.4599609375# proc.memory.rssMB‚ 2803.88671875* proc.memory.percent‚17.113566398620605 network.recv‚ 5.852743e+06* disk./.usagePercent‚38.005809957865665 disk.sda.out‚ 1398.08984375" disk.dm-0.out‚152350.044921875 disk.rbd3.in‚0 disk.nvme0n1.in‚ 54734.484375 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-1.in‚ 108.921875& disk.rbd0.in‚1.759218604441423e+13 disk.rbd0.out‚0 disk.sda1.in‚0 network.sent‚ 2.7469591e+07 disk.dm-0.in‚ 54734.484375 disk.sda1.out‚0 disk.drbd2165.in‚0 proc.cpu.threads‚17 disk.sda.in‚ 108.921875 disk.drbd2165.out‚0% disk.nvme0n1.out‚152350.044921875 cpu‚25.58498697532471 disk.dm-1.out‚ 1398.08984375 disk.sda3.in‚ 108.921875" disk.nvme0c0n1.in‚ 54734.484375' disk.nvme0c0n1.out‚152350.044921875$ memory_percent‚82.89601802825928$ disk./.usageGB‚566.3140335083008¨”ñ‘4ª8/ _wandbruntime‚23516  _runtime‚23516\š ñË«8Å ' lr-AdamW/pg1‚0.00013906774195800436 ' lr-AdamW/pg2‚0.00013906774195800436 trainer/global_step‚14199 $ _timestamp‚1.784313332386961e+09 _runtime‚23516.604141441 _step‚1167¨.šÐ¬8Ê trainer/global_step‚14199 $ _timestamp‚1.784313332386961e+09 _runtime‚23516.604141441 _step‚1167 ' lr-AdamW/pg1‚0.00013906774195800436 ' lr-AdamW/pg2‚0.00013906774195800436 ’DÚŠ4­8/ _wandbruntime‚23516  _runtime‚23516‰Ó§(®8”P S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚4095034704737280 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚17223885061120 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 121149775 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13357510857728 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚4081677220052992 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13358856247296 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 122984251 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚4079914551144448 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 122984251 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚17223921694720 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 _step‚1168 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚36426934 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 121149781 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13358856247296 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1102183037145088 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚4095034704737280 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1119406958839808 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1102182669053952 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚4095036076300288 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13335999102841 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 121149781 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚262 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚36426929 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚4081677193879552 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚36426934 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 121149781 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13358856247296 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1102183037145088 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚4095034704737280 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1119406958839808 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1102182669053952 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚4095036076300288 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13335999102841 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 121149781 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚262 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚36426929 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚4081677193879552 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚4095034704737280 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚17223885061120 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 121149775 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13357510857728 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚4081677220052992 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13358856247296 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 122984251 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚4079914551144448 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 122984251 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚17223921694720 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 _step‚1168 JDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚40236895 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚4080192432922624 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚36429472 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 121157987 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 121157981 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚4081955220783104 trainer/global_step‚14199 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚40237150 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚4095315002073088 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 _step‚1169 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚4095315002073088 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚255 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _runtime‚23517.992623664 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚4080192458350592 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1119485225481216 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17225070019584 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚76666367 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚4095313630502912 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚4081955220783104 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 122992743 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1102260523552768 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13336922569199 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pooê½kÞdl.current‚ 1344600434 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13359781289984 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 244140737 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 122992743 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚76666627 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 121157981 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13359781289984 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13358435893248 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚4093529380919791 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚4093528010891389 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚36429477 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404717568 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 244150730 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚260 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚01]2L)'²8£N OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 121157987 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 121157981 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚4081955220783104 trainer/global_step‚14199 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚40237150 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚4095315002073088 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 _step‚1169 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚4095315002073088 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚255 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _runtime‚23517.992623664 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚4080192458350592 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1119485225481216 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17225070019584 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚76666367 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚4095313630502912 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚4081955220783104 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 122992743 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1102260523552768 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13336922569199 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13359781289984 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 244140737 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 122992743 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚76666627 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 121157981 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13359781289984 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13358435893248 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚4093529380919791 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚4093528010891389 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚36429477 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404717568 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 244150730 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚260 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 244140737 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚40236895 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚4080192432922624 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚36429472‘ ªÄ4³8/ _wandbruntime‚23518  _runtime‚23518©yA½´8 1loss/FRt2/assignment_loss‚0.15717748569536813 2loss/FBt2/assignment_loss‚6.82833513910363e-06 0loss/FRt1/detection_loss‚0.41988384206404844 /loss/FRt2/detection_loss‚1.3111004747568977 /loss/FBt1/detection_loss‚0.2946204896047564 trainer/global_step‚14199 2loss/SRqqt1/assignment_loss‚0.0678093752279257 epoch‚35 % _timestamp‚1.7843133338597553e+09 _step‚1170 3loss/FBt1/assignment_loss‚0.0002588210627379448 1loss/SRqqt1/detection_loss‚0.5017204460101097 2loss/SRqqt2/detection_loss‚0.43782536414931583 0loss/FBt2/detection_loss‚0.15811588314198644 0loss/FRt1/assignment_loss‚0.9469653708755913 3loss/SRqqt2/assignment_loss‚0.14645293714131624 %loss/total_loss‚4.441937318065193 _runtime‚23518.076696082'Ãpµ8’ 3loss/FBt1/assignment_loss‚0.0002588210627379448 1loss/SRqqt1/detection_loss‚0.5017204460101097 2loss/SRqqt2/detection_loss‚0.43782536414931583 0loss/FBt2/detection_loss‚0.15811588314198644 0loss/FRt1/assignment_loss‚0.9469653708755913 3loss/SRqqt2/assignment_loss‚0.14645293714131624 %loss/total_loss‚4.441937318065193 _runtime‚23518.076696082 1loss/FRt2/assignment_loss‚0.15717748569536813 2loss/FBt2/assignment_loss‚6.82833513910363e-06 0loss/FRt1/detection_loss‚0.41988384206404844 /loss/FRt2/detection_loss‚1.3111004747568977 /loss/FBt1/detection_loss‚0.2946204896047564 trainer/global_step‚14199 2loss/SRqqt1/assignment_loss‚0.0678093752279257 epoch‚35 % _timestamp‚1.7843133338597553e+09 _step‚1170’ j€'”¶8‚(:‰ ÷ëéÒÅïöö gpu.0.gpu‚92.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚38.0 gpu.0.powerWatts‚110.527' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.1054 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0þ{–íC·8‚(:¸ ÷ëéÒÔû network.recv‚ 5.856391e+06& disk.rbd0.in‚1.759218604441423e+13 disk.dm-0.in‚ 54734.703125 disk.rbd0.out‚0 disk.rbd3.out‚ 0.4951171875 disk.nvme0n1.in‚ 54734.703125 disk.rbd1.in‚ 0.53564453125 disk.rbd1.out‚ 61.5224609375$ disk.nvme0n1.out‚152359.56640625 cpu‚25.584875783563366) disk./.usagePercent‚38.00748322310946 disk.sda1.out‚0 disk.sda2.out‚0 disk.sda.in‚ 108.921875 disk.sda.out‚ 1398.54296875 disk.dm-1.out‚ 1398.54296875 disk.sda3.out‚ 1398.54296875 disk.drbd2165.in‚0 proc.cpu.threads‚17$ disk./.usageGB‚566.3389663696289 disk.dm-1.in‚ 108.921875 disk.sda2.in‚0* proc.memory.percent‚17.113566398620605 disk.sda1.in‚0 disk.sda3.in‚ 108.921875 disk.rbd3.in‚0) proc.memory.availableMB‚ 2802.05078125# proc.memory.rssMB‚ 2803.88671875 network.sent‚ 2.7476207e+07! disk.dm-0.out‚152359.56640625 disk.drbd2165.out‚0" disk.nvme0c0n1.in‚ 54734.703125& disk.nvme0c0n1.out‚152359.56640625$ memory_percent‚82.89763927459717Üä^Ο¸8‚(:” †ìéÒ¼ýØù gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚263.406' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚52.681200000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‹Ã3¹8‚(:¨ †ìéÒû«‚ü disk.drbd2165.out‚0 disk.rbd1.in‚ 0.53564453125$ memory_percent‚82.89685249328613 disk.sda1.out‚0 disk.sda3.in‚ 108.921875 disk.nvme0c0n1.in‚ 54735.125& disk.nvme0n1.out‚152375.3408203125 disk.sda.out‚ 1398.8125 network.recv‚ 5.861472e+06 disk.dm-0.in‚ 54735.125# disk.dm-0.out‚152375.3408203125( proc.memory.availableMB‚ 2802.1796875* proc.memory.percent‚17.113566398620605 proc.cpu.threads‚17 disk.rbd3.in‚0 disk.rbd3.out‚ 0.4951171875 disk.sda2.out‚0 disk.rbd1.out‚ 61.5224609375 network.sent‚ 2.7538562e+07 disk.dm-1.in‚ 108.921875 disk.drbd2165.in‚0( disk.nvme0c0n1.out‚152375.3408203125& disk.rbd0.in‚1.759218604441423e+13 disk.sda2.in‚0 disk.sda.in‚ 108.921875$ disk./.usageGB‚566.3456039428711 disk.dm-1.out‚ 1398.8125# proc.memory.rssMB‚ 2803.88671875) disk./.usagePercent‚38.00792867621904 disk.sda1.in‚0 disk.nvme0n1.in‚ 54735.125 cpu‚25.584724254835898 disk.rbd0.out‚0 disk.sda3.out‚ 1398.8125í²*žº8‚(:“ •ìéÒ³ÙÆø gpu.0.gpu‚55.0 gpu.0.memory‚18, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚330.288' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚66.05760000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ZˆÀK»8‚(:À •ìéÒïµÛú) disk.nvme0c0n1.out‚152383.74462890625) proc.memory.availableMB‚ 2802.40234375 cpu‚25.58460972241774 disk.sda.in‚ 108.921875 disk.sda.out‚ 1399.1015625 disk.dm-0.in‚ 54837.6328125 disk.sda1.in‚0 disk.drbd2165.in‚0 disk.rbd3.in‚0 disk.sda2.in‚0 network.recv‚ 5.864596e+06 disk.dm-1.out‚ 1399.1015625& disk.rbd0.in‚1.759218604441423e+13$ disk.dm-0.out‚152383.74462890625 disk.sda1.out‚0 disk.sda3.in‚ 108.921875 disk.sda3.out‚ 1399.1015625 disk.rbd3.out‚ 0.4951171875* disk./.usagePercent‚38.008253805787525$ disk./.usageGB‚566.3504486083984 disk.dm-1.in‚ 108.921875# disk.nvme0c0n1.in‚ 54837.6328125 disk.rbd1.in‚ 0.53564453125' disk.nvme0n1.out‚152383.74462890625* proc.memory.percent‚17.113566398620605 disk.rbd1.out‚ 61.6005859375 disk.rbd0.out‚0! disk.nvme0n1.in‚ 54837.6328125 disk.sda2.out‚0 proc.cpu.threads‚17 network.sent‚ 2.754315e+07 disk.drbd2165.out‚0$ memory_percent‚82.89549350738525# proc.memory.rssMB‚ 2803.88671875Ž[2”¼8‚(:‰ ¤ìéÒæ¯îö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚263.118' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.6236 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0boÖA½8‚(:¶ ¤ìéÒ²›Æú disk.rbd3.in‚0 disk.sda2.in‚0 disk.sda.in‚ 108.921875 disk.dm-1.out‚ 1399.23046875 network.sent‚ 2.7548865e+07 disk.rbd3.out‚ 0.4951171875& disk.nvme0c0n1.out‚152394.85546875 disk.sda1.out‚0* proc.memory.percent‚17.113566398620605) disk./.usagePercent‚38.00871359531902 disk.drbd2165.in‚0 disk.drbd2165.out‚0# disk.nvme0c0n1.in‚ 54837.8515625 disk.sda3.out‚ 1399.23046875 disk.sda2.out‚0 disk.dm-0.in‚ 54837.8515625# proc.memory.rssMB‚ 2803.88671875 disk.rbd1.in‚ 0.53564453125 disk.rbd1.out‚ 61.6005859375& disk.rbd0.in‚1.759218604441423e+13 cpu‚25.584471460795594! disk.nvme0n1.in‚ 54837.8515625 disk.dm-1.in‚ 108.921875 disk.sda1.in‚0 proc.cpu.threads‚17$ disk.nvme0n1.out‚152394.85546875 disk.sda.out‚ 1399.23046875 disk.rbd0.out‚0! disk.dm-0.out‚152394.85546875$ memory_percent‚82.89871215820312 network.recv‚ 5.868178e+06$ disk./.usageGB‚566.3572998046875 disk.sda3.in‚ 108.921875$ proc.memory.availableMB‚2801.875íÍ[3”¾8‚(:‰ ³ìéÒêÆÍö gpu.0.gpu‚100.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚342.225' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚68.445 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0À;çH¿8‚(:½ ³ìéÒÿæ½ú disk.sda1.out‚0 disk.drbd2165.in‚0 disk.drbd2165.out‚0% disk.nvme0n1.out‚152407.505859375 disk.sda2.out‚0# proc.memory.rssMB‚ 2803.88671875 network.sent‚ 2.7553951e+07 network.recv‚ 5.871349e+06$ disk./.usageGB‚566.3639450073242' disk.nvme0c0n1.out‚152407.505859375 disk.rbd1.out‚ 61.7255859375 cpu‚25.58435437537875 disk.sda.out‚ 1399.49609375 disk.sda2.in‚0 proc.cpu.threads‚17" disk.dm-0.out‚152407.505859375 disk.rbd3.in‚0 disk.rbd3.out‚ 0.4951171875! disk.nvme0n1.in‚ 54838.2734375 disk.rbd1.in‚ 0.53564453125* proc.memory.percent‚17.113566398620605 disk.dm-1.out‚ 1399.49609375& disk.rbd0.in‚1.759218604441423e+13 disk.sda1.in‚0# disk.nvme0c0n1.in‚ 54838.2734375 disk.sda.in‚ 108.921875 disk.dm-1.in‚ 108.921875) disk./.usagePercent‚38.00915956044367 disk.rbd0.out‚0 disk.dm-0.in‚ 54838.2734375 disk.sda3.out‚ 1399.49609375$ memory_percent‚82.90021419525146 disk.sda3.in‚ 108.921875) proc.memory.availableMB‚ 2801.62890625”<•”À8‚(:‰ ÂìéÒƒˆ’÷ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚281.966' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚56.3932 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÿ÷Ÿ HÁ8‚(:½ ÂìéÒŒñÀù disk.sda2.out‚0 disk.dm-0.in‚ 54838.4765625" disk.dm-0.out‚152418.322265625* proc.memory.percent‚17.113566398620605 proc.cpu.threads‚17 disk.sda.in‚ 108.921875$ memory_percent‚82.90107250213623 disk.sda3.out‚ 1399.85546875% disk.nvme0n1.out‚152418.322265625 disk.sda3.in‚ 108.921875 cpu‚25.58421833079028' disk.nvme0c0n1.out‚152418.322265625 disk.rbd1.out‚ 61.7255859375 disk.sda.out‚ 1399.85546875) disk./.usagePercent‚38.00940967980462 disk.rbd0.out‚0 disk.sda1.in‚0) proc.memory.availableMB‚ 2801.48828125# proc.memory.rssMB‚ 2803.88671875 disk.rbd3.out‚ 0.4951171875& disk.rbd0.in‚1.759218604441423e+13 disk.drbd2165.out‚0 network.sent‚ 2.7559027e+07 network.recv‚ 5.874768e+06$ disk./.usageGB‚566.3676719665527# disk.nvme0c0n1.in‚ 54838.4765625 disk.sda2.in‚0 disk.dm-1.out‚ 1399.85546875 disk.rbd3.in‚0! disk.nvme0n1.in‚ 54838.4765625 disk.rbd1.in‚ 0.53564453125 disk.dm-1.in‚ 108.921875 disk.sda1.out‚0 disk.drbd2165.in‚0>l™4Â8/ _wandbruntime‚23595  _runtime‚23595#f‘–ÌÃ8Æ _step‚1171 ' lr-AdamW/pg1‚0.00013685236069255744 ' lr-AdamW/pg2‚0.00013685236069255744 trainer/global_step‚14249 % _timestamp‚1.7843134108984215e+09 _runtime‚23595.115634468zª9‹ÑÄ8Ë ' lr-AdamW/pg1‚0.00013685236069255744 ' lr-AdamW/pg2‚0.00013685236069255744 trainer/global_step‚14249 % _timestamp‚1.7843134108984215e+09 _runtime‚23595.115634468 _step‚1171“ ¼-‚4Å8/ _wandbruntime‚23595  _runtime‚23595±á &VÆ8”P F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 244970084 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 121560081 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚17283439170048 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚4095577256558592 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚4108981057173504 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13403826788352 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 123410003 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1106056989437952 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13380891615627 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 244960105 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚4108982428736512 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 123400030 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1123340796699136 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚76927156 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚4095577230385152 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _step‚1172 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚4107190901697165 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13382236215949 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚4095577230385152 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚40372808 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 123410003 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚17283402536448 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚4107189531668875 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13405172177920 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚4108981057173504 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚76926887 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36633600 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 121560081 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404724736 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 trainer/global_step‚14249 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 VGDeviceStatsMonitor.on_train_batch_start/alloc/¼†ÓÄ ated_bytes.small_pool.peak‚ 1680116736 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1123340391974400 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1106057357529088 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13405172177920 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚269 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚4095577256558592 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 121560075 _runtime‚23595.119953142 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚36554084 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚36554079ÿFåV(Ç8™P NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚4093808665481216 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚4093808640053248 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚4108982428736512 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚4108981057173504 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13403826788352 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 123410003 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1106056989437952 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13380891615627 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 244960105 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚4108982428736512 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 123400030 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1123340796699136 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚76927156 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚4095577230385152 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _step‚1172 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚4107190901697165 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13382236215949 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚4095577230385152 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚40372808 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 123410003 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚17283402536448 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚4107189531668875 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13405172177920 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚4108981057173504 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚76926887 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36633600 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 121560081 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404724736 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 trainer/global_step‚14249 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1123340391974400 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1106057357529088 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13405172177920 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚269 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚4095577256558592 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 121560075 _runtime‚23595.119953142 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚36554084 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚36554079 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 244970084 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 121560081 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚17283439170048 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚4095577256558592 JDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚4095855257288704 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚36556627 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1106134475845632 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚76932201 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 244986782 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚76932453 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚4094086547259392 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 123408508 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 _runtime‚23596.501559279 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 244976789 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚4095855231115264 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1123419478772224 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13404751823872 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 123418495 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1106134843936768 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚4109259982939136 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13381815081873 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚1173 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 123418495 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚4095855231115264 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 121568287 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17284600306176 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚4095855257288704 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 244976789 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 121568281 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13406097220608 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13406097220608 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13404751823872 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚40375826 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚4109261354509312 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 244986782 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13383159682307 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚4094086521831424 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚247 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚40375579 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚17284634835456 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚4109261354509312 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚4107468336913297 % _timestamp‚1.7843134122841945e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1123419076151808 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 trainer/global_step‚14249 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚36556622 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 1234085085´8i¢%Ê8£N KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚247 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚40375579 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚17284634835456 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚4109261354509312 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚4107468336913297 % _timestamp‚1.7843134122841945e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1123419076151808 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 trainer/global_step‚14249 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚36556622 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 123408508 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚4095855257288704 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚36556627 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1106134475845632 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚76932201 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 244986782 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚76932453 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚4094086547259392 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 123408508 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 _runtime‚23596.501559279 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 244976789 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚4095855231115264 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1123419478772224 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13404751823872 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 123418495 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1106134843936768 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚4109259982939136 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13381815081873 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚1173 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 123418495 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚4095855231115264 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 121568287 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17284600306176 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚4095855257288704 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 244976789 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 121568281 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13406097220608 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13406097220608 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13404751823872 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚40375826 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚4109261354509312 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 244986782 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 ?6DeviceStatsMo ~Ò¡‡nitor.on_train_batch_end/num_device_alloc‚1995 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13383159682307 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚4094086521831424• )ÏmV4Ë8/ _wandbruntime‚23596  _runtime‚23596Á:hÌ8‰ 1loss/SRqqt2/assignment_loss‚0.156737782175699 4loss/FBt1/assignment_loss‚3.6124024983440572e-09 /loss/FBt1/detection_loss‚0.3506485274916816 % _timestamp‚1.7843134123706417e+09 _runtime‚23596.587578776 _step‚1174 epoch‚35 0loss/FRt1/assignment_loss‚0.9698196076280251 1loss/FRt2/assignment_loss‚0.14290094359883532 -loss/FRt2/detection_loss‚1.20029351685873 2loss/SRqqt1/detection_loss‚0.46834599134578087 0loss/FBt2/detection_loss‚0.19305819103220145 %loss/total_loss‚4.587489347545318 /loss/FRt1/detection_loss‚0.4682615667772305 0loss/SRqqt2/detection_loss‚0.499507347663816 3loss/SRqqt1/assignment_loss‚0.13790690183479642 2loss/FBt2/assignment_loss‚8.96752611843554e-06 trainer/global_step‚14249&jqòÍ8Ž 1loss/FRt2/assignment_loss‚0.14290094359883532 -loss/FRt2/detection_loss‚1.20029351685873 2loss/SRqqt1/detection_loss‚0.46834599134578087 0loss/FBt2/detection_loss‚0.19305819103220145 %loss/total_loss‚4.587489347545318 /loss/FRt1/detection_loss‚0.4682615667772305 0loss/SRqqt2/detection_loss‚0.499507347663816 3loss/SRqqt1/assignment_loss‚0.13790690183479642 2loss/FBt2/assignment_loss‚8.96752611843554e-06 trainer/global_step‚14249 1loss/SRqqt2/assignment_loss‚0.156737782175699 4loss/FBt1/assignment_loss‚3.6124024983440572e-09 /loss/FBt1/detection_loss‚0.3506485274916816 % _timestamp‚1.7843134123706417e+09 _runtime‚23596.587578776 _step‚1174 epoch‚35 0loss/FRt1/assignment_loss‚0.9698196076280251– ¸w[|”Î8‚(:‰ ÑìéÒ©¥ùö gpu.0.gpu‚90.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚302.696' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚60.5392 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0+mOÏ8‚(:Ä Ñìéҳºù$ disk./.usageGB‚566.3786964416504) disk.nvme0c0n1.out‚152426.76220703125$ memory_percent‚82.90002346038818 disk.sda3.out‚ 1400.3046875 disk.rbd3.in‚0 disk.sda.in‚ 108.921875& disk.rbd0.in‚1.759218604441423e+13) proc.memory.availableMB‚ 2801.66015625 cpu‚25.584090933343425$ disk.nvme0c0n1.in‚54952.08984375 disk.rbd1.in‚ 0.53564453125 disk.sda.out‚ 1400.3046875) disk./.usagePercent‚38.01014954157858 disk.rbd3.out‚ 0.4951171875* proc.memory.percent‚17.113566398620605 proc.cpu.threads‚17 disk.dm-0.in‚54952.08984375 disk.sda1.in‚0" disk.nvme0n1.in‚54952.08984375' disk.nvme0n1.out‚152426.76220703125 disk.sda2.in‚0 disk.sda2.out‚0$ disk.dm-0.out‚152426.76220703125 disk.drbd2165.in‚0 disk.rbd0.out‚0 disk.sda1.out‚0 disk.sda3.in‚ 108.921875 disk.drbd2165.out‚0 disk.dm-1.in‚ 108.921875# proc.memory.rssMB‚ 2803.88671875 network.recv‚ 5.878385e+06 disk.rbd1.out‚ 61.8037109375 disk.dm-1.out‚ 1400.3046875 network.sent‚ 2.7565578e+07˜û¬nžÐ8‚(:“ àìéÒÚÕðö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚283.318' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚56.66359999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0”e‰ÑIÑ8‚(:¾ àìéÒ÷›Êù& disk.nvme0n1.out‚152441.5830078125 disk.rbd1.in‚ 0.53564453125$ memory_percent‚82.90326595306396# proc.memory.rssMB‚ 2803.88671875 proc.cpu.threads‚17 disk.rbd3.in‚0 network.sent‚ 2.7629095e+07) disk./.usagePercent‚37.82662542050774 disk.sda2.in‚0& disk.rbd0.in‚1.759218604441423e+13 disk.rbd0.out‚0$ disk./.usageGB‚563.6440544128418$ disk.nvme0c0n1.in‚54952.30859375( disk.nvme0c0n1.out‚152441.5830078125 disk.dm-1.out‚ 1401.140625 disk.sda1.out‚0 disk.drbd2165.out‚0) proc.memory.availableMB‚ 2801.12890625 disk.sda3.out‚ 1401.140625 disk.drbd2165.in‚0 network.recv‚ 5.883346e+06 disk.sda2.out‚0# disk.dm-0.out‚152441.5830078125 cpu‚25.583963326579862" disk.nvme0n1.in‚54952.30859375 disk.rbd1.out‚ 61.8037109375 disk.sda.out‚ 1401.140625 disk.sda.in‚ 108.921875 disk.dm-1.in‚ 108.921875 disk.dm-0.in‚54952.30859375 disk.sda1.in‚0 disk.sda3.in‚ 108.921875 disk.rbd3.out‚ 0.4951171875* proc.memory.percent‚17.113566398620605ÒF¨/“Ò8‚(:ˆ ïìéÒÀþö gpu.0.gpu‚99.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚282.295' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚56.459 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0»(IÓ8‚(:¾ ïìéÒö‚ú" disk.nvme0n1.in‚54952.51171875* proc.memory.percent‚17.113566398620605( disk./.usagePercent‚37.8270475769317 disk.sda.out‚ 1402.3671875& disk.rbd0.in‚1.759218604441423e+13 disk.rbd1.in‚ 0.53564453125 cpu‚25.583835184753983 proc.cpu.threads‚17& disk.nvme0n1.out‚152455.7861328125 disk.sda2.out‚0 disk.dm-0.in‚54952.51171875 disk.sda1.in‚0 disk.rbd3.out‚ 0.4951171875( disk.nvme0c0n1.out‚152455.7861328125# proc.memory.rssMB‚ 2803.88671875 network.sent‚ 2.7632825e+07 disk.sda.in‚ 108.921875 disk.dm-1.out‚ 1402.3671875 disk.rbd0.out‚0 disk.sda2.in‚0 disk.rbd1.out‚ 61.8662109375 network.recv‚ 5.886771e+06 disk.sda3.in‚ 108.921875 disk.drbd2165.in‚0) proc.memory.availableMB‚ 2801.34765625 disk.sda1.out‚0$ disk./.usageGB‚563.6503448486328 disk.sda3.out‚ 1402.3671875 disk.rbd3.in‚0" memory_percent‚82.901930809021 disk.dm-1.in‚ 108.921875# disk.dm-0.out‚152455.7861328125 disk.drbd2165.out‚0$ disk.nvme0c0n1.in‚54952.51171875™¨¾°ŸÔ8‚(:” þìéÒúßñö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚39.0 gpu.0.powerWatts‚119.872' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.974400000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0á¼ëgQÕ8‚(:Æ þìéÒÞ“ù disk.sda2.out‚0 disk.rbd1.in‚ 0.53564453125 disk.dm-1.in‚ 108.921875 disk.dm-1.out‚ 1403.01953125 disk.drbd2165.in‚0) proc.memory.availableMB‚ 2801.34765625 disk.sda1.in‚0 network.recv‚ 5.890343e+06 disk.sda.in‚ 108.921875 disk.dm-0.in‚54952.93359375 disk.rbd3.in‚0) disk.nvme0c0n1.out‚152468.84423828125 disk.sda2.in‚0$ disk.nvme0c0n1.in‚54952.93359375* proc.memory.percent‚17.113566398620605 proc.cpu.threads‚17 network.sent‚ 2.7637946e+07 disk.rbd1.out‚ 61.8662109375 disk.sda.out‚ 1403.01953125 cpu‚25.583710089858233' disk.nvme0n1.out‚152468.84423828125& disk.rbd0.in‚1.759218604441423e+13 disk.rbd0.out‚0 disk.sda3.out‚ 1403.01953125 disk.rbd3.out‚ 0.4951171875" memory_percent‚82.901930809021# proc.memory.rssMB‚ 2803.88671875* disk./.usagePercent‚37.805732389630855 disk.sda3.in‚ 108.921875 disk.drbd2165.out‚0$ disk./.usageGB‚563.3327331542969" disk.nvme0n1.in‚54952.93359375$ disk.dm-0.out‚152468.84423828125 disk.sda1.out‚0ëÙ¦,”Ö8‚(:‰ íéÒêƒÓö gpu.0.gpu‚90.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚288.246' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚57.6492 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0cäüÎL×8‚(:Á íéÒªÒåø network.recv‚ 5.893892e+06$ disk./.usageGB‚563.3370895385742 disk.sda.in‚ 108.921875 disk.dm-1.out‚ 1403.83984375 disk.sda3.in‚ 108.921875 disk.drbd2165.out‚0# memory_percent‚82.9021692276001) proc.memory.availableMB‚ 2801.30859375 network.sent‚ 2.7644579e+07* disk./.usagePercent‚37.806024750234954 disk.dm-1.in‚ 108.921875 disk.sda1.out‚0 disk.rbd1.in‚ 0.53564453125' disk.nvme0n1.out‚152481.25537109375* proc.memory.percent‚17.113566398620605 disk.rbd0.out‚0 disk.dm-0.in‚ 55043.828125 proc.cpu.threads‚17 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd3.out‚ 0.4951171875" disk.nvme0c0n1.in‚ 55043.828125# proc.memory.rssMB‚ 2803.88671875 cpu‚25.583584038131463 disk.sda.out‚ 1403.83984375& disk.rbd0.in‚1.759218604441423e+13$ disk.dm-0.out‚152481.25537109375 disk.drbd2165.in‚0) disk.nvme0c0n1.out‚152481.25537109375 disk.sda3.out‚ 1403.83984375 disk.rbd3.in‚0 disk.sda1.in‚0 disk.nvme0n1.in‚ 55043.828125 disk.rbd1.out‚ 61.9443359375æyp4Ø8/ _wandbruntime‚23673  _runtime‚23673}Å»úÌÙ8Æ _runtime‚23673.915598009 _step‚1175 ' lr-AdamW/pg1‚0.00013465040772233267 ' lr-AdamW/pg2‚0.00013465040772233267 trainer/global_step‚14299 % _timestamp‚1.7843134896983418e+09¸yÑ^ÑÚ8Ë % _timestamp‚1.7843134896983418e+09 _runtime‚23673.915598009 _step‚1175 ' lr-AdamW/pg1‚0.00013465040772233267 ' lr-AdamW/pg2‚0.00013465040772233267 trainer/global_step‚14299— kBŸ4Û8/ _wandbruntime‚23673  _runtime‚23673x48e(Ü8‘P TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚40509809 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚4109477266890752 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 245796157 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13427128728735 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 121970375 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13451488108544 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 123825782 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 trainer/global_step‚14299 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1127274646345216 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚4109477293064192 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚4121129857690783 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 121970375 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚4121131227719073 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1109931677913088 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 123835755 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚77191291 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚4122927409609728 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚253 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚4107702728962048 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚4122927409609728 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 123835755 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1127275048972800 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚17343371059712 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚36681229 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1109931309821952 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚4109477266890752 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 123825782 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13450142718976 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚4122928781172736 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚1176 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 121970381 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 $ _timestamp‚1.784313489699512e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚77191038)"kl(Ý8–P LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚4109477293064192 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚4109477293064192 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚4121129857690783 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 121970375 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚4121131227719073 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1109931677913088 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 123835755 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚77191291 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚4122927409609728 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚253 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚4107702728962048 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚4122927409609728 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 123835755 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1127275048972800 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚17343371059712 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚36681229 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1109931309821952 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚4109477266890752 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 123825782 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13450142718976 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚4122928781172736 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚1176 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 121970381 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 $ _timestamp‚1.784313489699512e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚77191038 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚40509809 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚4109477266890752 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 245796157 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13427128728735 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 121970375 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13451488108544 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 123825782 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 trainer/global_step‚14299 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1127274646345216 JDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚14299 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 123844247 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚4123207706945536 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚241 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚17344564145664 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 _step‚1177 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13429396795415 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚4121410032963607 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚77196296 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1110008796229632 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17344529616384 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13452413151232 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1127353325846016 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 245822834 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 % _timestamp‚1.7843134910836065e+09 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚40512765 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 121978581 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚36683777 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 245812841 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚4121408662935205gEìp)'à8£N K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 123834260 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚4109755267620864 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13452413151232 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 245812841 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 245822834 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 % _timestamp‚1.7843134910836065e+09 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚40512765 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 121978581 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚36683777 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 245812841 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚4121408662935205 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚14299 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 123844247 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚4123207706945536 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚241 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚17344564145664 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 _step‚1177 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13429396795415 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚4121410032963607 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚77196296 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1110008796229632 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17344529616384 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13452413151232 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1127353325846016 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚77455454 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13497804039168 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 RCDeviceStatsMonitor.on_train_batch_start@ rý/allocated_bytes.all.current‚ 1371563008 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 122380675 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 124251534 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚4123377303396352 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚4121596843298816 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1131209382082048 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 124261507 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚36808379 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1113805630205952 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚4135070183712691 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1131208979454464 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13473365841843 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 _step‚1180 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13496458649600 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚40647075 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚4123377303396352 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚4136875133608960 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚4136873762045952 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚77455705 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚251 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚4123377329569792 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚4135071553740981ø(ó8˜P VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 124251534 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚4123377303396352 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚4121596843298816 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1131209382082048 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 124261507 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚36808379 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1113805630205952 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚4135070183712691 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1131208979454464 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13473365841843 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 _step‚1180 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13496458649600 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚40647075 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚4123377303396352 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚4136875133608960 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚4136873762045952 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚77455705 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚251 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚4123377329569792 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚4135071553740981 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 124251534 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚77455454 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13497804039168 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 122380675 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097œ —94ô8/ _wandbruntime‚23754  _runtime‚23754¤‚§×#'õ8N OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 122388887 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚4137154059381760 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚4135348988957113 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚4123655304126464 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚77460992 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 124269999 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚77460743 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 $ _timestamp‚1.784313569840626e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 246658886 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 124269999 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 122388881 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1131288060895744 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13497383685120 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚249 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚40649821 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17404541661696 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 _runtime‚23754.058019632 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚244 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1113883484704768 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 QDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 246648893 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1131287658275328 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13497383685120 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 246658886 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚40650065 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1113883116613632 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚17404576190976 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 124260012 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚4121874699649024 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13498729081856 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚36810927 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 122388881 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚4135350358985515 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 124260012 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚4123655330299904 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 trainer/global_step‚14349 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚4137154059381760 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 122388887 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13475633908523 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _step‚1181 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚4123655304126464 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 246648893 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13474289308089 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0—Яgkö8¢N C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚77460992 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 124269999 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚77460743 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 $ _timestamp‚1.784313569840626e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 246658886 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 124269999 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 122388881 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1131288060895744 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13497383685120 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚249 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚40649821 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17404541661696 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 _runtime‚23754.058019632 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚244 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1113883484704768 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 QDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 246648893 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1131287658275328 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13497383685120 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 246658886 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚40650065 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1113883116613632 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚17404576190976 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 124260012 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚4121874699649024 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13498729081856 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚36810927 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 122388881 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚4135350358985515 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 124260012 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚4123655330299904 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 trainer/global_step‚14349 U@DeviceStatsMonitor.on_train_batch_end/acti`åáZ½ ve_bytes.all.allocated‚4137154059381760 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 122388887 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13475633908523 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _step‚1181 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚4123655304126464 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 246648893 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13474289308089 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 122388887 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚4137154059381760 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚4135348988957113 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚4123655304126464 í¥¼4÷8/ _wandbruntime‚23754  _runtime‚23754‡ß:ø8Œ 1loss/FRt2/assignment_loss‚0.16349761330700935 epoch‚35 _step‚1182 3loss/SRqqt1/assignment_loss‚0.06724533393231735 3loss/FBt1/assignment_loss‚0.0003054378640047663 4loss/FBt2/assignment_loss‚1.3475826306049662e-08 0loss/FRt1/detection_loss‚0.41429681696771525 2loss/SRqqt2/detection_loss‚0.49748994358052606 /loss/SRqqt1/detection_loss‚0.50959663696369 /loss/FBt1/detection_loss‚0.3153725867262384 %loss/total_loss‚4.590634684993407 % _timestamp‚1.7843135699275365e+09 _runtime‚23754.144468312 /loss/FBt2/detection_loss‚0.1874973160011025 0loss/FRt1/assignment_loss‚1.0403291416257174 2loss/SRqqt2/assignment_loss‚0.1786435986960439 /loss/FRt2/detection_loss‚1.2163602458532161 trainer/global_step‚143493?Kzù8‘ 3loss/SRqqt1/assignment_loss‚0.06724533393231735 3loss/FBt1/assignment_loss‚0.0003054378640047663 4loss/FBt2/assignment_loss‚1.3475826306049662e-08 0loss/FRt1/detection_loss‚0.41429681696771525 2loss/SRqqt2/detection_loss‚0.49748994358052606 /loss/SRqqt1/detection_loss‚0.50959663696369 /loss/FBt1/detection_loss‚0.3153725867262384 %loss/total_loss‚4.590634684993407 % _timestamp‚1.7843135699275365e+09 _runtime‚23754.144468312 /loss/FBt2/detection_loss‚0.1874973160011025 0loss/FRt1/assignment_loss‚1.0403291416257174 2loss/SRqqt2/assignment_loss‚0.1786435986960439 /loss/FRt2/detection_loss‚1.2163602458532161 trainer/global_step‚14349 1loss/FRt2/assignment_loss‚0.16349761330700935 epoch‚35 _step‚1182ž «—-ú”ú8‚(:‰ çíéÒÕÜØö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚259.878' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚51.9756 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0E¶IPû8‚(:Å çíéÒÁ£Ñø disk.sda.in‚ 108.921875 disk.sda3.in‚ 108.921875) disk.nvme0c0n1.out‚152572.33056640625 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.sda2.out‚0 proc.cpu.threads‚17 disk.dm-1.in‚ 108.921875 disk.rbd1.in‚ 0.53564453125$ memory_percent‚82.71982669830322 network.recv‚ 5.91623e+06& disk.rbd0.in‚1.759218604441423e+13 disk.dm-0.in‚55135.26171875" disk.nvme0n1.in‚55135.26171875' disk.nvme0n1.out‚152572.33056640625* proc.memory.percent‚17.113566398620605 network.sent‚ 2.7732353e+07) disk./.usagePercent‚37.80844171736572 disk.sda2.in‚0 disk.rbd1.out‚ 62.1474609375 cpu‚25.582837127317973 disk.rbd0.out‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd3.out‚ 0.4951171875# proc.memory.rssMB‚ 2803.88671875$ disk.dm-0.out‚152572.33056640625 disk.sda3.out‚ 1405.08984375 disk.rbd3.in‚0$ disk.nvme0c0n1.in‚55135.26171875# disk./.usageGB‚563.373104095459 disk.sda.out‚ 1405.08984375 disk.dm-1.out‚ 1405.08984375) proc.memory.availableMB‚ 2831.18359375@-Xì”ü8‚(:‰ öíéҥâù gpu.0.gpu‚82.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚336.177' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚67.2354 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0cL—Rý8‚(:Ç öíéÒõÇ™û$ disk./.usageGB‚563.3797378540039 disk.sda1.out‚0$ memory_percent‚82.71691799163818) proc.memory.availableMB‚ 2831.66015625 cpu‚25.582726627905956 disk.sda3.out‚ 1405.29296875 disk.dm-1.out‚ 1405.29296875 disk.dm-0.in‚55135.46484375& disk.rbd0.in‚1.759218604441423e+13) disk.nvme0c0n1.out‚152584.44970703125 disk.rbd1.in‚ 0.53564453125* proc.memory.percent‚17.113566398620605$ disk.dm-0.out‚152584.44970703125 disk.drbd2165.out‚0 disk.rbd3.out‚ 0.4951171875 disk.rbd1.out‚ 62.1474609375# proc.memory.rssMB‚ 2803.88671875 disk.sda.in‚ 108.921875 network.recv‚ 5.921584e+06 disk.dm-1.in‚ 108.921875 disk.sda3.in‚ 108.921875 disk.drbd2165.in‚0$ disk.nvme0c0n1.in‚55135.46484375" disk.nvme0n1.in‚55135.46484375' disk.nvme0n1.out‚152584.44970703125) disk./.usagePercent‚37.80888691446776 disk.rbd0.out‚0 disk.sda2.in‚0 disk.sda2.out‚0 proc.cpu.threads‚17 network.sent‚ 2.7797439e+07 disk.sda1.in‚0 disk.rbd3.in‚0 disk.sda.out‚ 1405.292968754õdžþ8‚(:“ …îéÒ·Üö gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚279.34' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚55.867999999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Øn5Hÿ8‚(:½ …îéÒ»ÁÃú disk.sda3.out‚ 1405.6328125 disk.drbd2165.out‚0 disk.rbd3.in‚0# proc.memory.rssMB‚ 2803.88671875 disk.sda2.in‚0 disk.dm-0.in‚ 55267.2109375# disk.dm-0.out‚152597.2275390625# disk.nvme0c0n1.in‚ 55267.2109375$ memory_percent‚82.72144794464111* proc.memory.percent‚17.113566398620605 disk.sda.out‚ 1405.6328125 network.sent‚ 2.780116e+07 network.recv‚ 5.925016e+06 disk.sda2.out‚0 disk.rbd1.in‚ 0.53564453125 disk.sda.in‚ 108.921875 disk.sda1.out‚0 disk.drbd2165.in‚0* disk./.usagePercent‚37.809337743735554& disk.rbd0.in‚1.759218604441423e+13 disk.sda3.in‚ 108.921875( disk.nvme0c0n1.out‚152597.2275390625! disk.nvme0n1.in‚ 55267.2109375) proc.memory.availableMB‚ 2830.91796875 proc.cpu.threads‚17 disk.rbd0.out‚0$ disk./.usageGB‚563.3864555358887 disk.rbd1.out‚ 62.2255859375 disk.dm-1.in‚ 108.921875 disk.dm-1.out‚ 1405.6328125 disk.sda1.in‚0 disk.rbd3.out‚ 0.4951171875& disk.nvme0n1.out‚152597.2275390625 cpu‚25.58258154692436ïO±°ž€9‚(:“ ”îéÒ‰Á÷ gpu.0.gpu‚99.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚46.0 gpu.0.powerWatts‚334.587' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚66.91739999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ŠSÝJK9‚(:À ”îéÒŸ¿šû disk.drbd2165.in‚0# disk.nvme0c0n1.in‚ 55267.4296875& disk.nvme0n1.out‚152607.2373046875 disk.sda.in‚ 108.921875 disk.dm-1.in‚ 108.921875 disk.rbd3.out‚ 0.4951171875! disk.nvme0n1.in‚ 55267.4296875 cpu‚25.582475607638163 disk.sda3.out‚ 1405.80859375( disk.nvme0c0n1.out‚152607.2373046875 disk.sda2.out‚0 disk.rbd1.in‚ 0.53564453125$ memory_percent‚82.72004127502441 proc.cpu.threads‚17) disk./.usagePercent‚37.80965852117596 disk.rbd3.in‚0( proc.memory.availableMB‚ 2831.1484375 network.recv‚ 5.928407e+06& disk.rbd0.in‚1.759218604441423e+13 disk.dm-1.out‚ 1405.80859375 disk.dm-0.in‚ 55267.4296875 disk.sda2.in‚0# disk.dm-0.out‚152607.2373046875 disk.sda1.out‚0 disk.sda.out‚ 1405.80859375* proc.memory.percent‚17.113566398620605 network.sent‚ 2.7806307e+07 disk.drbd2165.out‚0 disk.rbd1.out‚ 62.2255859375# proc.memory.rssMB‚ 2803.88671875$ disk./.usageGB‚563.3912353515625 disk.rbd0.out‚0 disk.sda1.in‚0 disk.sda3.in‚ 108.921875)=OÅ”‚9‚(:‰ £îéÒѲûö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚280.614' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚56.1228 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0_åQMƒ9‚(: £îéÒÈÖ›ù disk.rbd1.in‚ 0.53564453125 disk.sda.out‚ 1406.1953125 disk.sda2.out‚0 disk.rbd1.out‚ 62.2880859375 disk.sda.in‚ 108.921875 disk.sda1.in‚0 disk.sda3.in‚ 108.921875 proc.cpu.threads‚17* disk./.usagePercent‚37.810129575038964 disk.dm-0.in‚55267.64453125 disk.sda3.out‚ 1406.1953125 disk.rbd0.out‚0" disk.nvme0n1.in‚55267.64453125 cpu‚25.58233381066164$ disk./.usageGB‚563.3982543945312 disk.dm-1.out‚ 1406.1953125 disk.sda1.out‚0) disk.nvme0c0n1.out‚152615.64990234375* proc.memory.percent‚17.113566398620605& disk.rbd0.in‚1.759218604441423e+13$ disk.nvme0c0n1.in‚55267.64453125# memory_percent‚82.7186107635498( proc.memory.availableMB‚ 2831.3828125 network.sent‚ 2.7811274e+07$ disk.dm-0.out‚152615.64990234375 disk.drbd2165.in‚0 disk.rbd3.in‚0 disk.rbd3.out‚ 0.4951171875 disk.sda2.in‚0 network.recv‚ 5.931007e+06 disk.dm-1.in‚ 108.921875 disk.drbd2165.out‚0' disk.nvme0n1.out‚152615.64990234375# proc.memory.rssMB‚ 2803.88671875 Ôp4„9/ _wandbruntime‚23831  _runtime‚23831ŽU¦DË…9Å ' lr-AdamW/pg1‚0.00013028735150529122 ' lr-AdamW/pg2‚0.00013028735150529122 trainer/global_step‚14399 % _timestamp‚1.7843136470009336e+09 _runtime‚23831.21805725 _step‚1183ý™FÛІ9Ê _step‚1183 ' lr-AdamW/pg1‚0.00013028735150529122 ' lr-AdamW/pg2‚0.00013028735150529122 trainer/global_step‚14399 % _timestamp‚1.7843136470009336e+09 _runtime‚23831.21805725Ÿ äæõ4‡9/ _wandbruntime‚23831  _runtime‚23831¨ ¡(ˆ9”P \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13544119969792 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 122790981 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚4150821486045184 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚77721015 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚253 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚17463369901568 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚4149010509734599 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13542774580224 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚77720762 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚36935534 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚4149011879762889 _step‚1184 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 124687259 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 247478240 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1135143723119104 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚17463404438016 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚36935529 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚4135490932207616 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 124687259 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚40785481 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 247468261 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 122790981 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚4137277366075392 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1117680318681088 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13519602954951 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚4137277339901952 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 _runtime‚23831.219294055 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚40785233 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13520947555273 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚14399 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1117679950589952 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚4137277339901952 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 122790975 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚4150820114482176 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚4137277366075392 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚4150821486045184 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13544119969792 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136Æ67CÃ%‰9™P WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚17463404438016 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚36935529 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚4135490932207616 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 124687259 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚40785481 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 247468261 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 122790981 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚4137277366075392 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1117680318681088 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13519602954951 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚4137277339901952 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 _runtime‚23831.219294055 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚40785233 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13520947555273 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚14399 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1117679950589952 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚4137277339901952 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 122790975 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚4150820114482176 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚4137277366075392 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚4150821486045184 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13544119969792 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13544119969792 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 122790981 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚4150821486045184 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚77721015 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚253 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚17463369901568 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚4149010509734599 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13542774580224 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚77720762 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚36935534 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZÍWh{\EDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚4149011879762889 _step‚1184 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 124687259 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 247478240 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1135143723119104 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1  =~”¤4Š9/ _wandbruntime‚23832  _runtime‚23832b ,M$'‹9žN L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 247494938 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1117757436997632 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 247484945 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 122799187 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 124685764 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 122799181 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚36938077 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚244 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚4137555340632064 % _timestamp‚1.7843136483886063e+09 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚40788206 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚36938072 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1135221999410688 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚77726283 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1135222402031104 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚4137555340632064 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚77726034 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17464562413056 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13545045012480 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚4151099040247808 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 124695751 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚4151100411817984ÑåÁr)'Œ9£N YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 124685764 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 122799181 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚36938077 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚244 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚4137555340632064 % _timestamp‚1.7843136483886063e+09 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚40788206 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚36938072 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1135221999410688 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚77726283 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1135222402031104 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚4137555340632064 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚77726034 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17464562413056 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13545045012480 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚4151099040247808 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 124695751 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚4151100411817984 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 247494938 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1117757436997632 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 247484945 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 122799187 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440¡ ®¿¿49/ _wandbruntime‚23832  _runtime‚23832n|Ž9ˆ 0loss/FRt1/assignment_loss‚0.9052923307382108 0loss/FRt2/assignment_loss‚0.1073754718944208 2loss/SRqqt2/assignment_loss‚0.1347008637471138 3loss/FBt1/assignment_loss‚0.0013445403536775302 /loss/FBt1/detection_loss‚0.5399608244435512 epoch‚35 $ _timestamp‚1.784313648474407e+09 /loss/FRt1/detection_loss‚0.4652465534036988 trainer/global_step‚14399 _step‚1186 3loss/SRqqt1/assignment_loss‚0.10981941664933625 3loss/FBt2/assignment_loss‚0.0002713143317415759 /loss/FRt2/detection_loss‚1.2451034423256127 1loss/SRqqt1/detection_loss‚0.5531645528255601 2loss/SRqqt2/detection_loss‚0.43722539577261293 /loss/FBt2/detection_loss‚0.1775825881276636 _runtime‚23832.691331061 #loss/total_loss‚4.6770872946132Ù,[€9 /loss/FRt2/detection_loss‚1.2451034423256127 1loss/SRqqt1/detection_loss‚0.5531645528255601 2loss/SRqqt2/detection_loss‚0.43722539577261293 /loss/FBt2/detection_loss‚0.1775825881276636 _runtime‚23832.691331061 #loss/total_loss‚4.6770872946132 0loss/FRt1/assignment_loss‚0.9052923307382108 0loss/FRt2/assignment_loss‚0.1073754718944208 2loss/SRqqt2/assignment_loss‚0.1347008637471138 3loss/FBt1/assignment_loss‚0.0013445403536775302 /loss/FBt1/detection_loss‚0.5399608244435512 epoch‚35 $ _timestamp‚1.784313648474407e+09 /loss/FRt1/detection_loss‚0.4652465534036988 trainer/global_step‚14399 _step‚1186 3loss/SRqqt1/assignment_loss‚0.10981941664933625 3loss/FBt2/assignment_loss‚0.0002713143317415759¢ ¼4ž9‚(:“ ²îéÒ·š ù gpu.0.gpu‚97.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚47.0 gpu.0.powerWatts‚332.561' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚66.51219999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ØÝÿM‘9‚(: ²îéÒ„à¢û disk.drbd2165.out‚0 disk.rbd3.in‚0 disk.rbd1.in‚ 0.53564453125 disk.rbd1.out‚ 62.2880859375$ memory_percent‚82.71746635437012( disk./.usagePercent‚37.8092939664472 disk.dm-1.out‚ 1406.48828125 disk.sda2.in‚0* proc.memory.percent‚17.113566398620605 disk.rbd0.out‚0! disk.nvme0n1.in‚ 55268.0546875 network.recv‚ 5.934585e+06 disk.sda.out‚ 1406.48828125 disk.dm-0.in‚ 55268.0546875 disk.sda1.out‚0 disk.rbd3.out‚ 0.4951171875 proc.cpu.threads‚17$ disk./.usageGB‚563.3858032226562 disk.sda.in‚ 108.921875# disk.nvme0c0n1.in‚ 55268.0546875) disk.nvme0c0n1.out‚152630.88818359375 disk.sda2.out‚0 disk.dm-1.in‚ 108.921875& disk.rbd0.in‚1.759218604441423e+13' disk.nvme0n1.out‚152630.88818359375 network.sent‚ 2.7817051e+07( proc.memory.availableMB‚ 2831.5703125# proc.memory.rssMB‚ 2803.88671875 cpu‚25.582361243329597$ disk.dm-0.out‚152630.88818359375 disk.sda1.in‚0 disk.sda3.in‚ 108.921875 disk.sda3.out‚ 1406.48828125 disk.drbd2165.in‚0W¯”’9‚(:‰ ÁîéÒ¢£ðö gpu.0.gpu‚98.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚46.0 gpu.0.powerWatts‚342.514' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚68.5028 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ªf@“9‚(:µ ÁîéÒ‰ ‡ú$ disk./.usageGB‚563.3913040161133% disk.nvme0n1.out‚152640.689453125 disk.dm-1.out‚ 1406.76953125 disk.sda3.in‚ 108.921875 disk.sda3.out‚ 1406.76953125 disk.nvme0c0n1.in‚ 55377.4375 cpu‚25.583914433433087 disk.rbd0.out‚0 disk.sda1.out‚0 disk.drbd2165.out‚0 disk.nvme0n1.in‚ 55377.4375 disk.rbd1.in‚ 0.53564453125$ memory_percent‚82.71844387054443) proc.memory.availableMB‚ 2831.41015625* proc.memory.percent‚17.113566398620605 disk.sda.out‚ 1406.76953125 disk.rbd3.in‚0 disk.rbd1.out‚ 62.3662109375& disk.rbd0.in‚1.759218604441423e+13" disk.dm-0.out‚152640.689453125 disk.sda2.out‚0 network.recv‚ 5.938874e+06) disk./.usagePercent‚37.80966312931157' disk.nvme0c0n1.out‚152640.689453125# proc.memory.rssMB‚ 2803.88671875 disk.dm-1.in‚ 108.921875 proc.cpu.threads‚17 network.sent‚ 2.7879254e+07 disk.sda.in‚ 108.921875 disk.dm-0.in‚ 55377.4375 disk.sda1.in‚0 disk.drbd2165.in‚0 disk.rbd3.out‚ 0.4951171875 disk.sda2.in‚0©p¢j4”9/ _wandbruntime‚23859  _runtime‚23859ly¥U€•9û{ *jet/accuracy_1_of_3‚0.8940445780754089 particle/accuracy_2_of_2‚1 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.6875 *loss/val_total_loss‚10.641703396246095 6 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.989932656288147 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.17075924575328827 +particle/sensitivity‚0.5914403200149536 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.15102319419384003 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7195802330970764 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.5712853670120239 jet/accuracy_5_of_5‚0 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9913343787193298 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5856624245643616 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6805392503738403 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.13858246803283691 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.12910370528697968 +particle/specificity‚0.8990112543106079 trainer/global_step‚14399 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6467224955558777 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.025756021961569786 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.15007030963897705 'particle/f_score‚0.6032717823982239 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9877625703811646 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7250073552131653 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9869948625564575 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0024456202518194914 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7361111044883728 particle/accuracy_2_of_4‚1 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9919310212135315 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6227883696556091 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9888888597488403 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 particle/accuracy_3_of_4‚1 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.8970557451248169 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5210123062133789 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7142857313156128 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.2870370149612427 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.4490741193294525 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6008891463279724 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.09666667133569717 ,jet/accuracy_3_of_3‚0.043976519256830215 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9651955366134644 - Purity/0FRt2SRqqt1FBt/FBt_purity‚0.890625 8"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.655545175075531 particle/accuracy_1_of_1‚1 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6631652116775513 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.34907880425453186 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.028024451807141304 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9768514037132263 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.2222222238779068 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6457530856132507 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.05000000074505806 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9897676110267639 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 *jet/accuracy_2_of_2‚0.1212790459394455 *jet/accuracy_2_of_4‚0.7073877453804016 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.19288651645183563 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6587109565734863 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.022731194272637367 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.2083333283662796 jet/accuracy_4_of_5‚0 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.11965911835432053 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6145762801170349 particle/accuracy_3_of_3‚1 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.22767899930477142 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6253620386123657 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.23845522105693817 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7538990378379822 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.12316514551639557 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7309851050376892 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.019718095660209656 particle/accuracy_3_of_5‚1 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6794451475143433 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 epoch‚35 _step‚1187 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9806118011474609 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9944936633110046 _runtime‚23859.077836737 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.5610156059265137 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3429461717605591 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.1509123593568802 ,jet/accuracy_4_of_4‚0.013899499550461769 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.2870370149612427 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9848144054412842 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9529496431350708 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.11027032136917114 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 jet/accuracy_2_of_5‚1 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5985441207885742 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 )jet/accuracy_1_of_4‚0.954925000667572 7"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.65593421459198 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.7175359725952148 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.01652807928621769 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9708871245384216 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8888888955116272 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_2‚1 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1ñ5o!1 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5097518563270569 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.14393770694732666 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.027373041957616806 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4571710228919983 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.4891666769981384 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5985441207885742 *"Purity/2FRt*SRqqt2FBt/event_purity‚0.5 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.13807746767997742 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.5822663903236389 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.004086548928171396 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.18672722578048706 *jet/accuracy_1_of_1‚0.2814231514930725 jet/accuracy_3_of_5‚1 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.12350466102361679 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.46207156777381897 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.24019934237003326 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.0914696529507637 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.12670835852622986 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.6506425142288208 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.8461538553237915 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 8"Purity/*FRt0SRqqt2FBt/event_purity‚0.891805112361908 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.20791269838809967 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.7974257469177246 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9755224585533142 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9259259700775146 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8147995471954346 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6865019798278809 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.008848981000483036 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.3134480118751526 ."Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.46875 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8147995471954346 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4543052911758423 particle/accuracy_1_of_3‚1 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8055050373077393 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.8564814329147339 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6469953656196594 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.026170780882239342 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9915008544921875 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.13055843114852905 (particle/accuracy‚0.8329573273658752 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.1055942177772522 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.05000000074505806 6validation_average_jet_accuracy‚0.3237082064151764 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.3076923191547394 particle/accuracy_1_of_5‚1 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9975056648254395 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9259259700775146 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 particle/accuracy_5_of_5‚1 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.1027422621846199 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3499999940395355 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.20289242267608643 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6347337961196899 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6787518262863159 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 particle/accuracy_4_of_5‚1 8"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.615181565284729 5 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.13604836165905 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6481581926345825 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.019718095660209656 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.47638294100761414 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.3076923191547394 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.027474621310830116 +jet/accuracy_3_of_4‚0.04390200972557068 /particle/accuracy_4_of_4‚0.9366528391838074 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7209372520446777 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7484137415885925 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.5442708730697632 $ _timestamp‚1.784313674860018e+09 - Purity/0FRt2SRqqt*FBt/FBt_purity‚0.890625 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7115835547447205 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 particle/accuracy_1_of_4‚1 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9882096648216248 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7361111044883728 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9971743822097778 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9960516691207886 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.2222222238779068 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.13858246803283691 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7500433325767517 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 particle/accuracy_2_of_5‚1 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9933173060417175 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.13279995322227478 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9937976002693176 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4305555522441864 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.5714285969734192 6 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.980441153049469 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.21195518970489502 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6274626851081848 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.26001641154289246 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.13678492605686188 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6867058277130127 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.10829509049654007 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.14290247857570648 *jet/accuracy_1_of_2‚0.7175661325454712 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 jet/accuracy_1_of_5‚1 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8332271575927734 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.7174603939056396 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7302083969116211 *jet/accuracy_2_of_3‚0.4915095269680023 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.6180996894836426 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7295117378234863 particle/accuracy_2_of_3‚1 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7193878889083862 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9960516691207886 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 ("Purity/2FRt2SRqqt*FBt/event_purity‚0ÇõC>–9€| 8"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.655545175075531 particle/accuracy_1_of_1‚1 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6631652116775513 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.34907880425453186 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.028024451807141304 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9768514037132263 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.2222222238779068 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6457530856132507 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.05000000074505806 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9897676110267639 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 *jet/accuracy_2_of_2‚0.1212790459394455 *jet/accuracy_2_of_4‚0.7073877453804016 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.19288651645183563 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6587109565734863 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.022731194272637367 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.2083333283662796 jet/accuracy_4_of_5‚0 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.11965911835432053 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6145762801170349 particle/accuracy_3_of_3‚1 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.22767899930477142 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6253620386123657 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.23845522105693817 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7538990378379822 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.12316514551639557 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7309851050376892 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.019718095660209656 particle/accuracy_3_of_5‚1 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6794451475143433 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 epoch‚35 _step‚1187 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9806118011474609 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9944936633110046 _runtime‚23859.077836737 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.5610156059265137 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3429461717605591 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.1509123593568802 ,jet/accuracy_4_of_4‚0.013899499550461769 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.2870370149612427 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9848144054412842 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9529496431350708 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.11027032136917114 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 jet/accuracy_2_of_5‚1 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5985441207885742 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 )jet/accuracy_1_of_4‚0.954925000667572 7"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.65593421459198 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.7175359725952148 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.01652807928621769 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9708871245384216 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8888888955116272 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_2‚1 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5097518563270569 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.14393770694732666 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.027373041957616806 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4571710228919983 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.4891666769981384 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5985441207885742 *"Purity/2FRt*SRqqt2FBt/event_purity‚0.5 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.13807746767997742 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.5822663903236389 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.004086548928171396 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.18672722578048706 *jet/accuracy_1_of_1‚0.2814231514930725 jet/accuracy_3_of_5‚1 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.12350466102361679 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.46207156777381897 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.24019934237003326 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.0914696529507637 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.12670835852622986 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.6506425142288208 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.8461538553237915 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 8"Purity/*FRt0SRqqt2FBt/event_purity‚0.891805112361908 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.20791269838809967 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.7974257469177246 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9755224585533142 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9259259700775146 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8147995471954346 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6865019798278809 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.008848981000483036 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.3134480118751526 ."Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.46875 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8147995471954346 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4543052911758423 particle/accuracy_1_of_3‚1 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8055050373077393 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.8564814329147339 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6469953656196594 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.026170780882239342 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9915008544921875 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.13055843114852905 (particle/accuracy‚0.8329573273658752 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.1055942177772522 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.05000000074505806 6validation_average_jet_accuracy‚0.3237082064151764 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.3076923191547394 particle/accuracy_1_of_5‚1 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9975056648254395 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9259259700775146 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 particle/accuracy_5_of_5‚1 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.1027422621846199 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3499999940395355 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.20289242267608643 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6347337961196899 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6787518262863159 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 particle/accuracy_4_of_5‚1 8"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.615181565284729 5 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.13604836165905 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6481581926345825 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.019718095660209656 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.47638294100761414 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.3076923191547394 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.027474621310830116 +jet/accuracy_3_of_4‚0.04390200972557068 /particle/accuracy_4_of_4‚0.9366528391838074 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7209372520446777 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7484137415885925 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.5442708730697632 $ _timestamp‚1.784313674860018e+09 - Purity/0FRt2SRqqt*FBt/FBt_purity‚0.890625 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7115835547447205 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 particle/accuracy_1_of_4‚1 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9882096648216248 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7361111044883728 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9971743822097778 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9960516691207886 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.2222222238779068 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.13858246803283691 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7500433325767517 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 particle/accuracy_2_of_5‚1 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9933173060417175 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.13279995322227478 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9937976002693176 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4305555522441864 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.5714285969734192 6 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.980441153049469 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.21195518970489502 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6274626851081848 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.26001641154289246 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.13678492605686188 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6867058277130127 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.10829509049654007 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.14290247857570648 *jet/accuracy_1_of_2‚0.7175661325454712 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 jet/accuracy_1_of_5‚1 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8332271575927734 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.7174603939056396 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7302083969116211 *jet/accuracy_2_of_3‚0.4915095269680023 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.6180996894836426 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7295117378234863 particle/accuracy_2_of_3‚1 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7193878889083862 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9960516691207886 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 *jet/accuracy_1_of_3‚0.8940445780754089 particle/accuracy_2_of_2‚1 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.6875 *loss/val_total_loss‚10.641703396246095 6 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.989932656288147 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.17075924575328827 +particle/sensitivity‚0.5914403200149536 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.15102319419384003 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7195802330970764 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.5712853670120239 jet/accuracy_5_of_5‚0 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9913343787193298 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5856624245643616 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6805392503738403 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.13858246803283691 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.12910370528697968 +particle/specificity‚0.8990112543106079 trainer/global_step‚14399 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6467224955558777 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.025756021961569786 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.15007030963897705 'particle/f_score‚0.6032717823982239 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9877625703811646 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7250073552131653 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9869948625564575 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0024456202518194914 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7361111044883728 particle/accuracy_2_of_4‚1 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9919310212135315 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6227883696556091 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9888888597488403 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 particle/accuracy_3_of_4‚1 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.8970557451248169 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5210123062133789 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7142857313156128 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.2870370149612427 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.4490741193294525 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6008891463279724 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.09666667133569717 ,jet/accuracy_3_of_3‚0.043976519256830215 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9651955366134644 - Purity/0FRt2SRqqt1FBt/FBt_purity‚0.890625£ y‘èš”—9‚(:‰ ÐîéÒê„ó÷ gpu.0.gpu‚90.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚261.766' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.3532 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0v ü2<˜9‚(:± ÐîéÒàËàú" disk.nvme0c0n1.in‚ 55377.671875 disk.sda.in‚ 108.921875 disk.rbd0.out‚0 disk.sda2.out‚0 network.recv‚ 5.942739e+06) disk./.usagePercent‚37.81013648724239$ disk./.usageGB‚563.3983573913574 disk.nvme0n1.in‚ 55377.671875) proc.memory.percent‚17.12357997894287 cpu‚25.584979995955376 proc.cpu.threads‚17 disk.drbd2165.in‚0# disk.nvme0n1.out‚152653.8359375 disk.dm-0.in‚ 55377.671875 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.in‚0 disk.rbd1.out‚ 62.3662109375 disk.sda3.out‚ 1406.95703125% disk.nvme0c0n1.out‚152653.8359375$ memory_percent‚82.72762298583984& proc.memory.availableMB‚ 2829.90625 disk.dm-1.in‚ 108.921875 network.sent‚ 2.7884441e+07 disk.sda3.in‚ 108.921875 disk.rbd3.in‚0 disk.rbd1.in‚ 0.53564453125# proc.memory.rssMB‚ 2805.52734375 disk.drbd2165.out‚0 disk.rbd3.out‚ 0.4951171875 disk.sda.out‚ 1406.95703125 disk.dm-1.out‚ 1406.95703125& disk.rbd0.in‚1.759218604441423e+13 disk.dm-0.out‚152653.8359375±]]”™9‚(:‰ ßîéÒöü†÷ gpu.0.gpu‚49.0 gpu.0.memory‚15, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚39.0 gpu.0.powerWatts‚314.241' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚62.8482 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0­²°‰@š9‚(:µ ßîéÒí³¨ú disk.sda1.out‚0) proc.memory.availableMB‚ 2829.66015625# proc.memory.rssMB‚ 2805.52734375 disk.sda2.in‚0# disk.dm-0.out‚152666.2744140625 proc.cpu.threads‚17 network.sent‚ 2.7942608e+07 network.recv‚ 5.947613e+06 disk.drbd2165.out‚0 disk.nvme0c0n1.in‚ 55378.0625 disk.rbd1.out‚ 62.4287109375 disk.sda.out‚ 1407.33203125) disk./.usagePercent‚37.81192393184645# disk./.usageGB‚563.424991607666$ memory_percent‚82.72912502288818 disk.rbd0.out‚0) proc.memory.percent‚17.12357997894287 disk.sda3.out‚ 1407.33203125 disk.rbd3.out‚ 0.4951171875( disk.nvme0c0n1.out‚152666.2744140625 disk.dm-1.in‚ 108.921875 disk.dm-0.in‚ 55378.0625 disk.sda1.in‚0 disk.sda3.in‚ 108.921875 disk.drbd2165.in‚0 disk.rbd1.in‚ 0.53564453125 cpu‚25.58485440801211 disk.sda.in‚ 108.921875 disk.dm-1.out‚ 1407.33203125& disk.rbd0.in‚1.759218604441423e+13 disk.rbd3.in‚0 disk.nvme0n1.in‚ 55378.0625& disk.nvme0n1.out‚152666.2744140625 disk.sda2.out‚0¢pI”›9‚(:‰ îîéÒþ¶Ùö gpu.0.gpu‚90.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚273.366' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.6732 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0!¯HO3œ9‚(:¨ îîéÒìáÀú network.recv‚ 5.950886e+06 disk.rbd3.out‚ 0.4951171875' disk.nvme0n1.out‚152676.58154296875 disk.sda.in‚ 108.921875 disk.dm-1.out‚1407.5& disk.rbd0.in‚1.759218604441423e+13$ memory_percent‚82.73155689239502# proc.memory.rssMB‚ 2805.52734375 disk.nvme0n1.in‚ 55378.28125 disk.sda2.out‚0 disk.rbd0.out‚0 disk.sda1.out‚0 disk.sda3.in‚ 108.921875) proc.memory.availableMB‚ 2829.26171875 cpu‚25.584717178694618 disk.dm-0.in‚ 55378.28125 network.sent‚ 2.7947684e+07 disk.rbd1.out‚ 62.4287109375 disk.dm-1.in‚ 108.921875) disk./.usagePercent‚37.81230871117042 disk.drbd2165.out‚0) disk.nvme0c0n1.out‚152676.58154296875 disk.sda2.in‚0 disk.sda.out‚1407.5$ disk.dm-0.out‚152676.58154296875) proc.memory.percent‚17.12357997894287 proc.cpu.threads‚17! disk.nvme0c0n1.in‚ 55378.28125 disk.rbd1.in‚ 0.53564453125 disk.sda3.out‚1407.5$ disk./.usageGB‚563.4307250976562 disk.drbd2165.in‚0 disk.rbd3.in‚0 disk.sda1.in‚0©àɲ”9‚(:‰ ýîéÒšöéö gpu.0.gpu‚35.0 gpu.0.memory‚11, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚325.841' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚65.1682 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0FM´Lž9‚(:Á ýîéÒ°Šù disk.rbd3.out‚ 0.4951171875' disk.nvme0c0n1.out‚152684.408203125 proc.cpu.threads‚17" disk.dm-0.out‚152684.408203125 disk.sda1.in‚0& disk.rbd0.in‚1.759218604441423e+13 disk.rbd0.out‚0# proc.memory.rssMB‚ 2805.52734375* disk./.usagePercent‚37.812786933244375 disk.rbd1.in‚ 0.53564453125 disk.dm-1.in‚ 108.921875 disk.dm-1.out‚ 1407.99609375 disk.sda1.out‚0$ memory_percent‚82.73017406463623 cpu‚25.584615024412287 disk.sda2.in‚0 network.recv‚ 5.954164e+06$ disk./.usageGB‚563.4378509521484" disk.nvme0n1.in‚55392.44140625) proc.memory.availableMB‚ 2829.48828125 disk.drbd2165.out‚0 disk.rbd1.out‚ 62.5068359375 network.sent‚ 2.7952804e+07 disk.drbd2165.in‚0 disk.rbd3.in‚0$ disk.nvme0c0n1.in‚55392.44140625% disk.nvme0n1.out‚152684.408203125 disk.sda.out‚ 1407.99609375 disk.sda2.out‚0 disk.sda3.in‚ 108.921875 disk.sda3.out‚ 1407.99609375) proc.memory.percent‚17.12357997894287 disk.sda.in‚ 108.921875 disk.dm-0.in‚55392.44140625R*ÊŸŸ9‚(:” ŒïéÒñ¬Ôö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚271.506' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚54.301199999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0)¥o$O 9‚(:Ä ŒïéÒú# proc.memory.rssMB‚ 2805.52734375 network.recv‚ 5.957674e+06 disk.sda.in‚ 108.921875 disk.rbd0.out‚0 disk.dm-0.in‚55479.72265625$ disk.dm-0.out‚152698.75048828125) proc.memory.availableMB‚ 2829.30859375 disk.sda.out‚ 1408.23046875 disk.dm-1.in‚ 108.921875 disk.sda1.out‚0 disk.rbd3.out‚ 0.4951171875' disk.nvme0n1.out‚152698.75048828125( disk./.usagePercent‚37.8130675175019 disk.drbd2165.out‚0 disk.dm-1.out‚ 1408.23046875 disk.drbd2165.in‚0 disk.rbd1.in‚ 0.53564453125 disk.sda3.out‚ 1408.23046875 cpu‚25.584475517046737" disk.nvme0n1.in‚55479.72265625 network.sent‚ 2.7959346e+07$ disk./.usageGB‚563.4420318603516 disk.sda2.in‚0 disk.rbd1.out‚ 62.5068359375 disk.sda1.in‚0 disk.rbd3.in‚0& disk.rbd0.in‚1.759218604441423e+13 disk.sda3.in‚ 108.921875$ disk.nvme0c0n1.in‚55479.72265625) disk.nvme0c0n1.out‚152698.75048828125# memory_percent‚82.7312707901001) proc.memory.percent‚17.12357997894287 proc.cpu.threads‚17 disk.sda2.out‚0£¬)b4¡9/ _wandbruntime‚23936  _runtime‚23936¡W9'Ê¢9Ä % _timestamp‚1.7843137521383514e+09 _runtime‚23936.355510902 _step‚1188 & lr-AdamW/pg1‚0.0001281265284894219 & lr-AdamW/pg2‚0.0001281265284894219 trainer/global_step‚14449¢ööÏ£9É _step‚1188 & lr-AdamW/pg1‚0.0001281265284894219 & lr-AdamW/pg2‚0.0001281265284894219 trainer/global_step‚14449 % _timestamp‚1.7843137521383514e+09 _runtime‚23936.355510902¤ þ64¤9/ _wandbruntime‚23936  _runtime‚23936)ÉvhÈ¥9”P F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 248495301 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 QCDeviceStatsMonitor.on_train_batch_start/alloÄËßR'cation.small_pool.freed‚ 125162020 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚17530866218496 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13594933122560 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 123333287 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚78027932 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚40956575 trainer/global_step‚14449 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 _runtime‚23936.356743837 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚4168600290964992 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚4155005357842432 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚4168601662528000 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 248505280 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚37071597 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚4168601662528000 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚78028177 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚245 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1139133216999936 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚4168600290964992 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1139133617530368 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚240 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 400530432 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 125162020 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚17530866218496 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13594933122560 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 123333287 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚78027932 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚40956575 trainer/global_step‚14449 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 _runtime‚23936.356743837 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚4168600290964992 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚4155005357842432 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚4168601662528000 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 248505280 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚37071597 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚4168601662528000 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚78028177 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚245 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1139133216999936 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚4168600290964992¥ <á[4§9/ _wandbruntime‚23937  _runtime‚23937ûoJ$'¨9žN ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 125170498 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚32432128 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13595858158080 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13573937266176 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 125180485 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚4168879216730624 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚78033456 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1121680237719552 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚4153488786399232 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚4168880588300800 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 248521978 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 125180485 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17532045517312 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 123341493 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚37074140 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 248521978 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚78033214 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1139212315668992 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚4153488811827200 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 400523264 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13572592665742 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13595858158080 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 123341487 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13597203554816 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚4167062749093376 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚40959311 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13597203554816 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 trainer/global_step‚14449 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚237 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 123341493 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚4155283384745984 _step‚1190 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154iƒÐ{ ©9£N L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 248521978 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚78033214 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1139212315668992 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚4153488811827200 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 400523264 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13572592665742 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 E?DeviceStatsMonitor.on_train_batch_end/active.large_poW¥âŠol.current‚6 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13595858158080 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 123341487 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13597203554816 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚4167062749093376 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚40959311 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13597203554816 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 trainer/global_step‚14449 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚237 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 123341493 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚4155283384745984 _step‚1190 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 125170498 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚32432128 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13595858158080 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13573937266176 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 125180485 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚4168879216730624 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚78033456 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1121680237719552 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚4153488786399232 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚4168880588300800 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 248521978 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 125180485 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17532045517312 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 123341493 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚37074140 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008¦ ‚Ž„è4ª9/ _wandbruntime‚23937  _runtime‚23937w-ˆ²«9Ž epoch‚36 3loss/SRqqt2/assignment_loss‚0.15265558972962226 3loss/FBt2/assignment_loss‚2.793018045130942e-06 /loss/FRt2/detection_loss‚1.1704995469247426 0loss/FBt1/detection_loss‚0.17247486033465975 trainer/global_step‚14449 _runtime‚23937.82857479 _step‚1191 /loss/FBt2/detection_loss‚0.2234847454762634 %loss/total_loss‚4.249030700694023 0loss/FRt1/assignment_loss‚0.9200984611176849 1loss/FRt2/assignment_loss‚0.10178809218546923 3loss/SRqqt1/assignment_loss‚0.09305998695821209 4loss/FBt1/assignment_loss‚0.00022047808378733436 1loss/SRqqt1/detection_loss‚0.4733771106788072 2loss/SRqqt2/detection_loss‚0.46914139753942286 /loss/FRt1/detection_loss‚0.4722276386473065 % _timestamp‚1.7843137536116452e+09žY³¬9“ 1loss/FRt2/assignment_loss‚0.10178809218546923 3loss/SRqqt1/assignment_loss‚0.09305998695821209 4loss/FBt1/assignment_loss‚0.00022047808378733436 1loss/SRqqt1/detection_loss‚0.4733771106788072 2loss/SRqqt2/detection_loss‚0.46914139753942286 /loss/FRt1/detection_loss‚0.4722276386473065 % _timestamp‚1.7843137536116452e+09 epoch‚36 3loss/SRqqt2/assignment_loss‚0.15265558972962226 3loss/FBt2/assignment_loss‚2.793018045130942e-06 /loss/FRt2/detection_loss‚1.1704995469247426 0loss/FBt1/detection_loss‚0.17247486033465975 trainer/global_step‚14449 _runtime‚23937.82857479 _step‚1191 /loss/FBt2/detection_loss‚0.2234847454762634 %loss/total_loss‚4.249030700694023 0loss/FRt1/assignment_loss‚0.9200984611176849§ œ·”­9‚(:‰ ›ïéÒš§›ø gpu.0.gpu‚93.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚46.0 gpu.0.powerWatts‚341.826' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚68.3652 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0§ …WN®9‚(:à ›ïéÒÚ–ºù) proc.memory.percent‚17.12357997894287) disk./.usagePercent‚37.81356647618614 disk.sda.out‚ 1408.65234375) disk.nvme0c0n1.out‚152706.20263671875 disk.sda2.in‚0 disk.sda3.out‚ 1408.65234375 disk.rbd3.in‚0 disk.dm-1.in‚ 108.921875$ disk.dm-0.out‚152706.20263671875' proc.memory.availableMB‚ 2828.984375$ disk./.usageGB‚563.4494667053223 disk.rbd1.in‚ 0.53564453125 disk.rbd0.out‚0 disk.dm-0.in‚55479.94140625 disk.sda3.in‚ 108.921875 disk.rbd3.out‚ 0.4951171875" disk.nvme0n1.in‚55479.94140625 disk.sda2.out‚0 disk.sda1.out‚0$ memory_percent‚82.73324966430664 cpu‚25.584362203976262 proc.cpu.threads‚17 network.sent‚ 2.7963084e+07 network.recv‚ 5.96119e+06 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.dm-1.out‚ 1408.65234375& disk.rbd0.in‚1.759218604441423e+13 disk.sda1.in‚0$ disk.nvme0c0n1.in‚55479.94140625' disk.nvme0n1.out‚152706.20263671875 disk.rbd1.out‚ 62.5693359375 disk.sda.in‚ 108.921875# proc.memory.rssMB‚ 2805.52734375´Ø§JŸ¯9‚(:” ªïéÒ¦²´÷ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚248.911' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚49.782199999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0*ÌRõQ°9‚(:Æ ªïéÒ¸úÑú disk.rbd0.out‚0 disk.dm-0.in‚55480.36328125$ disk.dm-0.out‚152716.08544921875) disk./.usagePercent‚37.81404777035051 disk.sda1.out‚0 disk.sda3.in‚ 108.921875$ memory_percent‚82.73355960845947 disk.rbd3.out‚ 0.4951171875" disk.nvme0n1.in‚55480.36328125 network.sent‚ 2.8027947e+07 disk.sda1.in‚0 disk.sda2.out‚0 disk.drbd2165.out‚0& disk.rbd0.in‚1.759218604441423e+13) proc.memory.percent‚17.12357997894287 disk.dm-1.out‚ 1408.79296875) disk.nvme0c0n1.out‚152716.08544921875$ disk./.usageGB‚563.4566383361816 disk.sda3.out‚ 1408.79296875 disk.drbd2165.in‚0 disk.sda.out‚ 1408.79296875 proc.cpu.threads‚17' disk.nvme0n1.out‚152716.08544921875 disk.rbd1.in‚ 0.53564453125# proc.memory.rssMB‚ 2805.52734375 cpu‚25.584233804722224 disk.rbd3.in‚0$ disk.nvme0c0n1.in‚55480.36328125 disk.sda2.in‚0 disk.rbd1.out‚ 62.5693359375 disk.sda.in‚ 108.921875 disk.dm-1.in‚ 108.921875 network.recv‚ 5.966114e+06) proc.memory.availableMB‚ 2828.93359375Îlïž±9‚(:“ ¹ïéÒ²Âïö gpu.0.gpu‚99.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚332.561' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚66.51219999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0u/³TB²9‚(:· ¹ïéÒîéáú& proc.memory.availableMB‚ 2828.90625 disk.drbd2165.out‚0 disk.rbd1.in‚ 0.53564453125) disk./.usagePercent‚37.81432553852516$ disk./.usageGB‚563.4607772827148) disk.nvme0c0n1.out‚152727.98681640625 disk.sda2.in‚0) proc.memory.percent‚17.12360382080078 disk.rbd3.out‚ 0.4951171875$ disk.nvme0c0n1.in‚55480.56640625 disk.drbd2165.in‚0 network.sent‚ 2.8031681e+07 proc.memory.rssMB‚ 2805.53125 disk.sda.in‚ 108.921875 disk.dm-1.out‚ 1409.09375& disk.rbd0.in‚1.759218604441423e+13 proc.cpu.threads‚17" disk.nvme0n1.in‚55480.56640625 disk.sda2.out‚0 disk.sda1.out‚0 disk.sda3.out‚ 1409.09375 network.recv‚ 5.969183e+06' disk.nvme0n1.out‚152727.98681640625 disk.rbd0.out‚0 disk.dm-0.in‚55480.56640625$ disk.dm-0.out‚152727.98681640625 disk.rbd3.in‚0 disk.sda3.in‚ 108.921875 cpu‚25.584107916884324 disk.rbd1.out‚ 62.7099609375 disk.sda.out‚ 1409.09375 disk.dm-1.in‚ 108.921875 disk.sda1.in‚0$ memory_percent‚82.73372650146484øì’³9‚(:‡ ÈïéÒ¾Œóö gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚283.36' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚56.672 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ßøH´9‚(:½ ÈïéÒ¼Žú network.recv‚ 5.972884e+06 disk.sda2.in‚0 disk.sda.out‚ 1409.28515625# disk.dm-0.out‚152738.6923828125 disk.sda1.in‚0 disk.sda2.out‚0 disk.dm-1.out‚ 1409.28515625 disk.dm-0.in‚55654.58203125 disk.sda3.in‚ 108.921875$ memory_percent‚82.73000717163086' proc.memory.availableMB‚ 2829.515625 disk.rbd1.in‚ 0.53564453125 disk.dm-1.in‚ 108.921875 disk.rbd3.in‚0( disk.nvme0c0n1.out‚152738.6923828125" disk.nvme0n1.in‚55654.58203125& disk.nvme0n1.out‚152738.6923828125# disk./.usageGB‚563.467529296875 disk.sda3.out‚ 1409.28515625 disk.drbd2165.out‚0* disk./.usagePercent‚37.814778671860765 proc.cpu.threads‚17 disk.sda.in‚ 108.921875 disk.rbd1.out‚ 62.7099609375 disk.drbd2165.in‚0$ disk.nvme0c0n1.in‚55654.58203125 proc.memory.rssMB‚ 2805.53125& disk.rbd0.in‚1.759218604441423e+13 disk.rbd3.out‚ 0.4951171875) proc.memory.percent‚17.12360382080078 cpu‚25.583973285019095 disk.sda1.out‚0 disk.rbd0.out‚0 network.sent‚ 2.803693e+07ö0œ”µ9‚(:‰ ×ïéÒ‘÷Îö gpu.0.gpu‚99.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚284.178' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚56.8356 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0.ã†E¶9‚(:º ×ïéÒžÂÂù disk.dm-0.in‚55655.00390625 disk.drbd2165.out‚0$ disk.nvme0c0n1.in‚55655.00390625' disk.nvme0n1.out‚152746.08740234375 disk.sda2.out‚0 network.recv‚ 5.976163e+06$ disk./.usageGB‚563.4746551513672& disk.rbd0.in‚1.759218604441423e+13) proc.memory.availableMB‚ 2829.01953125 disk.sda.out‚ 1409.53125 disk.dm-1.in‚ 108.921875 disk.drbd2165.in‚0 disk.rbd3.in‚0" disk.nvme0n1.in‚55655.00390625$ memory_percent‚82.73303508758545) proc.memory.percent‚17.12360382080078 proc.cpu.threads‚17 disk.sda.in‚ 108.921875$ disk.dm-0.out‚152746.08740234375 disk.sda3.out‚ 1409.53125) disk.nvme0c0n1.out‚152746.08740234375 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.in‚0 proc.memory.rssMB‚ 2805.53125 cpu‚25.583870385625577 network.sent‚ 2.8042225e+07 disk.rbd3.out‚ 0.4951171875 disk.rbd1.in‚ 0.53564453125 disk.rbd1.out‚ 62.7724609375) disk./.usagePercent‚37.81525689393473 disk.rbd0.out‚0 disk.sda3.in‚ 108.921875 disk.dm-1.out‚ 1409.53125~?NŸ·9‚(:” æïéÒ´¢¬÷ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚284.911' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚56.982200000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0jŒ/K¸9‚(:À æïéÒÜŒÂù$ memory_percent‚82.73274898529053$ disk./.usageGB‚563.4768142700195 disk.dm-1.in‚ 108.921875$ disk.nvme0c0n1.in‚55655.00390625( disk.nvme0c0n1.out‚152759.4814453125 disk.sda2.out‚0 disk.dm-0.in‚55655.00390625 disk.sda1.out‚0 disk.sda.out‚ 1409.76953125& disk.rbd0.in‚1.759218604441423e+13 disk.sda2.in‚0 disk.rbd1.in‚ 0.53564453125 disk.rbd1.out‚ 62.7724609375 proc.cpu.threads‚17" disk.nvme0n1.in‚55655.00390625) proc.memory.availableMB‚ 2829.06640625) proc.memory.percent‚17.12360382080078 network.recv‚ 5.980321e+06 disk.sda3.in‚ 108.921875 disk.drbd2165.in‚0 disk.rbd3.in‚0 proc.memory.rssMB‚ 2805.53125 disk.drbd2165.out‚0 network.sent‚ 2.8047565e+07* disk./.usagePercent‚37.815401794199104 disk.sda.in‚ 108.921875 disk.dm-1.out‚ 1409.76953125 disk.rbd0.out‚0 cpu‚25.58372443551819# disk.dm-0.out‚152759.4814453125 disk.sda1.in‚0 disk.sda3.out‚ 1409.76953125 disk.rbd3.out‚ 0.4951171875& disk.nvme0n1.out‚152759.4814453125mË:V4¹9/ _wandbruntime‚24015  _runtime‚24015‡£D9̺9Æ trainer/global_step‚14499 % _timestamp‚1.7843138309505415e+09 _runtime‚24015.167698384 _step‚1192 ' lr-AdamW/pg1‚0.00012597969421267188 ' lr-AdamW/pg2‚0.00012597969421267188©©à!Ñ»9Ë trainer/global_step‚14499 % _timestamp‚1.7843138309505415e+09 _runtime‚24015.167698384 _step‚1192 ' lr-AdamW/pg1‚0.00012597969421267188 ' lr-AdamW/pg2‚0.00012597969421267188¨ 0ú4¼9/ _wandbruntime‚24015  _runtime‚24015~¢RÐ(½9“P MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 trainer/global_step‚14499 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 249341332 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 123743587 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 125587772 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1143067593659904 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13619250912926 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚4182548014964224 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚37198747 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 $ _timestamp‚1.784313830951596e+09 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚4182548014964224 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚4168905394348032 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 123743581 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚4182546643401216 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13641249053184 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚17590924591616 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚266 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚4180722899842460 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚4167105018957824 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 249331353 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚4168905420521472 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚17590890055168 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚4182546643401216 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13642594442752 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 125597745 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13641249053184 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚4167104993529856 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚78290617 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1125477071695872 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 125587772 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 123743587 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 125597745 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322ìÖ^ƒ ¾9˜P LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13642594442752 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 _runtime‚24015.168957081 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚78290351 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚41091865 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1125476703604736 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚4180722899842460 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚4167105018957824 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 249331353 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚4168905420521472 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚17590890055168 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚4182546643401216 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13642594442752 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 125597745 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13641249053184 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚4167104993529856 JDeviceStatsMonitor.on_train_batch_start/act ñ7›ive.small_pool.peak‚12097 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚78290617 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1125477071695872 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 125587772 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 123743587 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 125597745 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 trainer/global_step‚14499 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 249341332 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 123743587 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 125587772 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1143067593659904 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13619250912926 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚4182548014964224 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚37198747 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 $ _timestamp‚1.784313830951596e+09 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚4182548014964224 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚4168905394348032 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 123743581 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚4182546643401216 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13641249053184 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚17590924591616 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚266 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚4167382875308032 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚250 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚4181001705086882 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 249348037 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 249358030 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 123751793 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 123751793 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 _runtime‚24016.554238438 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚4181003075115284 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚78295599 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _step‚1194 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚4167382900736000 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚4182826940737024 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚4169183395078144 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚41094559 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1143146679713280 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1125554558103552 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚4182826940737024 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚4169183395078144 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13642174088704 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 % _timestamp‚1.7843138323368506e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13618829778850 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008_/)'Á9£N UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 % _timestamp‚1.7843138323368506e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13618829778850 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 125606237 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 125596250 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚37201295 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚17592121609728 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚4167382875308032 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚250 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚4181001705086882 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 249348037 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 249358030 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 123751793 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 123751793 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 _runtime‚24016.554238438 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚4181003075115284 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚78295599 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _step‚1194 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚4167382900736000 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚4182826940737024 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚4169183395078144 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚41094559 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1143146679713280 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1125554558103552 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚4182826940737024 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚4169183395078144 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13642174088704 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736ª üŒ‘¿4Â9/ _wandbruntime‚24016  _runtime‚240160k’Ã9 /loss/FRt2/detection_loss‚1.2404917020988824 0loss/FBt2/detection_loss‚0.13479953661353117 _runtime‚24016.640339957 3loss/SRqqt1/assignment_loss‚0.06051697688297835 % _timestamp‚1.7843138324234354e+09 %loss/total_loss‚4.311418042387015 0loss/FRt1/assignment_loss‚0.8762440062047355 1loss/FRt2/assignment_loss‚0.13281103872958525 4loss/FBt1/assignment_loss‚0.00015051461343283524 1loss/SRqqt1/detection_loss‚0.5631560990667193 1loss/SRqqt2/detection_loss‚0.4723314341644143 _step‚1195 3loss/SRqqt2/assignment_loss‚0.13940286743212604 0loss/FRt1/detection_loss‚0.41428492720583476 0loss/FBt1/detection_loss‚0.27664344073604424 trainer/global_step‚14499 epoch‚36 3loss/FBt2/assignment_loss‚0.0005854986387312781#¥§.Ä9• 3loss/SRqqt1/assignment_loss‚0.06051697688297835 % _timestamp‚1.7843138324234354e+09 %loss/total_loss‚4.311418042387015 0loss/FRt1/assignment_loss‚0.8762440062047355 1loss/FRt2/assignment_loss‚0.13281103872958525 4loss/FBt1/assignment_loss‚0.00015051461343283524 1loss/SRqqt1/detection_loss‚0.5631560990667193 1loss/SRqqt2/detection_loss‚0.4723314341644143 _step‚1195 3loss/SRqqt2/assignment_loss‚0.13940286743212604 0loss/FRt1/detection_loss‚0.41428492720583476 0loss/FBt1/detection_loss‚0.27664344073604424 trainer/global_step‚14499 epoch‚36 3loss/FBt2/assignment_loss‚0.0005854986387312781 /loss/FRt2/detection_loss‚1.2404917020988824 0loss/FBt2/detection_loss‚0.13479953661353117 _runtime‚24016.640339957« EÁD ’Å9‚(:‡ õïéÒËÊàö gpu.0.gpu‚64.0 gpu.0.memory‚21, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚287.91' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚57.582 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0éxÿJKÆ9‚(:À õïéÒˆú disk.sda2.in‚0 disk.sda.out‚ 1410.22265625$ disk.dm-0.out‚152769.79052734375 disk.drbd2165.in‚0 disk.drbd2165.out‚0 proc.memory.rssMB‚ 2805.53125 disk.sda1.in‚0# disk./.usageGB‚563.483341217041$ disk.nvme0c0n1.in‚55655.42578125) disk.nvme0c0n1.out‚152769.79052734375 disk.dm-1.out‚ 1410.22265625& disk.rbd0.in‚1.759218604441423e+13' disk.nvme0n1.out‚152769.79052734375 disk.sda.in‚ 108.921875 disk.dm-0.in‚55655.42578125 proc.cpu.threads‚17" disk.nvme0n1.in‚55655.42578125 disk.rbd1.out‚ 62.8505859375 disk.rbd3.out‚ 0.4951171875( proc.memory.availableMB‚ 2829.0390625 cpu‚25.583608648057318 network.sent‚ 2.805773e+07* disk./.usagePercent‚37.815839823090194 disk.sda1.out‚0 disk.sda3.in‚ 108.921875 disk.sda3.out‚ 1410.22265625) proc.memory.percent‚17.12360382080078 network.recv‚ 5.990762e+06 disk.dm-1.in‚ 108.921875 disk.rbd3.in‚0# memory_percent‚82.7329158782959 disk.sda2.out‚0 disk.rbd1.in‚ 0.53564453125 disk.rbd0.out‚0uBžÇ9‚(:“ „ðéÒÞßö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚279.79' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚55.958000000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0šÁN¼5È9‚(:ª „ðéÒùƪú network.recv‚ 5.995381e+06 disk.dm-1.out‚ 1410.5703125 disk.drbd2165.in‚0) proc.memory.percent‚17.12360382080078# disk./.usageGB‚563.487964630127 cpu‚25.58347286415281 disk.sda1.out‚0 network.sent‚ 2.8119683e+07 disk.rbd3.in‚0' disk.nvme0n1.out‚152779.71337890625 proc.cpu.threads‚17 disk.sda3.in‚ 108.921875 disk.nvme0c0n1.in‚ 55747.375) disk.nvme0c0n1.out‚152779.71337890625 disk.sda2.in‚0 disk.rbd1.in‚ 0.53564453125 proc.memory.rssMB‚ 2805.53125 disk.sda.in‚ 108.921875 disk.sda3.out‚ 1410.5703125 disk.nvme0n1.in‚ 55747.375 disk.dm-1.in‚ 108.921875$ disk.dm-0.out‚152779.71337890625 disk.rbd3.out‚ 0.4951171875 disk.sda2.out‚0$ memory_percent‚82.73391723632812$ proc.memory.availableMB‚2828.875) disk./.usagePercent‚37.81615010422169& disk.rbd0.in‚1.759218604441423e+13 disk.rbd0.out‚0 disk.sda1.in‚0 disk.sda.out‚ 1410.5703125 disk.dm-0.in‚ 55747.375 disk.drbd2165.out‚0 disk.rbd1.out‚ 62.8505859375«Ã6žÉ9‚(:“ “ðéÒ‘Èûö gpu.0.gpu‚89.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚291.631' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚58.32619999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0þö¦ÖÊ9‚(:· “ðéүʘù disk.dm-0.in‚ 55747.796875 network.sent‚ 2.812482e+07 disk.drbd2165.out‚0( disk.nvme0c0n1.out‚152788.1083984375& disk.rbd0.in‚1.759218604441423e+13" disk.nvme0c0n1.in‚ 55747.796875 disk.nvme0n1.in‚ 55747.796875 disk.drbd2165.in‚0 disk.rbd3.out‚ 0.4951171875 disk.sda.in‚ 108.921875 disk.sda1.out‚0 disk.sda.out‚ 1410.9609375) proc.memory.percent‚17.12360382080078 disk.sda3.in‚ 108.921875 disk.sda3.out‚ 1410.9609375& disk.nvme0n1.out‚152788.1083984375 disk.rbd0.out‚0) proc.memory.availableMB‚ 2828.62109375 proc.cpu.threads‚17* disk./.usagePercent‚37.816753257972366$ disk./.usageGB‚563.4969520568848 disk.rbN*ªld3.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd1.in‚ 0.53564453125 disk.rbd1.out‚ 62.9130859375 disk.dm-1.in‚ 108.921875# disk.dm-0.out‚152788.1083984375 proc.memory.rssMB‚ 2805.53125 cpu‚25.583361103640268 network.recv‚ 5.998595e+06 disk.dm-1.out‚ 1410.9609375 disk.sda1.in‚0$ memory_percent‚82.73546695709229²ÂÔªžË9‚(:“ ¢ðéÒÝ̈÷ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚280.405' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚56.08099999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0]¡¬?Ì9‚(:´ ¢ðéÒùÁÀù disk.sda3.in‚ 108.921875 disk.rbd3.out‚ 0.4951171875& disk.rbd0.in‚1.759218604441423e+13$ memory_percent‚82.73565769195557( disk.nvme0c0n1.out‚152804.1259765625 disk.rbd1.out‚ 62.9130859375) proc.memory.availableMB‚ 2828.58984375) disk./.usagePercent‚37.81707070731482 disk.sda2.in‚0 disk.rbd1.in‚ 0.53564453125 disk.dm-1.in‚ 108.921875 disk.rbd0.out‚0 disk.sda1.in‚0 disk.sda1.out‚0$ disk./.usageGB‚563.5016822814941 proc.memory.rssMB‚ 2805.53125 disk.drbd2165.in‚0 disk.sda2.out‚0 disk.dm-0.in‚ 55748.015625 disk.sda3.out‚ 1411.203125 cpu‚25.583246275392074 network.recv‚ 6.002239e+06 disk.rbd3.in‚0 disk.sda.out‚ 1411.203125 disk.dm-1.out‚ 1411.203125 disk.drbd2165.out‚0& disk.nvme0n1.out‚152804.1259765625) proc.memory.percent‚17.12360382080078 proc.cpu.threads‚17 network.sent‚ 2.8131357e+07" disk.nvme0c0n1.in‚ 55748.015625 disk.nvme0n1.in‚ 55748.015625 disk.sda.in‚ 108.921875# disk.dm-0.out‚152804.1259765625êîµkŸÍ9‚(:” ±ðéÒèÎ÷ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚271.258' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚54.251599999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Á_O®?Î9‚(:´ ±ðéÒ׎†ù disk.rbd3.out‚ 0.4951171875 network.recv‚ 6.005609e+06 disk.dm-1.out‚ 1411.4921875& disk.rbd0.in‚1.759218604441423e+13 disk.sda3.in‚ 108.921875) disk.nvme0c0n1.out‚152812.52099609375* disk./.usagePercent‚37.817514368371654 disk.sda.in‚ 108.921875 disk.sda.out‚ 1411.4921875$ memory_percent‚82.73582458496094' disk.nvme0n1.out‚152812.52099609375 disk.rbd1.out‚ 62.9912109375 disk.dm-1.in‚ 108.921875 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.rbd1.in‚ 0.53564453125! disk.nvme0c0n1.in‚ 55748.21875 disk.nvme0n1.in‚ 55748.21875 disk.sda2.in‚0) proc.memory.percent‚17.12360382080078 disk.rbd0.out‚0 disk.rbd3.in‚0 disk.sda2.out‚0% proc.memory.availableMB‚ 2828.5625 cpu‚25.583112561067303$ disk./.usageGB‚563.5082931518555 disk.sda1.out‚0 disk.sda3.out‚ 1411.4921875 proc.memory.rssMB‚ 2805.53125 proc.cpu.threads‚17 network.sent‚ 2.8136541e+07 disk.dm-0.in‚ 55748.21875$ disk.dm-0.out‚152812.52099609375 disk.sda1.in‚0Þá'€4Ï9/ _wandbruntime‚24093  _runtime‚24093t,jÌÐ9Æ _runtime‚24093.692616435 _step‚1196 ' lr-AdamW/pg1‚0.00012384698656219505 ' lr-AdamW/pg2‚0.00012384698656219505 trainer/global_step‚14549 % _timestamp‚1.7843139094754941e+09Æ5+xÑÑ9Ë ' lr-AdamW/pg2‚0.00012384698656219505 trainer/global_step‚14549 % _timestamp‚1.7843139094754941e+09 _runtime‚24093.692616435 _step‚1196 ' lr-AdamW/pg1‚0.00012384698656219505¬ ¥R¶&4Ò9/ _wandbruntime‚24093  _runtime‚24093h!sá(Ó9”P G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7843139094765348e+09 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚4196494367400448 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚4196494367400448 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚4182805430853632 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 126013524 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚263 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚4196492995837440 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚4196492995837440 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚78553215 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1129351392079872 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚4182805457027072 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097‘Š3(Ô9™P aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚4182805457027072 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7843139094765348e+09 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚4196494367400448 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚4196494367400448 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚4182805430853632 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 126013524 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚263 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚4196492995837440 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚4196492995837440 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚78553215 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1129351392079872 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008­ l"Ã"4Õ9/ _wandbruntime‚24095  _runtime‚240958|%; Ö9žN HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 250194082 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚4181276989644800 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 250194082 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚4194942031108790 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚254 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 124162087 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13688490019328 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 QDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 250184089 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 _step‚1198 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 126022002 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 126031989 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚249 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 trainer/global_step‚14549 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13689835416064 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚41229753 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚4181276964216832 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚4183083431583744 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13666411492392 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚4183083431583744 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚17651983665152 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 _runtime‚24095.079835896 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚37328440 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1129428510396416 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚37328445 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 126031989 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚4183083457757184 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 124162093 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1147080459532288 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 REDeviceStatsMonitor.on_trais)%n_batch_end/active_bytes.large_pool.current‚26173440 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13689835416064 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 250184089 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 124162093 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚78558193 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13688490019328 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚4183083457757184 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚4196771921603072 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 % _timestamp‚1.7843139108624249e+09 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚4196773293173248 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 126022002 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚41230002 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995ͬ )'×9£N XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚4183083431583744 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13666411492392 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚4183083431583744 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚17651983665152 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 _runtime‚24095.079835896 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚37328440 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1129428510396416 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚37328445 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 126031989 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚4183083457757184 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 124162093 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1147080459532288 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13689835416064 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 250184089 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 124162093 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚78558193 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13688490019328 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚4183083457757184 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚4196771921603072 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 % _timestamp‚1.7843139108624249e+09 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚4196773293173248 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 126022002 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚41230002 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 250194082 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚4181276989644800 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 250194082 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚4194942031108790 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚254 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 124162087 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13688490019328 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 QDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 250184089 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 _step‚1198 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 126022002 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 126031989 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚249 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 trainer/global_step‚14549 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13689835416064 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚41229753 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚4181276964216832 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0® 즑4Ø9/ _wandbruntime‚24095  _runtime‚24095þáÌÙ9Ž 4loss/FBt1/assignment_loss‚0.00018845012677736072 3loss/FBt2/assignment_loss‚0.0001754582588191268 1loss/SRqqt1/detection_loss‚0.4782174328607677 1loss/SRqqt2/detection_loss‚0.4438754061264624 0loss/FBt2/detection_loss‚0.11065165981066721 % _timestamp‚1.7843139109476774e+09 0loss/FRt1/assignment_loss‚0.9969483602614629 3loss/SRqqt1/assignment_loss‚0.07006144621221304 %loss/total_loss‚4.468548016374044 epoch‚36 trainer/global_step‚14549 _runtime‚24095.164653367 _step‚1199 3loss/SRqqt2/assignment_loss‚0.14097146428928137 /loss/FRt1/detection_loss‚0.4032779235754376 1loss/FRt2/assignment_loss‚0.09482365223033004 /loss/FRt2/detection_loss‚1.1719012329374496 /loss/FBt1/detection_loss‚0.5574555296843755"ôUÚ9“ 1loss/SRqqt1/detection_loss‚0.4782174328607677 1loss/SRqqt2/detection_loss‚0.4438754061264624 0loss/FBt2/detection_loss‚0.11065165981066721 % _timestamp‚1.7843139109476774e+09 0loss/FRt1/assignment_loss‚0.9969483602614629 3loss/SRqqt1/assignment_loss‚0.07006144621221304 %loss/total_loss‚4.468548016374044 epoch‚36 trainer/global_step‚14549 _runtime‚24095.164653367 _step‚1199 3loss/SRqqt2/assignment_loss‚0.14097146428928137 /loss/FRt1/detection_loss‚0.4032779235754376 1loss/FRt2/assignment_loss‚0.09482365223033004 /loss/FRt2/detection_loss‚1.1719012329374496 /loss/FBt1/detection_loss‚0.5574555296843755 4loss/FBt1/assignment_loss‚0.00018845012677736072 3loss/FBt2/assignment_loss‚0.0001754582588191268¯ Ü=”Û9‚(:‰ ÀðéÒàŒâö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚334.442' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚66.8884 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0R—pèHÜ9‚(:½ ÀðéÒõ€ú disk.sda.in‚ 108.921875 disk.sda.out‚ 1412.0078125 disk.dm-0.in‚55901.44140625 disk.rbd3.out‚ 0.4951171875 disk.sda2.in‚0 disk.sda1.in‚0$ memory_percent‚82.73570537567139 disk.sda2.out‚0 disk.rbd1.out‚ 62.9912109375# disk.dm-0.out‚152823.9736328125( disk.nvme0c0n1.out‚152823.9736328125 proc.memory.rssMB‚ 2805.53125 network.sent‚ 2.8141667e+07* disk./.usagePercent‚37.817962637564115 disk.rbd3.in‚0 proc.cpu.threads‚17 network.recv‚ 6.009008e+06" disk.nvme0n1.in‚55901.44140625 disk.dm-1.in‚ 108.921875 disk.drbd2165.in‚0 disk.rbd1.in‚ 0.53564453125 disk.sda1.out‚0 disk.sda3.out‚ 1412.0078125 disk.drbd2165.out‚0$ disk.nvme0c0n1.in‚55901.44140625& disk.rbd0.in‚1.759218604441423e+13 disk.rbd0.out‚0 disk.sda3.in‚ 108.921875) proc.memory.availableMB‚ 2828.58203125) proc.memory.percent‚17.12360382080078 cpu‚25.58299634584928$ disk./.usageGB‚563.5149726867676 disk.dm-1.out‚ 1412.0078125& disk.nvme0n1.out‚152823.9736328125uÍà”Ý9‚(:‰ ÏðéÒ¹áæö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚287.327' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚57.4654 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0«æ³´GÞ9‚(:¼ ÏðéÒ­×ù) proc.memory.percent‚17.12360382080078 disk.sda2.in‚0 disk.drbd2165.in‚0 cpu‚25.58285255846606 disk.rbd1.in‚ 0.53564453125 disk.dm-1.in‚ 108.921875 disk.rbd0.out‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda3.out‚ 1412.20703125$ disk.nvme0c0n1.in‚55901.66015625' disk.nvme0c0n1.out‚152837.482421875 network.recv‚ 6.013304e+06& disk.rbd0.in‚1.759218604441423e+13" disk.dm-0.out‚152837.482421875 network.sent‚ 2.8203618e+07 disk.rbd1.out‚ 63.1162109375 disk.sda.in‚ 108.921875 disk.dm-1.out‚ 1412.20703125 disk.sda3.in‚ 108.921875 disk.rbd3.in‚0$ memory_percent‚82.73718357086182$ disk./.usageGB‚563.5192985534668 disk.dm-0.in‚55901.66015625% disk.nvme0n1.out‚152837.482421875) disk./.usagePercent‚37.81825295010794 disk.sda2.out‚0 disk.rbd3.out‚ 0.4951171875" disk.nvme0n1.in‚55901.66015625 proc.memory.rssMB‚ 2805.53125 proc.cpu.threads‚17 disk.sda.out‚ 1412.20703125 disk.drbd2165.out‚0) proc.memory.availableMB‚ 2828.33984375ª;e8”ß9‚(:‰ ÞðéÒÖ—ôö gpu.0.gpu‚31.0 gpu.0.memory‚10, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚336.177' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚67.2354 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Üp…Là9‚(:Á ÞðéÒ¯•ýø disk.sda3.out‚ 1412.55859375 disk.rbd3.out‚ 0.4951171875) disk.nvme0c0n1.out‚152849.39501953125 cpu‚25.58274060380874$ disk./.usageGB‚563.5265197753906 disk.sda.in‚ 108.921875 disk.sda1.out‚0" disk.nvme0n1.in‚55901.86328125 disk.sda2.in‚0 disk.rbd1.out‚ 63.1162109375 proc.memory.rssMB‚ 2805.53125$ disk.dm-0.out‚152849.39501953125 disk.sda.out‚ 1412.55859375 disk.dm-1.in‚ 108.921875$ memory_percent‚82.73568153381348 disk.sda3.in‚ 108.921875( proc.memory.availableMB‚ 2828.5859375 disk.rbd1.in‚ 0.53564453125 disk.drbd2165.in‚0 disk.rbd3.in‚0) proc.memory.percent‚17.12360382080078 disk.drbd2165.out‚0 proc.cpu.threads‚17 network.sent‚ 2.8208739e+07 network.recv‚ 6.017046e+06) disk./.usagePercent‚37.81873757237026 disk.dm-1.out‚ 1412.55859375' disk.nvme0n1.out‚152849.39501953125 disk.sda2.out‚0 disk.sda1.in‚0$ disk.nvme0c0n1.in‚55901.86328125 disk.dm-0.in‚55901.86328125& disk.rbd0.in‚1.759218604441423e+13 disk.rbd0.out‚0,:ø·žá9‚(:“ íðéÒÔ»åö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚279.255' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚55.85099999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0c¥z‘Jâ9‚(:¿ íðéÒ‘Ëù$ disk.dm-0.out‚152862.71630859375 disk.drbd2165.in‚0 disk.rbd3.out‚ 0.4951171875$ disk.nvme0c0n1.in‚55902.28515625& proc.memory.availableMB‚ 2828.34375 cpu‚25.582605153621703 disk.rbd1.out‚ 63.1943359375# memory_percent‚82.7371597290039) proc.memory.percent‚17.12360382080078 proc.cpu.threads‚17" disk.nvme0n1.in‚55902.28515625' disk.nvme0n1.out‚152862.71630859375 disk.sda2.in‚0 disk.dm-1.out‚ 1412.81640625 disk.sda3.in‚ 108.921875 network.recv‚ 6.020229e+06 disk.dm-0.in‚55902.28515625 disk.rbd3.in‚0) disk.nvme0c0n1.out‚152862.71630859375 proc.memory.rssMB‚ 2805.53125$ disk./.usageGB‚563.5333786010742 disk.sda.in‚ 108.921875& disk.rbd0.in‚1.759218604441423e+13 disk.sda1.in‚0 disk.drbd2165.out‚0) disk./.usagePercent‚37.81919787391682 disk.sda2.out‚0 disk.sda.out‚ 1412.81640625 disk.dm-1.in‚ 108.921875 disk.sda3.out‚ 1412.81640625 network.sent‚ 2.8215295e+07 disk.rbd1.in‚ 0.53564453125 disk.rbd0.out‚0 disk.sda1.out‚0‡à­•ã9‚(:Š üðéÒ¬¥„÷ gpu.0.gpu‚100.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚333.378' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚66.6756 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0öÌË]Gä9‚(:¼ üðéÒèâÈù$ memory_percent‚82.72829055786133 disk.rbd1.in‚ 0.53564453125 disk.sda.in‚ 108.921875 disk.dm-1.in‚ 108.921875 disk.dm-1.out‚ 1413.01171875 disk.rbd0.out‚0 network.sent‚ 2.8219025e+07$ disk./.usageGB‚563.5177116394043# disk.nvme0c0n1.in‚ 56047.8515625) disk.nvme0c0n1.out‚152873.63916015625 disk.sda3.out‚ 1413.01171875' proc.memory.availableMB‚ 2829.796875 disk.drbd2165.in‚0! disk.nvme0n1.in‚ 56047.8515625 disk.sda2.in‚0 disk.sda1.out‚0 cpu‚25.58250233228245 disk.rbd3.in‚0& disk.rbd0.in‚1.759218604441423e+13$ disk.dm-0.out‚152873.63916015625) proc.memory.percent‚17.12360382080078( disk./.usagePercent‚37.8181464509737 disk.rbd3.out‚ 0.4951171875' disk.nvme0n1.out‚152873.63916015625 disk.sda2.out‚0 disk.sda3.in‚ 108.921875 disk.drbd2165.out‚0 proc.cpu.threads‚17 proc.memory.rssMB‚ 2805.53125 network.recv‚ 6.023664e+06 disk.rbd1.out‚ 63.1943359375 disk.sda.out‚ 1413.01171875 disk.dm-0.in‚ 56047.8515625 disk.sda1.in‚0wâÜ$4å9/ _wandbruntime‚24172  _runtime‚241727ãjDÌæ9Æ ' lr-AdamW/pg1‚0.00012172854251781837 ' lr-AdamW/pg2‚0.00012172854251781837 trainer/global_step‚14599 % _timestamp‚1.7843139882266014e+09 _runtime‚24172.443912658 _step‚1200NßZŽÑç9Ë % _timestamp‚1.7843139882266014e+09 _runtime‚24172.443912658 _step‚1200 ' lr-AdamW/pg1‚0.00012172854251781837 ' lr-AdamW/pg2‚0.00012172854251781837 trainer/global_step‚14599° ÷P¹—4è9/ _wandbruntime‚24172  _runtime‚24172 ™.s(é9“P OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 124564181 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 trainer/global_step‚14599 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13735226304000 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚78815506 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 $ _timestamp‚1.784313988227857e+09 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1150936340359168 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 124564187 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚4210439348273664 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 126449249 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1150935939828736 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 124564187 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚37453052 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚37453047 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 _runtime‚24172.445186152 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13733880914432 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 400530432 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚4210439348273664 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚4208604921914566 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚252 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚247 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚41362459 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚17710627895296 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚4196705467359232 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 124564181 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚78815758 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13733880914432 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 251013436 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚41362706 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚4210440719836672 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 251003457 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚4208603551886276 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13735226304000 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1133225712463872 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚4194893196775424 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚4194893171347456 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚17710595456000 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13710380538820 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚4196705467359232 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 [HDeviceStatsMonitor.on_train_bañÅp”Ì$tch_start/allocated_bytes.small_pool.freed‚13733880914432 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 251013436 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚41362706 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚4210440719836672 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 251003457 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚4208603551886276 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13735226304000 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1133225712463872 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚4194893196775424 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚4194893171347456 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚17710595456000 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13710380538820 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚4196705467359232 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 124564181 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 trainer/global_step‚14599 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13735226304000 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚78815506 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 $ _timestamp‚1.784313988227857e+09 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1150936340359168 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 124564187 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚4210439348273664 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 126449249 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1150935939828736 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 124564187 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚37453052 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚37453047 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 _runtime‚24172.445186152 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13733880914432 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 400530432 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚4210439348273664 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚4208604921914566 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚252 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚247 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚41362459 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚17710627895296 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0± ñVú4ë9/ _wandbruntime‚24173  _runtime‚24173>|_#'ì9N QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 251020141 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚41365168 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚4208882357130698 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚4195171078553600 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13712648605500 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 251030134 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚4210719645609472 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚4196983468089344 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚37455595 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚239 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 251030134 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _step‚1202 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚4196983468089344 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1133302830780416 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 124572393 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13734805949952 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚78821002 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚37455590 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 $ _timestamp‚1.784313989612653e+09 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13711304005066 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 126457741 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13734805949952 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 124572393 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17711801533440 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚4210718274039296 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚244 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 251020141 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚78820758 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚41365407 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 126447754 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚41969834942627846iVê('í9¢N ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚4196983494262784 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 400523264 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚78820758 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚41365407 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 126447754 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚4196983494262784 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 251020141 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚41365168 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚4208882357130698 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚4195171078553600 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13712648605500 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 251030134 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚4210719645609472 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚4196983468089344 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚37455595 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚239 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 251030134 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _step‚1202 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚4196983468089344 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1133302830780416 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 124572393 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13734805949952 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚78821002 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚37455590 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 $ _timestamp‚1.784313989612653e+09 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13711304005066 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 126457741 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13734805949952 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 124572393 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17711801533440 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚4210718274039296 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚244 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 251020141 EDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 126865028 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚4210605503864832 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚252 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1137099664756736 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1154870141047808 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13780196845056 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚32439296 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚17770476291072 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚4222545247936474 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚4224385700709888 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 124974481 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚4224385700709888 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚4210605530038272 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚4208787260256256 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚79079041 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 126865028 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 126875001 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 % _timestamp‚1.7843140670500276e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13781542234624 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚4208787285684224 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚4210605530038272 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13780196845056 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 400530432 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚37580197 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚4224387072272896 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚4222543877908184 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 _runtime‚24251.267356843 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 126875001 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚37580202 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13757962252250 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 251849488 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 124974487 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚247 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1137100032847872 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚79078789 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51rá÷(€:™P QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚79079041 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 126865028 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 126875001 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 % _timestamp‚1.7843140670500276e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13781542234624 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚4208787285684224 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚4210605530038272 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13780196845056 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 400530432 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚37580197 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚4224387072272896 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚4222543877908184 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 _runtime‚24251.267356843 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 126875001 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚37580202 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13757962252250 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 251849488 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 124974487 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚247 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1137100032847872 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚79078789 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1154870541578240 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 126865028 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚4210605503864832 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚252 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1137099664756736 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1154870141047808 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13780196845056 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚32439296 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚17770476291072 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚4222545247936474 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚4224385700709888 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 124974481 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚4224385700709888 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚4210605530038272 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚4208787260256256 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175µ —$ù4:/ _wandbruntime‚24252  _runtime‚24252T®þú¹‚:žN XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚4224665998045696 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 251866186 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚4224665998045696 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 124982687 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 126873506 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚4209065167462400 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚4222822683152606 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚79084358 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚37582740 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13782467277312 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 124982687 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 251866186 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 124982693 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚253 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚4224664626475520 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 251856193 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1137177519255552 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.$Ã#kallocated‚13758885718608 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 126873506 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚248 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1137177151164416 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚4222824053181008 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚79084105 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _runtime‚24252.652844802 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚4210883504594944 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13781121880576 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚17771704372224 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 126883493 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚4210883530768384 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚4209065142034432 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13757541118174 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 % _timestamp‚1.7843140684353356e+09 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13781121880576¥fªx)'ƒ:£N ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚4209065167462400 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚4222822683152606 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚79084358 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚37582740 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13782467277312 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 124982687 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 251866186 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 124982693 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚253 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚4224664626475520 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 251856193 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1137177519255552 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13758885718608 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 126873506 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚248 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1137177151164416 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚4222824053181008 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚79084105 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _runtime‚24252.652844802 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚4210883504594944 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13781121880576 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚17771704372224 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 126883493 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚4210883530768384 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚4209065142034432 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13757541118174 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 % _timestamp‚1.7843140684353356e+09 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13781121880576 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚4224665998045696 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 251866186 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚4224665998045696 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 124982687 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 126873506 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572¶ Ê=­4„:/ _wandbruntime‚24252  _runtime‚24252i´áY…:Ž 3loss/FBt2/assignment_loss‚6.402073408788701e-06 %loss/total_loss‚4.176154286647469 0loss/FRt2/assignment_loss‚0.1628387359428859 4loss/FBt1/assignment_loss‚1.8801492756701712e-05 1loss/SRqqt1/detection_loss‚0.5178017594224245 0loss/FBt1/detection_loss‚0.20451887556886172 epoch‚36 trainer/global_step‚14649 0loss/FRt1/assignment_loss‚0.8729374527352528 3loss/SRqqt1/assignment_loss‚0.06685051247518199 /loss/FRt2/detection_loss‚1.1377432501879428 0loss/FBt2/detection_loss‚0.19131454898116287 _step‚1207 3loss/SRqqt2/assignment_loss‚0.14914040395568218 /loss/FRt1/detection_loss‚0.4249513353830045 1loss/SRqqt2/detection_loss‚0.4480322084289038 % _timestamp‚1.7843140685203006e+09 _runtime‚24252.737242373¥Æy2†:“ 0loss/FRt2/assignment_loss‚0.1628387359428859 4loss/FBt1/assignment_loss‚1.8801492756701712e-05 1loss/SRqqt1/detection_loss‚0.5178017594224245 0loss/FBt1/detection_loss‚0.20451887556886172 epoch‚36 trainer/global_step‚14649 0loss/FRt1/assignment_loss‚0.8729374527352528 3loss/SRqqt1/assignment_loss‚0.06685051247518199 /loss/FRt2/detection_loss‚1.1377432501879428 0loss/FBt2/detection_loss‚0.19131454898116287 _step‚1207 3loss/SRqqt2/assignment_loss‚0.14914040395568218 /loss/FRt1/detection_loss‚0.4249513353830045 1loss/SRqqt2/detection_loss‚0.4480322084289038 % _timestamp‚1.7843140685203006e+09 _runtime‚24252.737242373 3loss/FBt2/assignment_loss‚6.402073408788701e-06 %loss/total_loss‚4.176154286647469· vÅq,‡:‚(:’ ÖñéÒ­´ñö gpu.0.gpu‚90.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚270.97' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚54.19400000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0²’ܧJˆ:‚(:¿ ÖñéÒ…š¦ù disk.sda1.in‚0( disk.nvme0c0n1.out‚153367.3798828125 disk.sda2.in‚0 disk.dm-1.in‚ 108.921875$ disk.nvme0c0n1.in‚56154.13671875 disk.sda3.in‚ 108.921875) proc.memory.percent‚17.12360382080078 disk.rbd3.out‚ 0.4951171875 disk.dm-1.out‚ 1417.70703125 cpu‚25.581754292961666 proc.cpu.threads‚17 disk.drbd2165.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 1417.70703125) proc.memory.availableMB‚ 2829.50390625 disk.dm-0.in‚56154.13671875# disk.dm-0.out‚153367.3798828125 network.recv‚ 6.044884e+06" disk.nvme0n1.in‚56154.13671875& disk.nvme0n1.out‚153367.3798828125 disk.rbd1.out‚ 63.3974609375& disk.rbd0.in‚1.759218604441423e+13$ memory_percent‚82.73007869720459) disk./.usagePercent‚37.84407796613197 network.sent‚ 2.8308735e+07 disk.drbd2165.out‚0 disk.rbd1.in‚ 0.53564453125 disk.sda.in‚ 108.921875 disk.sda1.out‚0 proc.memory.rssMB‚ 2805.53125# disk./.usageGB‚563.904109954834 disk.rbd3.in‚0 disk.sda.out‚ 1417.70703125 disk.rbd0.out‚0+)哉:‚(:ˆ åñéÒÍåùö gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚291.435' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚58.287 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0… ç)DŠ:‚(:¹ åñéÒùèŠú! disk.nvme0n1.in‚ 56154.6953125 disk.sda2.out‚0 disk.sda.out‚ 1418.2734375 disk.drbd2165.in‚0& proc.memory.availableMB‚ 2828.59375 disk.rbd1.in‚ 0.53564453125 disk.sda3.out‚ 1418.2734375# disk.nvme0c0n1.in‚ 56154.6953125) disk.nvme0c0n1.out‚153374.95947265625' disk.nvme0n1.out‚153374.95947265625 disk.sda2.in‚0 disk.drbd2165.out‚0$ disk./.usageGB‚564.1216011047363 disk.sda.in‚ 108.921875 disk.rbd0.out‚0 disk.sda1.out‚0 cpu‚25.581630047581356 proc.cpu.threads‚17 disk.rbd1.out‚ 63.5380859375 disk.dm-1.out‚ 1418.2734375& disk.rbd0.in‚1.759218604441423e+13$ disk.dm-0.out‚153374.95947265625 disk.rbd3.in‚0) proc.memory.percent‚17.12360382080078 network.sent‚ 2.8372055e+07$ memory_percent‚82.73563385009766 disk.dm-1.in‚ 108.921875 disk.dm-0.in‚ 56154.6953125 disk.sda1.in‚0 disk.sda3.in‚ 108.921875 disk.rbd3.out‚ 0.4951171875 proc.memory.rssMB‚ 2805.53125 network.recv‚ 6.04905e+06) disk./.usagePercent‚37.85867397968914)ÓGž‹:‚(:“ ôñéÒˆµýö gpu.0.gpu‚99.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚346.118' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚69.22359999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0(aåAŒ:‚(:¶ ôñéÒ•˜ˆù network.sent‚ 2.8375817e+07) disk./.usagePercent‚37.92727580662444 disk.rbd0.out‚0) proc.memory.availableMB‚ 2841.44921875 disk.rbd1.out‚ 63.5380859375 disk.sda3.in‚ 108.921875) proc.memory.percent‚17.12360382080078 disk.dm-1.out‚ 1418.56640625& disk.rbd0.in‚1.759218604441423e+13 disk.sda2.out‚0 disk.drbd2165.in‚0% disk.nvme0n1.out‚154040.587890625 disk.rbd1.in‚ 0.53564453125$ memory_percent‚82.65717029571533 network.recv‚ 6.052472e+06 disk.rbd3.out‚ 0.4951171875' disk.nvme0c0n1.out‚154040.587890625 disk.sda1.out‚0" disk.nvme0c0n1.in‚ 56266.359375 disk.nvme0n1.in‚ 56266.359375 disk.sda2.in‚0 disk.drbd2165.out‚0 proc.memory.rssMB‚ 2805.53125 proc.cpu.threads‚17 disk.dm-0.in‚ 56266.359375" disk.dm-0.out‚154040.587890625 disk.sda1.in‚0 disk.sda3.out‚ 1418.56640625$ disk./.usageGB‚565.1438179016113 disk.sda.in‚ 108.921875 disk.sda.out‚ 1418.56640625 disk.dm-1.in‚ 108.921875 disk.rbd3.in‚0 cpu‚25.58150990986454´Cƒ,’:‚(:‡ ƒòéÒʪø gpu.0.gpu‚96.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚41.0 gpu.0.powerWatts‚177.38' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.476 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Qµ‡`DŽ:‚(:¹ ƒòéÒøÄÀù proc.cpu.threads‚17 network.recv‚ 6.056065e+06$ disk./.usageGB‚565.1484184265137 disk.dm-1.in‚ 108.921875 disk.sda1.in‚0 disk.drbd2165.out‚0 disk.rbd3.out‚ 0.4951171875! disk.nvme0c0n1.in‚ 56266.59375) disk.nvme0c0n1.out‚154051.81005859375 network.sent‚ 2.8382427e+07 disk.sda.out‚ 1418.85546875 disk.sda3.in‚ 108.921875' disk.nvme0n1.out‚154051.81005859375 disk.sda2.out‚0 disk.rbd1.in‚ 0.53564453125 disk.rbd1.out‚ 63.6005859375( proc.memory.availableMB‚ 2841.4453125 disk.rbd3.in‚0& disk.rbd0.in‚1.759218604441423e+13 disk.dm-0.in‚ 56266.59375$ disk.dm-0.out‚154051.81005859375 disk.sda1.out‚0 disk.nvme0n1.in‚ 56266.59375 disk.sda2.in‚0) disk./.usagePercent‚37.92758455171074$ memory_percent‚82.65719413757324 cpu‚25.581376441553772 disk.rbd0.out‚0 disk.sda3.out‚ 1418.85546875 disk.drbd2165.in‚0 proc.memory.rssMB‚ 2805.53125 disk.sda.in‚ 108.921875 disk.dm-1.out‚ 1418.85546875) proc.memory.percent‚17.12360382080078¤X”:‚(:‰ ’òéÒÜŽ÷ gpu.0.gpu‚91.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚275.246' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.0492 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0=oѵE:‚(:º ’òéұłú disk.sda3.in‚ 108.921875$ disk.dm-0.out‚154635.56884765625 disk.drbd2165.out‚0 disk.sda2.in‚0) proc.memory.availableMB‚ 2841.67578125 disk.dm-0.in‚ 56266.84375 disk.sda.in‚ 108.921875 disk.dm-1.in‚ 108.921875& disk.rbd0.in‚1.759218604441423e+13! disk.nvme0c0n1.in‚ 56266.84375) disk.nvme0c0n1.out‚154635.56884765625 disk.rbd1.in‚ 0.53564453125 proc.memory.rssMB‚ 2805.53125) proc.memory.percent‚17.12360382080078 cpu‚25.581263546096675# disk./.usageGB‚565.097785949707 disk.rbd0.out‚0 disk.sda1.in‚0 disk.rbd3.out‚ 0.4951171875 disk.rbd1.out‚ 63.6005859375 disk.nvme0n1.in‚ 56266.84375' disk.nvme0n1.out‚154635.56884765625* disk./.usagePercent‚37.924186563708766 disk.sda1.out‚0 disk.drbd2165.in‚0$ memory_percent‚82.65578746795654 network.sent‚ 2.8387546e+07 network.recv‚ 6.059794e+06 disk.sda3.out‚ 1419.02734375 disk.rbd3.in‚0 disk.sda2.out‚0 disk.sda.out‚ 1419.02734375 proc.cpu.threads‚17 disk.dm-1.out‚ 1419.02734375ßù× Ÿ‘:‚(:” ¡òéÒ ²‰÷ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚284.867' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚56.973400000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ëòþF’:‚(:» ¡òéÒöôöù disk.dm-1.out‚ 1419.390625& disk.rbd0.in‚1.759218604441423e+13 disk.rbd0.out‚0 disk.sda3.in‚ 108.921875' disk.nvme0n1.out‚154649.10595703125) proc.memory.availableMB‚ 2841.38671875 cpu‚25.58114182003239) disk./.usagePercent‚37.92461921644163 disk.sda2.in‚0 proc.cpu.threads‚17 disk.sda.out‚ 1419.390625 disk.rbd1.in‚ 0.53564453125 disk.sda1.out‚0 disk.sda3.out‚ 1419.390625 disk.drbd2165.in‚0 disk.rbd3.out‚ 0.4951171875) disk.nvme0c0n1.out‚154649.10595703125 disk.dm-0.in‚56267.25390625 disk.dm-1.in‚ 108.921875 network.recv‚ 6.062973e+06$ disk.dm-0.out‚154649.10595703125$ disk.nvme0c0n1.in‚56267.25390625 proc.memory.rssMB‚ 2805.53125 network.sent‚ 2.8391309e+07$ disk./.usageGB‚565.1042327880859 disk.sda.in‚ 108.921875 disk.drbd2165.out‚0" disk.nvme0n1.in‚56267.25390625 disk.rbd1.out‚ 63.6787109375 disk.sda1.in‚0 disk.rbd3.in‚0 disk.sda2.out‚0# memory_percent‚82.6575517654419) proc.memory.percent‚17.12360382080078¿Ù24“:/ _wandbruntime‚24330  _runtime‚24330xQŸôÌ”:Æ ' lr-AdamW/pg1‚0.00011753498857731132 ' lr-AdamW/pg2‚0.00011753498857731132 trainer/global_step‚14699 % _timestamp‚1.7843141458253171e+09 _runtime‚24330.042939819 _step‚1208ŠÑ¡rÑ•:Ë ' lr-AdamW/pg1‚0.00011753498857731132 ' lr-AdamW/pg2‚0.00011753498857731132 trainer/global_step‚14699 % _timestamp‚1.7843141458253171e+09 _runtime‚24330.042939819 _step‚1208¸ âèC4–:/ _wandbruntime‚24330  _runtime‚24330Ž=pù;—:”P RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚4224505566543872 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1140974353231872 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚17830412004352 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13804199365358 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚4238333424709120 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚4224505540370432 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚37707347 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13827858165248 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1140973985140736 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 252685540 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1158804397145088 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚4236485573958382 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 127290780 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1158804799772672 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 % _timestamp‚1.7843141458268712e+09 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13827858165248 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚4238333424709120 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 127300753 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 125384781 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚4238332053146112 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚79339965 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RCDeviceStatsMonitor.on_train_batch_start/reques)6bß ted_bytes.all.current‚ 1370028290 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚79340219 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 252675561 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚254 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚37707352 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚41632867 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚4236484203930092 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 _runtime‚24330.044213564 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 125384787 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚4222681374593024 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 127300753 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 125384781 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚249 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13826512775680 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13826512775680 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820Ëz^g(˜:™P ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 125384781 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚4238332053146112 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚79339965 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚79340219 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 252675561 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚254 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚37707352 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚41632867 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚4236484203930092 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 _runtime‚24330.044213564 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 125384787 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚4222681374593024 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 127300753 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 125384781 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚249 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13826512775680 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13826512775680 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚4224505566543872 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1140974353231872 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚17830412004352 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13804199365358 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚4238333424709120 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚4224505540370432 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚37707347 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13827858165248 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1140973985140736 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 252685540 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1158804397145088 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚4236485573958382 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 127290780 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1158804799772672 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 % _timestamp‚1.7843141458268712e+09 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13827858165248 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚4238333424709120 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 127300753¹ ¡W4™:/ _wandbruntime‚24331  _runtime‚24331þûk<$'š:žN XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚4224783541100544 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13827437811200 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚4224783567273984 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17831607153664 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚4222959256371200 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 252702238 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 125392987 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚41635587 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 252692245 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 125392987 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚245 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 252702238 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚41635342 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13805122831716 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 127309245 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 252692245 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚79345232 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 125392993 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚250 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 % _timestamp‚1.7843141472192276e+09 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13828783207936 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1141051839639552 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 _step‚1210 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13827437811200 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1158883481322496 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚4236763009174514 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚4224783567273984 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1141051471548416 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚4222959230943232 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13828783207936 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚4236764379202916 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚4238612350481920 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 127309245 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚79345482 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚37709895 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640å‹.c‡#›:£N KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 252702238 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚41635342 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13805122831716 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 127309245 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 252692245 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚79345232 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 125392993 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚250 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 % _timestamp‚1.7843141472192276e+09 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13828783207936 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1141051839639552 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 _step‚1210 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13827437811200 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1158883481322496 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚4236763009174514 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚4224783567273984 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1141051471548416 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚4222959230943232 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13828783207936 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚4236764379202916 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚4238612350481920 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 127309245 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚79345482 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚37709895 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚4224783541100544 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13827437811200 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚4224783567273984 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17831607153664 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚4222959256371200 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 252702238 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 125392987 ª×Þ¢RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚41635587 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 252692245 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 125392987 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚245º P9†4œ:/ _wandbruntime‚24331  _runtime‚24331bâ:Ž 3loss/SRqqt2/assignment_loss‚0.15986580813588627 4loss/FBt1/assignment_loss‚1.0731139865545931e-05 /loss/FBt1/detection_loss‚0.6348755338699751 &loss/total_loss‚4.8916629778305705 % _timestamp‚1.7843141473018773e+09 _step‚1211 3loss/SRqqt1/assignment_loss‚0.16100313476698586 /loss/FRt2/detection_loss‚1.2618753904345936 /loss/FBt2/detection_loss‚0.1375010620647507 _runtime‚24331.519000718 0loss/FRt1/assignment_loss‚0.9316403458591636 4loss/FBt2/assignment_loss‚2.8950420301043777e-07 1loss/SRqqt2/detection_loss‚0.5205655409848426 trainer/global_step‚14699 0loss/FRt2/assignment_loss‚0.1029224170773637 /loss/FRt1/detection_loss‚0.4449826884135715 1loss/SRqqt1/detection_loss‚0.5364200355793682 epoch‚36ÿš½ž:“ &loss/total_loss‚4.8916629778305705 % _timestamp‚1.7843141473018773e+09 _step‚1211 3loss/SRqqt1/assignment_loss‚0.16100313476698586 /loss/FRt2/detection_loss‚1.2618753904345936 /loss/FBt2/detection_loss‚0.1375010620647507 _runtime‚24331.519000718 0loss/FRt1/assignment_loss‚0.9316403458591636 4loss/FBt2/assignment_loss‚2.8950420301043777e-07 1loss/SRqqt2/detection_loss‚0.5205655409848426 trainer/global_step‚14699 0loss/FRt2/assignment_loss‚0.1029224170773637 /loss/FRt1/detection_loss‚0.4449826884135715 1loss/SRqqt1/detection_loss‚0.5364200355793682 epoch‚36 3loss/SRqqt2/assignment_loss‚0.15986580813588627 4loss/FBt1/assignment_loss‚1.0731139865545931e-05 /loss/FBt1/detection_loss‚0.6348755338699751» ¹<2Ó”Ÿ:‚(:‰ °òéÒ„‚œ÷ gpu.0.gpu‚99.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚334.153' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚66.8306 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0I3¹¬9 :‚(:® °òéÒë»”ú disk.rbd1.in‚ 0.53564453125 cpu‚25.58102983369902 disk.dm-1.in‚ 108.921875 disk.dm-1.out‚ 1419.59765625& disk.rbd0.in‚1.759218604441423e+13 disk.sda1.in‚0 disk.sda3.out‚ 1419.59765625 disk.rbd0.out‚0 disk.drbd2165.in‚0 disk.sda2.in‚0 proc.cpu.threads‚17 network.sent‚ 2.8396501e+07 disk.drbd2165.out‚0! disk.nvme0c0n1.in‚ 56297.15625& disk.nvme0c0n1.out‚154664.61328125 proc.memory.rssMB‚ 2805.53125$ disk./.usageGB‚565.1091537475586 disk.sda.in‚ 108.921875 disk.dm-0.in‚ 56297.15625 disk.sda1.out‚0 disk.rbd3.out‚ 0.4951171875 disk.rbd1.out‚ 63.6787109375) proc.memory.percent‚17.12360382080078 network.recv‚ 6.06647e+06 disk.rbd3.in‚0$ memory_percent‚82.65886306762695 disk.sda.out‚ 1419.59765625! disk.dm-0.out‚154664.61328125 disk.sda2.out‚0' proc.memory.availableMB‚ 2841.171875* disk./.usagePercent‚37.924949466160804 disk.sda3.in‚ 108.921875 disk.nvme0n1.in‚ 56297.15625$ disk.nvme0n1.out‚154664.61328125Þ-–“¡:‚(:ˆ ¿òéÒ˃ñö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚261.535' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.307 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0MÙ¯?G¢:‚(:¼ ¿òéÒ²²áù disk.sda.out‚ 1419.9296875 disk.dm-1.in‚ 108.921875 disk.dm-1.out‚ 1419.9296875 disk.sda2.out‚0 proc.cpu.threads‚17 disk.sda3.in‚ 108.921875 disk.rbd3.in‚0 disk.rbd1.in‚ 0.53564453125 proc.memory.rssMB‚ 2805.53125) proc.memory.percent‚17.12360382080078 disk.rbd0.out‚0& proc.memory.availableMB‚ 2841.15625 network.sent‚ 2.8459854e+07 disk.sda.in‚ 108.921875 disk.drbd2165.out‚0) disk.nvme0c0n1.out‚154673.27783203125" disk.nvme0n1.in‚56344.60546875 network.recv‚ 6.070696e+06$ disk.nvme0c0n1.in‚56344.60546875 disk.sda2.in‚0 cpu‚25.580898810038544& disk.rbd0.in‚1.759218604441423e+13 disk.drbd2165.in‚0 disk.rbd3.out‚ 0.4951171875 disk.rbd1.out‚ 63.7412109375# memory_percent‚82.6589584350586) disk./.usagePercent‚37.92542026401627$ disk./.usageGB‚565.1161689758301 disk.dm-0.in‚56344.60546875 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda3.out‚ 1419.9296875' disk.nvme0n1.out‚154673.27783203125$ disk.dm-0.out‚154673.27783203125‰Bá•£:‚(:Š ÎòéÒº¿çö gpu.0.gpu‚100.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚344.528' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚68.9056 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ä¾>¤:‚(:³ ÎòéÒ‚ìÏù disk.sda.out‚ 1420.078125 disk.dm-1.in‚ 108.921875 disk.dm-1.out‚ 1420.078125( disk.nvme0c0n1.out‚154688.0732421875 disk.nvme0n1.in‚ 56345.015625 disk.dm-0.in‚ 56345.015625# disk.dm-0.out‚154688.0732421875 disk.sda3.out‚ 1420.078125 disk.rbd3.out‚ 0.4951171875& disk.nvme0n1.out‚154688.0732421875 disk.sda2.in‚0 disk.rbd1.in‚ 0.53564453125$ memory_percent‚82.65941143035889 disk.sda1.out‚0$ disk./.usageGB‚565.1227645874023 disk.drbd2165.in‚0) proc.memory.availableMB‚ 2841.08203125) proc.memory.percent‚17.12360382080078 proc.cpu.threads‚17" disk.nvme0c0n1.in‚ 56345.015625 proc.memory.rssMB‚ 2805.53125 disk.sda2.out‚0 disk.sda1.in‚0* disk./.usagePercent‚37.925862901042976 disk.sda.in‚ 108.921875 disk.drbd2165.out‚0 disk.rbd1.out‚ 63.7412109375 network.sent‚ 2.8464963e+07 network.recv‚ 6.07415e+06 disk.sda3.in‚ 108.921875& disk.rbd0.in‚1.759218604441423e+13 disk.rbd0.out‚0 disk.rbd3.in‚0 cpu‚25.58078788597883'3–“¥:‚(:ˆ ÝòéÒ÷ÒÓö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚261.535' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.307 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ‰‰^@¦:‚(:µ ÝòéÒµäãø& disk.rbd0.in‚1.759218604441423e+13 disk.drbd2165.in‚0# disk.nvme0c0n1.in‚ 56345.2421875 disk.dm-1.out‚ 1420.37890625( proc.memory.availableMB‚ 2841.0703125) disk./.usagePercent‚37.92616729400118 disk.rbd0.out‚0& disk.nvme0c0n1.out‚154697.81640625 disk.sda2.in‚0 disk.rbd1.out‚ 63.8193359375 disk.dm-0.in‚ 56345.2421875! disk.dm-0.out‚154697.81640625 disk.sda1.in‚0 disk.sda2.out‚0 proc.memory.rssMB‚ 2805.53125 network.sent‚ 2.8470045e+07 network.recv‚ 6.077488e+06! disk.nvme0n1.in‚ 56345.2421875 disk.sda.in‚ 108.921875) proc.memory.percent‚17.12360382080078 disk.drbd2165.out‚0$ disk.nvme0n1.out‚154697.81640625 disk.sda.out‚ 1420.37890625 disk.sda3.out‚ 1420.37890625$ disk./.usageGB‚565.1273002624512 disk.sda3.in‚ 108.921875 disk.rbd3.in‚0 disk.rbd1.in‚ 0.53564453125 disk.dm-1.in‚ 108.921875 cpu‚25.58065818922572 disk.sda1.out‚0 disk.rbd3.out‚ 0.4951171875$ memory_percent‚82.65948295593262 proc.cpu.threads‚17ð–Óµž§:‚(:“ ìòéÒ‰ƒ÷ gpu.0.gpu‚47.0 gpu.0.memory‚14, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚331.737' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚66.34740000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0/à3üE¨:‚(:º ìòéÒ”÷Öù disk.sda2.in‚0 disk.sda.in‚ 108.921875# disk.dm-0.out‚154708.3193359375 proc.memory.rssMB‚ 2805.53125 disk.rbd1.out‚ 63.8193359375 disk.dm-1.out‚ 1420.55859375 disk.drbd2165.in‚0 disk.drbd2165.out‚0 cpu‚25.58055550779754 proc.cpu.threads‚17# memory_percent‚82.6481580734253& disk.rbd0.in‚1.759218604441423e+13 disk.dm-0.in‚ 56345.6640625 network.sent‚ 2.8476637e+07( disk./.usagePercent‚37.9267932324224! disk.nvme0n1.in‚ 56345.6640625 disk.sda2.out‚0 disk.rbd1.in‚ 0.53564453125 disk.sda.out‚ 1420.55859375 disk.sda1.in‚0 disk.sda1.out‚0# disk.nvme0c0n1.in‚ 56345.6640625( disk.nvme0c0n1.out‚154708.3193359375 network.recv‚ 6.081042e+06) proc.memory.availableMB‚ 2842.92578125$ disk./.usageGB‚565.1366271972656& disk.nvme0n1.out‚154708.3193359375 disk.dm-1.in‚ 108.921875 disk.rbd0.out‚0 disk.sda3.in‚ 108.921875 disk.sda3.out‚ 1420.55859375 disk.rbd3.in‚0 disk.rbd3.out‚ 0.4951171875) proc.memory.percent‚17.12360382080078ìû7#4©:/ _wandbruntime‚24409  _runtime‚24409{¸IÒ̪:Æ ' lr-AdamW/pg2‚0.00011546014802531412 trainer/global_step‚14749 % _timestamp‚1.7843142251016743e+09 _runtime‚24409.318941834 _step‚1212 ' lr-AdamW/pg1‚0.00011546014802531412=ÓÃÑ«:Ë ' lr-AdamW/pg1‚0.00011546014802531412 ' lr-AdamW/pg2‚0.00011546014802531412 trainer/global_step‚14749 % _timestamp‚1.7843142251016743e+09 _runtime‚24409.318941834 _step‚1212¼ ±Êw4¬:/ _wandbruntime‚24409  _runtime‚24409Qª!¨(­:”P M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 125795081 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚4236575438073856 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 125795087 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 127716532 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 % _timestamp‚1.7843142251028519e+09 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1144848673615872 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚4236575463501824 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 253521592 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚41767487 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚4252278405582336 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚4238405603049472 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 trainer/global_step‚14749 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚79601984 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1144848673615872 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚4236575463501824 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 253521592 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚41767487 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚4252278405582336 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚4238405603049472 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 trainer/global_step‚14749 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚79601984 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 127716532 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 % _timestamp‚1.7843142251028519e+09 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 BÁ°:žN [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚4238683577606144 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13850015344390 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 253538290 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13873753741824 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚4236853319852032 % _timestamp‚1.7843142264926257e+09 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 125803287 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 N>DeviceStatsMonit5²Ù3c!or.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13875099138560 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 127734997 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13851359944824 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 127725010 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 127725010 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚4250703335196422 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚4238683577606144 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚41770489 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚4250704705224824 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 _runtime‚24410.710059384 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 253528297 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 125803293 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚4252558702918144 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 125803287 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 trainer/global_step‚14749 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚4238683603779584 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17891574772736 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚4252558702918144 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 253538290 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1162817769325568 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚79607280 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚37837040 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚249 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1144926160023552 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 125803293 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚254 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 _step‚1214 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚4252557331347968 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚79607534 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚4236853345280000 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 253528297 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352ž‹ )'±:£N QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 125803293 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚254 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 _step‚1214 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚4252557331347968 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚79607534 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚4236853345280000 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 253528297 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚4238683577606144 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13850015344390 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 253538290 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13873753741824 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚4236853319852032 % _timestamp‚1.7843142264926257e+09 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 125803287 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13875099138560 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 127734997 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13851359944824 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 127725010 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 127725010 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚4250703335196422 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚4238683577606144 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚41770489 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚4250704705224824 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 _runtime‚24410.710059384 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 253528297 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 125803293 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚4252558702918144 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 125803287 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 trainer/global_step‚14749 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚4238683603779584 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17891574772736 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚4252558702918144 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 253538290 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1162817769325568 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚79607280 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚37837040 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚249 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1144926160023552 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0¾ ó t™4²:/ _wandbruntime‚24410  _runtime‚24410 5wͳ:Ž 0loss/FRt1/assignment_loss‚0.9257059564539628 .loss/FRt2/detection_loss‚1.262372463153859 1loss/SRqqt1/detection_loss‚0.4632139499655795 1loss/SRqqt2/detection_loss‚0.4379298742116249 /loss/FBt1/detection_loss‚0.2620651318918265 % _timestamp‚1.7843142265763576e+09 3loss/FBt1/assignment_loss‚4.533210279638362e-06 0loss/FBt2/detection_loss‚0.13732727297524974 epoch‚36 trainer/global_step‚14749 _runtime‚24410.793281837 _step‚1215 1loss/FRt2/assignment_loss‚0.10888919322164704 3loss/SRqqt1/assignment_loss‚0.09438597611798517 3loss/SRqqt2/assignment_loss‚0.14493295660674382 0loss/FRt1/detection_loss‚0.44371294846420434 %loss/total_loss‚4.280543993017915 4loss/FBt2/assignment_loss‚3.7367449516940646e-06‰©ô´:“ 3loss/SRqqt2/assignment_loss‚0.14493295660674382 0loss/FRt1/detection_loss‚0.44371294846420434 %loss/total_loss‚4.280543993017915 4loss/FBt2/assignment_loss‚3.7367449516940646e-06 0loss/FRt1/assignment_loss‚0.9257059564539628 .loss/FRt2/detection_loss‚1.262372463153859 1loss/SRqqt1/detection_loss‚0.4632139499655795 1loss/SRqqt2/detection_loss‚0.4379298742116249 /loss/FBt1/detection_loss‚0.2620651318918265 % _timestamp‚1.7843142265763576e+09 3loss/FBt1/assignment_loss‚4.533210279638362e-06 0loss/FBt2/detection_loss‚0.13732727297524974 epoch‚36 trainer/global_step‚14749 _runtime‚24410.793281837 _step‚1215 1loss/FRt2/assignment_loss‚0.10888919322164704 3loss/SRqqt1/assignment_loss‚0.09438597611798517¿ PÍ0Ÿµ:‚(:” ûòéÒ´úÞø gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚283.559' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚56.711800000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0,öC¶:‚(:¸ ûòéÒ”̽û( proc.memory.availableMB‚ 2842.9140625" disk.nvme0c0n1.in‚ 56439.546875 disk.nvme0n1.in‚ 56439.546875 disk.sda1.in‚0 disk.sda3.in‚ 108.921875$ memory_percent‚82.64822959899902 network.recv‚ 6.084187e+06 disk.rbd3.out‚ 0.4951171875# disk.dm-0.out‚154722.0849609375 disk.sda1.out‚0 network.sent‚ 2.8480366e+07 disk.rbd3.in‚0& disk.nvme0n1.out‚154722.0849609375 disk.sda.in‚ 108.921875 disk.rbd0.out‚0 disk.drbd2165.out‚0) disk./.usagePercent‚37.92704130372308( disk.nvme0c0n1.out‚154722.0849609375 disk.sda.out‚ 1420.95703125 disk.dm-1.out‚ 1420.95703125 disk.sda3.out‚ 1420.95703125 cpu‚25.580410142578067 proc.cpu.threads‚17 disk.sda2.in‚0 disk.sda2.out‚0) proc.memory.percent‚17.12360382080078 disk.rbd1.out‚ 63.8818359375& disk.rbd0.in‚1.759218604441423e+13 disk.dm-0.in‚ 56439.546875 proc.memory.rssMB‚ 2805.53125# disk./.usageGB‚565.140323638916 disk.rbd1.in‚ 0.53564453125 disk.dm-1.in‚ 108.921875 disk.drbd2165.in‚0(ƒÌŸ·:‚(:” ŠóéÒ‰³çö gpu.0.gpu‚100.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚331.737' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚66.34740000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0npÿè6¸:‚(:« ŠóéÒÚ”Ãù disk.sda3.out‚ 1421.34765625$ disk.nvme0c0n1.in‚56439.78515625" disk.nvme0n1.in‚56439.78515625 disk.sda2.in‚0 disk.sda2.out‚0( disk./.usagePercent‚37.9273666892991 disk.dm-1.out‚ 1421.34765625& disk.rbd0.in‚1.759218604441423e+13 disk.rbd0.out‚0 disk.rbd1.out‚ 63.8818359375 cpu‚25.58030371975168 proc.memory.rssMB‚ 2805.53125$ disk./.usageGB‚565.1451721191406 disk.dm-1.in‚ 108.921875 disk.drbd2165.in‚0 disk.drbd2165.out‚0 proc.cpu.threads‚17 disk.dm-0.in‚56439.78515625 network.sent‚ 2.8543729e+07 disk.sda1.in‚0" disk.nvme0c0n1.out‚ 154732.6875 disk.rbd1.in‚ 0.53564453125 disk.sda.in‚ 108.921875) proc.memory.availableMB‚ 2842.38671875) proc.memory.percent‚17.12360382080078 disk.dm-0.out‚ 154732.6875 disk.rbd3.in‚0 disk.rbd3.out‚ 0.4951171875 disk.nvme0n1.out‚ 154732.6875 disk.sda.out‚ 1421.34765625 network.recv‚ 6.088718e+06 disk.sda1.out‚0 disk.sda3.in‚ 108.921875# memory_percent‚82.6514482498169l&ž”¹:‚(:‰ ™óéÒœ·ûö gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚286.356' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚57.2712 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ЪDëNº:‚(:à ™óéÒÀ®ïù disk.rbd3.in‚0' disk.nvme0n1.out‚154741.94384765625 cpu‚25.580172022994997 proc.cpu.threads‚17 disk.sda.out‚ 1421.53515625 disk.dm-1.in‚ 108.921875) disk.nvme0c0n1.out‚154741.94384765625 disk.sda2.out‚0 disk.rbd0.out‚0 disk.sda1.in‚0 disk.sda2.in‚0 disk.rbd1.out‚ 64.0224609375 network.sent‚ 2.8550161e+07 disk.dm-1.out‚ 1421.53515625 disk.rbd1.in‚ 0.53564453125$ memory_percent‚82.64997005462646) proc.memory.availableMB‚ 2842.62890625 proc.memory.rssMB‚ 2805.53125) proc.memory.percent‚17.12360382080078 disk.sda.in‚ 108.921875 disk.dm-0.in‚56440.20703125 disk.sda3.in‚ 108.921875 disk.sda3.out‚ 1421.53515625" disk.nvme0n1.in‚56440.20703125$ disk.dm-0.out‚154741.94384765625 disk.drbd2165.out‚0$ disk.nvme0c0n1.in‚56440.20703125) disk./.usagePercent‚37.92800389205182$ disk./.usageGB‚565.1546669006348 disk.rbd3.out‚ 0.4951171875 disk.drbd2165.in‚0 disk.sda1.out‚0 network.recv‚ 6.091933e+06& disk.rbd0.in‚1.759218604441423e+13T Ô Ÿ»:‚(:” ¨óéÒ€¼•÷ gpu.0.gpu‚91.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚258.875' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚51.775000000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Tæ¢qI¼:‚(:¾ ¨óéÒˆù‘ú disk.drbd2165.in‚0 disk.sda2.in‚0' disk.nvme0n1.out‚154759.35595703125 network.sent‚ 2.8553878e+07) disk./.usagePercent‚37.92830649295729 disk.sda.out‚ 1421.74609375 disk.dm-1.out‚ 1421.74609375 disk.sda3.in‚ 108.921875 disk.rbd3.in‚0$ memory_percent‚82.65151977539062 disk.drbd2165.out‚0 disk.rbd1.out‚ 64.0224609375 disk.sda1.in‚0 network.recv‚ 6.095282e+06& disk.rbd0.in‚1.759218604441423e+13) disk.nvme0c0n1.out‚154759.35595703125" disk.nvme0n1.in‚56440.41015625 disk.sda2.out‚0 proc.memory.rssMB‚ 2805.53125) proc.memory.percent‚17.12360382080078$ disk./.usageGB‚565.1591758728027 disk.dm-1.in‚ 108.921875 disk.sda1.out‚0 disk.sda3.out‚ 1421.74609375$ disk.nvme0c0n1.in‚56440.41015625$ disk.dm-0.out‚154759.35595703125 disk.sda.in‚ 108.921875$ proc.memory.availableMB‚2842.375 disk.dm-0.in‚56440.41015625 disk.rbd3.out‚ 0.4951171875 proc.cpu.threads‚17 disk.rbd0.out‚0 disk.rbd1.in‚ 0.53564453125 cpu‚25.580071279705066ž“ç;”½:‚(:‰ ·óéÒÕ¢ðö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚281.966' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚56.3932 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0PXPJH¾:‚(:½ ·óéÒ—ÛÙù network.sent‚ 2.8559018e+07 disk.sda2.in‚0" disk.dm-0.out‚154767.728515625$ memory_percent‚82.64858722686768) proc.memory.availableMB‚ 2842.85546875 disk.sda.in‚ 108.921875 disk.rbd3.in‚0" disk.nvme0n1.in‚56539.16796875) disk./.usagePercent‚37.92861882214907 disk.rbd1.in‚ 0.53564453125 disk.dm-1.in‚ 108.921875 disk.drbd2165.in‚0 disk.drbd2165.out‚0 proc.cpu.threads‚17 disk.sda2.out‚0 disk.rbd1.out‚ 64.0849609375 disk.dm-1.out‚ 1422.05859375 disk.sda1.in‚0% disk.nvme0n1.out‚154767.728515625 network.recv‚ 6.098455e+06$ disk./.usageGB‚565.1638298034668& disk.rbd0.in‚1.759218604441423e+13 disk.sda1.out‚0 disk.sda3.out‚ 1422.05859375 disk.rbd3.out‚ 0.4951171875$ disk.nvme0c0n1.in‚56539.16796875 cpu‚25.579931394765715 disk.sda.out‚ 1422.05859375 disk.dm-0.in‚56539.16796875 disk.sda3.in‚ 108.921875 proc.memory.rssMB‚ 2805.53125) proc.memory.percent‚17.12360382080078 disk.rbd0.out‚0' disk.nvme0c0n1.out‚154767.728515625T;¢4¿:/ _wandbruntime‚24487  _runtime‚24487@¶†“ÌÀ:Æ _runtime‚24487.991933165 _step‚1216 ' lr-AdamW/pg1‚0.00011340010975038442 ' lr-AdamW/pg2‚0.00011340010975038442 trainer/global_step‚14799 % _timestamp‚1.7843143037746902e+09ó ~ÖÑÁ:Ë trainer/global_step‚14799 % _timestamp‚1.7843143037746902e+09 _runtime‚24487.991933165 _step‚1216 ' lr-AdamW/pg1‚0.00011340010975038442 ' lr-AdamW/pg2‚0.00011340010975038442À `ã’x4Â:/ _wandbruntime‚24487  _runtime‚24487È:g%Ã:”P UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 128152257 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 _runtime‚24487.993173386 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13919144636928 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 128142284 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 128152257 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1166673223364608 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚4266224758018560 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13895328991252 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚4252305639555072 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13896673591574 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 _step‚1217 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13920490026496 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚32439296 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 254357644 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚4266226129581568 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 128142284 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚17950597455872 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 400530432 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚37961652 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 WJDeviceStatsMonáì·õitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚79864051 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚4264366226002198 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 % _timestamp‚1.7843143037758276e+09 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 126205381 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13919144636928 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚41902399 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 126205381 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚4252305639555072 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1148722625908736 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13920490026496 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚251 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 254347665 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚4266226129581568 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚37961647 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1148722993999872 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚4264364855973908 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚4250469552410624 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 400530432 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚37961652 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚79864051 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚4264366226002198 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 % _timestamp‚1.7843143037758276e+09 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 126205381 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13919144636928 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚41902399 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 126205381 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚4252305639555072 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1148722625908736 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13920490026496 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚251 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 254347665 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚4266226129581568 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚37961647 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1148722993999872 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚4264364855973908 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚4250469552410624 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚4266224758018560 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13895328991252 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚4252305639555072 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13896673591574 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 _step‚1217 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13920490026496 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚32439296 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 254357644 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚4266226129581568 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 128142284 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568Á íÎW4Å:/ _wandbruntime‚24489  _runtime‚24489£ê ˆ$'Æ:žN REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13921415069184 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 126213587 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚248 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚37964190 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚4252583640285184 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13896252457498 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 254374342 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 _runtime‚24489.380421707 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚4252583614111744 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚41904968 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13897597057932 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚4266505055354368 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚4252583640285184 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1148800480407552 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚4266505055354368 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚79868915 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚17951832082432 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 128160749 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 254364349 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 126213593 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13920069672448 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 trainer/global_step‚14799 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 254374342 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚37964195 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1148800112316416 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 128150762 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚4250747408760832 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚4250747434188800 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 128160749 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 254364349 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚79869163 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚243 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 128150762 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚4264645031246732 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119¸)jíqÇ:£N E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 _runtime‚24489.380421707 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚4252583614111744 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚41904968 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13897597057932 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚4266505055354368 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚4252583640285184 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1148800480407552 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚4266505055354368 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚79868915 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚17951832082432 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 128160749 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 254364349 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 126213593 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13920069672448 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 trainer/global_step‚14799 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 254374342 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚37964195 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1148800112316416 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 128150762 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚4250747408760832 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚4250747434188800 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 128160749 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 254364349 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚79869163 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚243 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 128150762 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚4264645031246732 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13921415069184 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 126213587 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚248 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚37964190 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚4252583640285184 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13896252457498 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 254374342 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 m «ä4È:/ _wandbruntime‚24489  _runtime‚24489ãÐœXÉ:’ /loss/FRt2/detection_loss‚1.2081513610925687 2loss/SRqqt1/detection_loss‚0.47040744472481005 _runtime‚24489.465311293 0loss/FRt1/assignment_loss‚0.9736304017685223 3loss/SRqqt1/assignment_loss‚0.09912483538298009 0loss/FBt2/detection_loss‚0.10582502264059446 trainer/global_step‚14799 % _timestamp‚1.7843143052483637e+09 1loss/FRt2/assignment_loss‚0.11889179512328908 4loss/FBt1/assignment_loss‚2.6822085708033683e-08 4loss/FBt2/assignment_loss‚0.00035306467630295715 1loss/SRqqt2/detection_loss‚0.4744769033042867 3loss/SRqqt2/assignment_loss‚0.15483290038234906 0loss/FRt1/detection_loss‚0.42237257594185174 0loss/FBt1/detection_loss‚0.36628171639704304 %loss/total_loss‚4.394348048256684 epoch‚36 _step‚1219úbŸÊ:— 0loss/FBt2/detection_loss‚0.10582502264059446 trainer/global_step‚14799 % _timestamp‚1.7843143052483637e+09 1loss/FRt2/assignment_loss‚0.11889179512328908 4loss/FBt1/assignment_loss‚2.6822085708033683e-08 4loss/FBt2/assignment_loss‚0.00035306467630295715 1loss/SRqqt2/detection_loss‚0.4744769033042867 3loss/SRqqt2/assignment_loss‚0.15483290038234906 0loss/FRt1/detection_loss‚0.42237257594185174 0loss/FBt1/detection_loss‚0.36628171639704304 %loss/total_loss‚4.394348048256684 epoch‚36 _step‚1219 /loss/FRt2/detection_loss‚1.2081513610925687 2loss/SRqqt1/detection_loss‚0.47040744472481005 _runtime‚24489.465311293 0loss/FRt1/assignment_loss‚0.9736304017685223 3loss/SRqqt1/assignment_loss‚0.09912483538298009à m3„k’Ë:‚(:‡ ÆóéÒ³”„÷ gpu.0.gpu‚3.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚41.0 gpu.0.powerWatts‚334.052' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚66.8104 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÚMGÌ:‚(:¼ ÆóéÒÏšÈù disk.dm-1.in‚ 108.921875 cpu‚25.580392636099198( disk./.usagePercent‚37.9278751202621 disk.sda1.out‚0 disk.rbd3.in‚0$ disk.nvme0c0n1.in‚56539.58984375 disk.sda2.out‚0 disk.dm-1.out‚ 1422.234375 disk.dm-0.in‚56539.58984375 disk.sda3.out‚ 1422.234375 proc.memory.rssMB‚ 2805.53125 network.recv‚ 6.101835e+06 disk.sda1.in‚0 disk.rbd3.out‚ 0.4951171875 disk.sda.in‚ 108.921875 network.sent‚ 2.8564097e+07 disk.sda3.in‚ 108.921875 disk.drbd2165.out‚0 disk.rbd1.in‚ 0.53564453125 proc.cpu.threads‚17$ disk.dm-0.out‚154776.70263671875" disk.nvme0n1.in‚56539.58984375 disk.rbd1.out‚ 64.0849609375$ memory_percent‚82.65130519866943) proc.memory.availableMB‚ 2842.41015625$ disk./.usageGB‚565.1527481079102 disk.drbd2165.in‚0) disk.nvme0c0n1.out‚154776.70263671875& disk.rbd0.in‚1.759218604441423e+13' disk.nvme0n1.out‚154776.70263671875 disk.sda2.in‚0 disk.sda.out‚ 1422.234375) proc.memory.percent‚17.12360382080078 disk.rbd0.out‚0ƒd‡-’Í:‚(:‡ ÕóéÒ™Öœø gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚40.0 gpu.0.powerWatts‚327.433' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚65.4866 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ÿžë\<Î:‚(:± ÕóéÒ¸œïù! disk.nvme0n1.in‚ 56539.8046875 cpu‚25.58200384319586 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-1.out‚ 1422.5625) proc.memory.percent‚17.12360382080078 proc.cpu.threads‚17 disk.rbd1.in‚ 0.53564453125$ disk.dm-0.out‚154789.24365234375 disk.sda3.in‚ 108.921875 disk.rbd3.out‚ 0.4951171875 disk.sda2.in‚0 disk.sda.out‚ 1422.5625 disk.sda3.out‚ 1422.5625 disk.drbd2165.out‚0( proc.memory.availableMB‚ 2842.6015625 network.recv‚ 6.106088e+06 disk.dm-1.in‚ 108.921875 disk.rbd0.out‚0 disk.dm-0.in‚ 56539.8046875) disk.nvme0c0n1.out‚154789.24365234375 network.sent‚ 2.8627513e+07( disk./.usagePercent‚37.9281569845573 disk.rbd1.out‚ 64.1630859375 disk.sda.in‚ 108.921875' disk.nvme0n1.out‚154789.24365234375$ memory_percent‚82.65013694763184 proc.memory.rssMB‚ 2805.53125$ disk./.usageGB‚565.1569480895996 disk.drbd2165.in‚0 disk.sda2.out‚0& disk.rbd0.in‚1.759218604441423e+13 disk.rbd3.in‚0# disk.nvme0c0n1.in‚ 56539.8046875‚›2Æ4Ï:/ _wandbruntime‚24516  _runtime‚24516Ùí›k >Ð:†| 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.15465430915355682 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.8148148059844971 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9675024151802063 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9888569116592407 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.4738966226577759 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.1082889661192894 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4583333432674408 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9898683428764343 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.12674087285995483 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.14135292172431946 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9917780756950378 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.2070690542459488 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6791369318962097 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.032771751284599304 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.09885440766811371 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8201090693473816 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6788597702980042 *loss/val_total_loss‚10.992282137870461 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 8"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.733178436756134 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.8846153616905212 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7857142686843872 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 6 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.976502001285553 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7268518209457397 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.05000000074505806 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.02471582405269146 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.6835978627204895 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5913946628570557 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.8999999761581421 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3434205651283264 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.4976852238178253 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.5938971042633057 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7317311763763428 particle/accuracy_2_of_3‚1 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 8"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.608867883682251 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9966966509819031 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.3898397982120514 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.05000000074505806 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.16412043571472168 particle/accuracy_1_of_1‚1 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.5792959928512573 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7403045296669006 +jet/accuracy_3_of_4‚0.05347250774502754 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.01994013413786888 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5913946628570557 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.2222222238779068 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.14178135991096497 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 jet/accuracy_2_of_5‚1 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.3051789104938507 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 $ _timestamp‚1.784314331813367e+09 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.14827671647071838 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.8999999761581421 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 jet/accuracy_3_of_5‚1 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4921875 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.10499999672174454 *jet/accuracy_2_of_4‚0.7330344915390015 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9909146428108215 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9924677610397339 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0030988692305982113 /particle/accuracy_3_of_4‚0.9974937438964844 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.23076923191547394 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9837511777877808 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 *jet/accuracy_1_of_1‚0.2802661955356598 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 6 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.682445228099823 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.2705172598361969 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.1537608951330185 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.625 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 particle/accuracy_2_of_4‚1 particle/accuracy_5_of_5‚1 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 +"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.25 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6434425711631775 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7620711326599121 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3462529182434082 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 ,jet/accuracy_4_of_4‚0.013899499550461769 particle/accuracy_1_of_4‚1 *particle/sensitivity‚0.594962477684021 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8280256986618042 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.02455909550189972 6validation_average_jet_accuracy‚0.3245682716369629 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.5963541269302368 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5934149622917175 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 particle/accuracy_4_of_5‚1 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9966966509819031 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.33166977763175964 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.6686697006225586 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.6535340547561646 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.023904161527752876 (particle/accuracy‚0.8355887532234192 _runtime‚24516.031411215 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6254529356956482 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.005507623311132193 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 particle/accuracy_2_of_5‚1 particle/accuracy_3_of_5‚1 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.22785785794258118 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.14135292172431946 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ) Purity/2FRt*SRqqt2FBt/FRt_purity‚0.75 8"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.619834303855896 jet/accuracy_5_of_5‚0 particle/accuracy_1_of_5‚1 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.6941421627998352 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6290315985679626 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.6073884963989258 *jet/accuracy_1_of_4‚0.9625317454338074 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6175370812416077 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.2222222238779068 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7571137547492981 particle/accuracy_1_of_2‚1 7 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.0242546945810318 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8888888955116272 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6451865434646606 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7268518209457397 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5792078971862793 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9888888597488403 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.02584935910999775 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9101652503013611 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.4600139856338501 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9935952425003052 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 _step‚1220 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7151234745979309 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9869948625564575 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.23076923191547394 trainer/global_step‚14799 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.12625433504581451 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4406282305717468 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.15177632868289948 6 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.997686505317688 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5284673571586609 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6555982232093811 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_3‚1 /particle/accuracy_4_of_4‚0.9289587140083313 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.1408676952123642 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.1744830459356308 - Purity/0FRt2SRqqt*FBt/FBt_purity‚0.828125 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 jet/accuracy_1_of_5‚1 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6582552194595337 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6084899306297302 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6526565551757812 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 particle/accuracy_3_of_3‚1 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.10974667221307755 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.6686697006225586 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.13349221646785736 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.5908263325691223 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9961451292037964 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 *jet/accuracy_2_of_3‚0.5050614476203918 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6435207724571228 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.12745578587055206 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6790653467178345 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9937976002693176 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.19211623072624207 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8172259330749512 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9939800500869751 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.8903210759162903 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.02470415271818638 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7284084558486938 *jet/accuracy_1_of_2‚0.7189023494720459 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7142857313156128 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8313804864883423 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9862018823623657 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8201090693473816 particle/accuracy_2_of_2‚1 +"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.25 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 - Purity/0FRt2SRqqt1FBt/FBt_purity‚0.828125 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7299906611442566 'particle/f_score‚0.6084818840026855 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9785764813423157 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.8148148059844971 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 7"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.68413245677948 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6784999966621399 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.13550154864788055 4 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9802206158638 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9762846827507019 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.010561845265328884 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 * Purity/*FRt1SRqqt2FBt/FBt_purity‚0.875 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9861180186271667 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.02471582405269146 +jet/accuracy_2_of_2‚0.13027137517929077 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.2239583283662796 epoch‚36 6 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.988335371017456 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.20752611756324768 *jet/accuracy_1_of_3‚0.9015376567840576 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5239419937133789 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.18807613849639893 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2623845934867859 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.1413891464471817 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.21150945127010345 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.11464817076921463 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4000000059604645 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6423277854919434 jet/accuracy_4_of_5‚0 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4476591646671295 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9860045313835144 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.11493271589279175 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5224999785423279 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 +jet/accuracy_3_of_3‚0.04900762438774109 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6481733918190002 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.13951480388641357 +particle/specificity‚0.9014005064964294ìX0«!Ñ:‹| 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.19211623072624207 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8172259330749512 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9939800500869751 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.8903210759162903 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.02470415271818638 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7284084558486938 *jet/accuracy_1_of_2‚0.7189023494720459 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7142857313156128 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8313804864883423 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9862018823623657 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8201090693473816 particle/accuracy_2_of_2‚1 +"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.25 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 - Purity/0FRt2SRqqt1FBt/FBt_purity‚0.828125 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7299906611442566 'particle/f_score‚0.6084818840026855 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9785764813423157 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.8148148059844971 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 7"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.68413245677948 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6784999966621399 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.13550154864788055 4 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9802206158638 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9762846827507019 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.010561845265328884 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 * Purity/*FRt1SRqqt2FBt/FBt_purity‚0.875 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9861180186271667 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.02471582405269146 +jet/accuracy_2_of_2‚0.13027137517929077 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.2239583283662796 epoch‚36 6 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.988335371017456 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.20752611756324768 *jet/accuracy_1_of_3‚0.9015376567840576 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5239419937133789 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.18807613849639893 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2623845934867859 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.1413891464471817 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.21150945127010345 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.11464817076921463 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4000000059604645 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6423277854919434 jet/accuracy_4_of_5‚0 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4476591646671295 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9860045313835144 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.11493271589279175 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5224999785423279 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 +jet/accuracy_3_of_3‚0.04900762438774109 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6481733918190002 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.13951480388641357 +particle/specificity‚0.9014005064964294 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.15465430915355682 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.8148148059844971 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9675024151802063 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9888569116592407 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.4738966226577759 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.1082889661192894 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4583333432674408 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9898683428764343 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.12674087285995483 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.14135292172431946 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9917780756950378 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.2070690542459488 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6791369318962097 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.032771751284599304 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.09885440766811371 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8201090693473816 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6788597702980042 *loss/val_total_loss‚10.992282137870461 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 8"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.733178436756134 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.8846153616905212 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7857142686843872 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 6 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.976502001285553 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7268518209457397 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.05000000074505806 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.02471582405269146 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.6835978627204895 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5913946628570557 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.8999999761581421 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3434205651283264 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.4976852238178253 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.5938971042633057 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7317311763763428 particle/accuracy_2_of_3‚1 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 8"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.608867883682251 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9966966509819031 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.3898397982120514 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.05000000074505806 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.16412043571472168 particle/accuracy_1_of_1‚1 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.5792959928512573 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7403045296669006 +jet/accuracy_3_of_4‚0.05347250774502754 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.01994013413786888 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5913946628570557 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.2222222238779068 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.14178135991096497 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 jet/accuracy_2_of_5‚1 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.3051789104938507 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 $ _timestamp‚1.784314331813367e+09 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.14827671647071838 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.8999999761581421 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 jet/accuracy_3_of_5‚1 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.4921875 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.10499999672174454 *jet/accuracy_2_of_4‚0.7330344915390015 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9909146428108215 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9924677610397339 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 <"Purity/2FRt*SRqqt0FBt/event_purity‚0ojzûf.0030988692305982113 /particle/accuracy_3_of_4‚0.9974937438964844 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.23076923191547394 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9837511777877808 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 *jet/accuracy_1_of_1‚0.2802661955356598 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 6 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.682445228099823 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.2705172598361969 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.1537608951330185 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.625 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 particle/accuracy_2_of_4‚1 particle/accuracy_5_of_5‚1 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 +"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.25 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6434425711631775 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7620711326599121 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3462529182434082 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 ,jet/accuracy_4_of_4‚0.013899499550461769 particle/accuracy_1_of_4‚1 *particle/sensitivity‚0.594962477684021 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8280256986618042 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.02455909550189972 6validation_average_jet_accuracy‚0.3245682716369629 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.5963541269302368 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5934149622917175 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 particle/accuracy_4_of_5‚1 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9966966509819031 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.33166977763175964 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.6686697006225586 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.6535340547561646 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.023904161527752876 (particle/accuracy‚0.8355887532234192 _runtime‚24516.031411215 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6254529356956482 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.005507623311132193 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 particle/accuracy_2_of_5‚1 particle/accuracy_3_of_5‚1 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.22785785794258118 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.14135292172431946 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ) Purity/2FRt*SRqqt2FBt/FRt_purity‚0.75 8"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.619834303855896 jet/accuracy_5_of_5‚0 particle/accuracy_1_of_5‚1 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.6941421627998352 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6290315985679626 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.6073884963989258 *jet/accuracy_1_of_4‚0.9625317454338074 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6175370812416077 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.2222222238779068 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7571137547492981 particle/accuracy_1_of_2‚1 7 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.0242546945810318 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8888888955116272 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6451865434646606 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7268518209457397 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5792078971862793 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9888888597488403 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.02584935910999775 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9101652503013611 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.4600139856338501 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9935952425003052 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 _step‚1220 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7151234745979309 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9869948625564575 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.23076923191547394 trainer/global_step‚14799 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.12625433504581451 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4406282305717468 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.15177632868289948 6 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.997686505317688 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5284673571586609 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6555982232093811 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_3‚1 /particle/accuracy_4_of_4‚0.9289587140083313 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.1408676952123642 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.1744830459356308 - Purity/0FRt2SRqqt*FBt/FBt_purity‚0.828125 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 jet/accuracy_1_of_5‚1 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6582552194595337 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6084899306297302 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6526565551757812 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 particle/accuracy_3_of_3‚1 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.10974667221307755 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.6686697006225586 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.13349221646785736 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.5908263325691223 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9961451292037964 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 *jet/accuracy_2_of_3‚0.5050614476203918 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6435207724571228 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.12745578587055206 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6790653467178345 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9937976002693176Ä ªIŸÒ:‚(:” äóéÒ——áö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚274.958' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚54.991600000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 128636991 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚38097720 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1170663133458944 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 % _timestamp‚1.7843144091923711e+09 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚4284004934501376 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13971303179264 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚80174502 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚4270033631322112 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚4284006306064384 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚255 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚32439296 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚4282136920059861 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 255374705 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13971303179264 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 trainer/global_step‚14849 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 _step‚1222 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 126747693 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚4284004934501376 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 128627018 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚4284006306064384 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 255384684 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51¹ ú§¢á:™P ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚4282138290088151 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚4268189524824064 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 126747687 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 126747687 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 128636991 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 128627018 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13972648568832 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 400530432 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 128636991 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚38097720 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1170663133458944 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 % _timestamp‚1.7843144091923711e+09 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚4284004934501376 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13971303179264 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚80174502 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚4270033631322112 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚4284006306064384 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚255 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚32439296 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚4282136920059861 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 255374705 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13971303179264 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 trainer/global_step‚14849 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 _step‚1222 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 126747693 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚4284004934501376 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 128627018 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚4284006306064384 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 255384684 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0Æ ­æï%4â:/ _wandbruntime‚24594  _runtime‚24594ÛDBÿ$'ã:žN I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚242 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 126755899 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 128645483 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚4270311658225664 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 128635496 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚247 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1170741829868032 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 _runtime‚24594.795374794 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 255391389 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 126755893 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚4284285231837184 % _timestamp‚1.7843144105779898e+09 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚42079504 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚4268467406602240 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13948318702043 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚80179520 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚32432128 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 255401382 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚80179767 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚4282415725304283 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13972228214784 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚4270311632052224 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13972228214784 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚18019317352960 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 126755893 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 128645483 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 trainer/global_step‚14849 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚38100263 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 128635496 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚42079262 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13949663302477 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚4270311658225664 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1152722913038336 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13973573611520 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 255391389 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 255401382 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1170742230391296 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚18019284920832 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 _step‚1223 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0X;B)'ä:£N D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 255391389 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 255401382 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1170742230391296 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚18019284920832 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 _step‚1223 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚242 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 126755899 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 128645483 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚4270311658225664 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 128635496 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚247 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1170741829868032 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 _runtime‚24594.795374794 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 255391389 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 126755893 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚4284285231837184 % _timestamp‚1.7843144105779898e+09 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚42079504 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚4268467406602240 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13948318702043 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚80179520 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚32432128 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 255401382 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚80179767 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚4282415725304283 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13972228214784 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚4270311632052224 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13972228214784 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚18019317352960 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 126755893 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 128645483 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 trainer/global_step‚14849 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚38100263 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 128635496 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚42079262 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13949663302477 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚4270311658225664 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1152722913038336 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13973573611520 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0Ç —6Ä>4å:/ _wandbruntime‚24594  _runtime‚245943×aöæ:‰ 0loss/FBt1/assignment_loss‚0.0140616515892473 3loss/FBt2/assignment_loss‚0.0006045269362335753 /loss/FRt2/detection_loss‚1.2293582642004286 0loss/FRt1/detection_loss‚0.44770112404344337 0loss/FBt2/detection_loss‚0.09842239759595493 %loss/total_loss‚4.171090034789243 trainer/global_step‚14849 % _timestamp‚1.7843144106639245e+09 epoch‚37 0loss/FRt1/assignment_loss‚0.9515765265245831 3loss/SRqqt1/assignment_loss‚0.07214902471890643 2loss/SRqqt2/assignment_loss‚0.1366483665148793 1loss/SRqqt1/detection_loss‚0.4898541969801504 1loss/SRqqt2/detection_loss‚0.4705942261248607 1loss/FRt2/assignment_loss‚0.09589990648892846 /loss/FBt1/detection_loss‚0.1642198230716271 _runtime‚24594.88086828 _step‚1224‡å'ç:Ž 0loss/FRt1/detection_loss‚0.44770112404344337 0loss/FBt2/detection_loss‚0.09842239759595493 %loss/total_loss‚4.171090034789243 trainer/global_step‚14849 % _timestamp‚1.7843144106639245e+09 epoch‚37 0loss/FRt1/assignment_loss‚0.9515765265245831 3loss/SRqqt1/assignment_loss‚0.07214902471890643 2loss/SRqqt2/assignment_loss‚0.1366483665148793 1loss/SRqqt1/detection_loss‚0.4898541969801504 1loss/SRqqt2/detection_loss‚0.4705942261248607 1loss/FRt2/assignment_loss‚0.09589990648892846 /loss/FBt1/detection_loss‚0.1642198230716271 _runtime‚24594.88086828 _step‚1224 0loss/FBt1/assignment_loss‚0.0140616515892473 3loss/FBt2/assignment_loss‚0.0006045269362335753 /loss/FRt2/detection_loss‚1.2293582642004286È ­‡M<”è:‚(:‰ ¯ôéÒÑïþö gpu.0.gpu‚93.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚46.0 gpu.0.powerWatts‚340.229' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚68.0458 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0_º¡Gé:‚(:¼ ¯ôéÒµæ¼ù$ memory_percent‚83.07821750640869) proc.memory.percent‚17.54589080810547) disk./.usagePercent‚37.93198020107364 disk.sda2.out‚0" disk.nvme0c0n1.in‚ 56684.796875 disk.sda.in‚ 108.921875 disk.sda.out‚ 1424.19140625 disk.drbd2165.out‚0 disk.dm-1.out‚ 1424.19140625& disk.rbd0.in‚1.759218604441423e+13$ disk.dm-0.out‚154855.14208984375 disk.rbd3.in‚0 network.sent‚ 2.8711546e+07 disk.sda2.in‚0 disk.rbd1.in‚ 0.53564453125 disk.rbd1.out‚ 64.4287109375 disk.dm-1.in‚ 108.921875 disk.sda1.out‚0 proc.cpu.threads‚17) disk.nvme0c0n1.out‚154855.14208984375 disk.nvme0n1.in‚ 56684.796875 disk.rbd0.out‚0 disk.sda3.out‚ 1424.19140625) proc.memory.availableMB‚ 2772.46484375 cpu‚25.58197906871524 disk.sda1.in‚0 disk.rbd3.out‚ 0.4951171875 proc.memory.rssMB‚ 2874.71875$ disk./.usageGB‚565.2139167785645 disk.dm-0.in‚ 56684.796875 disk.sda3.in‚ 108.921875 disk.drbd2165.in‚0 network.recv‚ 6.126875e+06' disk.nvme0n1.out‚154855.14208984375ꨞê:‚(:“ ¾ôéÒ©ù÷ gpu.0.gpu‚98.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚269.376' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚53.87519999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ù9 ­Bë:‚(:· ¾ôéÒ·´óù disk.rbd3.out‚ 0.4951171875 disk.rbd1.out‚ 64.4287109375 disk.sda.out‚ 1424.45703125 disk.sda3.out‚ 1424.45703125 disk.sda2.out‚0 disk.rbd1.in‚ 0.53564453125 proc.cpu.threads‚17# disk./.usageGB‚565.221019744873& disk.rbd0.in‚1.759218604441423e+13 disk.sda1.in‚0 disk.drbd2165.in‚0 disk.drbd2165.out‚0' disk.nvme0n1.out‚154867.12744140625 disk.dm-1.out‚ 1424.45703125$ memory_percent‚83.07497501373291$ disk.dm-0.out‚154867.12744140625 disk.dm-1.in‚ 108.921875 network.sent‚ 2.8774856e+07 disk.nvme0n1.in‚ 57050.90625 disk.sda2.in‚0 disk.sda3.in‚ 108.921875 disk.rbd0.out‚0 disk.rbd3.in‚0) disk.nvme0c0n1.out‚154867.12744140625 disk.sda1.out‚0! disk.nvme0c0n1.in‚ 57050.90625 disk.sda.in‚ 108.921875) proc.memory.availableMB‚ 2772.99609375 proc.memory.rssMB‚ 2874.71875) proc.memory.percent‚17.54589080810547 cpu‚25.58185721638582 network.recv‚ 6.131385e+06( disk./.usagePercent‚37.9324568871024 disk.dm-0.in‚ 57050.90625#Šñ”ì:‚(:‰ ÍôéÒ¬Êíö gpu.0.gpu‚92.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚46.0 gpu.0.powerWatts‚343.993' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚68.7986 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0£úŽ>í:‚(:³ ÍôéÒøªåù$ disk./.usageGB‚565.2279853820801 disk.sda1.out‚0 disk.sda3.in‚ 108.921875 disk.sda2.out‚0$ disk.dm-0.out‚154875.34716796875 disk.sda1.in‚0 disk.rbd3.in‚0 disk.sda2.in‚0 disk.rbd1.in‚ 0.53564453125 disk.dm-1.out‚ 1424.71875 disk.rbd0.out‚0 disk.nvme0n1.in‚ 57051.328125' disk.nvme0n1.out‚154875.34716796875) proc.memory.availableMB‚ 2772.75390625) disk./.usagePercent‚37.93292435685992 disk.sda3.out‚ 1424.71875 disk.sda.out‚ 1424.71875 network.recv‚ 6.134738e+06 disk.drbd2165.in‚0$ memory_percent‚83.07645320892334 proc.memory.rssMB‚ 2874.71875 proc.cpu.threads‚17 network.sent‚ 2.8778726e+07& disk.rbd0.in‚1.759218604441423e+13 cpu‚25.58173667465275 disk.dm-1.in‚ 108.921875 disk.dm-0.in‚ 57051.328125" disk.nvme0c0n1.in‚ 57051.328125) disk.nvme0c0n1.out‚154875.34716796875 disk.sda.in‚ 108.921875 disk.drbd2165.out‚0 disk.rbd3.out‚ 0.4951171875 disk.rbd1.out‚ 64.4287109375) proc.memory.percent‚17.54589080810547AÝëCŸî:‚(:” ÜôéÒ¶Öüö gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚298.451' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚59.690200000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0å ö9ï:‚(:® ÜôéÒûßßù# disk.dm-0.out‚154892.6904296875 disk.drbd2165.in‚0 disk.rbd3.in‚0" disk.nvme0c0n1.in‚ 57065.984375 disk.rbd1.out‚ 64.5068359375) proc.memory.availableMB‚ 2772.48828125 cpu‚25.581605969280087$ disk./.usageGB‚565.2323455810547 disk.dm-1.in‚ 108.921875 disk.sda3.in‚ 108.921875( disk.nvme0c0n1.out‚154892.6904296875 disk.sda.in‚ 108.921875 network.recv‚ 6.138264e+06& disk.rbd0.in‚1.759218604441423e+13& disk.nvme0n1.out‚154892.6904296875 disk.rbd1.in‚ 0.53564453125 proc.memory.rssMB‚ 2874.71875 disk.rbd0.out‚0 disk.sda2.in‚0) proc.memory.percent‚17.54589080810547 network.sent‚ 2.8783802e+07) disk./.usagePercent‚37.93321697347156 disk.dm-0.in‚ 57065.984375 disk.sda1.in‚0 disk.sda3.out‚ 1424.8125 disk.dm-1.out‚ 1424.8125 disk.drbd2165.out‚0 disk.sda.out‚ 1424.8125 disk.sda1.out‚0 disk.rbd3.out‚ 0.4951171875 disk.nvme0n1.in‚ 57065.984375 disk.sda2.out‚0$ memory_percent‚83.07807445526123 proc.cpu.threads‚17X…”ð:‚(:‰ ëôéÒÓùž÷ gpu.0.gpu‚47.0 gpu.0.memory‚15, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚337.922' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚67.5844 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0sU”ïHñ:‚(:½ ëôéÒŸÕÇù disk.rbd3.out‚ 0.4951171875 network.recv‚ 6.141392e+06$ disk.dm-0.out‚154901.12841796875 disk.sda1.in‚0 proc.memory.rssMB‚ 2874.71875 proc.cpu.threads‚17 disk.drbd2165.in‚0# proc.memory.availableMB‚2772.25 disk.rbd3.in‚0 disk.sda.out‚ 1425.14453125$ memory_percent‚83.07952880859375 disk.sda.in‚ 108.921875 disk.sda3.out‚ 1425.14453125 disk.dm-1.out‚ 1425.14453125 disk.dm-0.in‚57136.56640625 disk.drbd2165.out‚0" disk.nvme0n1.in‚57136.56640625 disk.rbd0.out‚0' disk.nvme0n1.out‚154901.12841796875& disk.rbd0.in‚1.759218604441423e+13$ disk.nvme0c0n1.in‚57136.56640625) disk.nvme0c0n1.out‚154901.12060546875 disk.sda2.out‚0) proc.memory.percent‚17.54589080810547 cpu‚25.581496018963623 disk.sda3.in‚ 108.921875 disk.sda2.in‚0 disk.rbd1.in‚ 0.53564453125 disk.rbd1.out‚ 64.5068359375 network.sent‚ 2.8788882e+07) disk./.usagePercent‚37.93367394692018$ disk./.usageGB‚565.2391548156738 disk.dm-1.in‚ 108.921875 disk.sda1.out‚0Ï‘×4ò:/ _wandbruntime‚24672  _runtime‚24672‡Ö-±ó:Ä & lr-AdamW/pg1‚0.0001093249683221419 & lr-AdamW/pg2‚0.0001093249683221419 trainer/global_step‚14899 % _timestamp‚1.7843144878754215e+09 _runtime‚24672.0q092578638 _step‚1225ŸñÒ`Ïô:É trainer/global_step‚14899 % _timestamp‚1.7843144878754215e+09 _runtime‚24672.092578638 _step‚1225 & lr-AdamW/pg1‚0.0001093249683221419 & lr-AdamW/pg2‚0.0001093249683221419É õAü4õ:/ _wandbruntime‚24672  _runtime‚24672ü‚ÂX(ö:”P ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 127157993 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚42215452 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14017619109888 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚38224865 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 256210757 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 129062743 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14018964499456 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚4296077246081769 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1174597862513152 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1174597459885568 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 127157987 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 256220736 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚18078115498496 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13993632348905 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 127157993 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚4282083639160832 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7843144878766983e+09 _runtime‚24672.094083399 _step‚1226 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 129052770 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚4297952658500608 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚4297951286937600 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚4297952658500608 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚4297951286937600 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚4283933667827712 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14018964499456 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚14899 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚270 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13994976949227 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 »,g(÷:™P RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 256220736 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚18078115498496 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13993632348905 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 127157993 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚4282083639160832 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7843144878766983e+09 _runtime‚24672.094083399 _step‚1226 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 129052770 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚4297952658500608 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚4297951286937600 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚4297952658500608 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚4297951286937600 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚4283933667827712 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14018964499456 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚14899 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚270 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13994976949227 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 127157993 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚42215452 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14017619109888 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚38224865 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 256210757 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 129062743 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14018964499456 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚4296077246081769 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1174597862513152 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1174597459885568 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 127157987 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0Ê }@DN4ø:/ _wandbruntime‚24673  _runtime‚24673nr+T#'ù:N OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 129071235 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚4284211668557824 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 256227441 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚4296357421354593 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1174676539794944 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1174676137174528 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13994555815151 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 $ _timestamp‚1.784314489261212e+09 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚4296356051326191 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13995900415585 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 127166193 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 256237434 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14018544145408 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 129061248 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 256237434 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 256227441 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚18079306372608 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 127166199 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚4284211668557824 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _step‚1227 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 127166193 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14019889542144 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1156596865331200 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚80445635H¼ãú:¢N J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 127166199 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚4284211668557824 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _step‚1227 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 127166193 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14019889542144 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1156596865331200 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚80445635 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 129071235 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚4284211668557824 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 256227441 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚4296357421354593 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1174676539794944 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1174676137174528 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13994555815151 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 $ _timestamp‚1.784314489261212e+09 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚4296356051326191 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13995900415585 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 127166193 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 256237434 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14018544145408 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 129061248 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 256237434 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 256227441 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚18079306372608 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 EI”ˆ;‚(:‰ ÅõéÒ—°‹÷ gpu.0.gpu‚90.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚297.922' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚59.5844 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¯c^C‰;‚(:¸ ÅõéÒ×ÛÛù disk.sda.in‚ 108.921875 disk.sda3.out‚ 1426.90625) disk.nvme0c0n1.out‚154971.01123046875 proc.cpu.threads‚17 disk.rbd1.in‚ 0.53564453125 disk.drbd2165.out‚0$ disk.nvme0c0n1.in‚57350.17578125" disk.nvme0n1.in‚57350.17578125' disk.nvme0n1.out‚154971.01123046875 network.sent‚ 2.8879326e+07$ disk./.usageGB‚565.2768516540527$ memory_percent‚83.07814598083496( proc.memory.availableMB‚ 2772.4765625 disk.dm-0.in‚57350.17578125 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda.out‚ 1426.90625 disk.sda1.in‚0 disk.rbd3.in‚0 disk.sda2.out‚0 cpu‚25.580775766175975) disk./.usagePercent‚37.93620381337354 disk.dm-1.in‚ 108.921875& disk.rbd0.in‚1.759218604441423e+13$ disk.dm-0.out‚154971.01123046875 network.recv‚ 6.16369e+06 disk.rbd0.out‚0 disk.rbd3.out‚ 0.4951171875 disk.rbd1.out‚ 64.7099609375 proc.memory.rssMB‚ 2874.71875 disk.dm-1.out‚ 1426.90625 disk.sda3.in‚ 108.921875 disk.drbd2165.in‚0) proc.memory.percent‚17.54589080810547²Euˆ4Š;/ _wandbruntime‚24750  _runtime‚24750…JÆIË‹;Å ' lr-AdamW/pg1‚0.00010731012690754796 ' lr-AdamW/pg2‚0.00010731012690754796 trainer/global_step‚14949 $ _timestamp‚1.784314566671387e+09 _runtime‚24750.888564329 _step‚1229ð”¶ÐŒ;Ê _runtime‚24750.888564329 _step‚1229 ' lr-AdamW/pg1‚0.00010731012690754796 ' lr-AdamW/pg2‚0.00010731012690754796 trainer/global_step‚14949 $ _timestamp‚1.784314566671387e+09Í ˆ•^“4;/ _wandbruntime‚24750  _runtime‚24750K¨(Ž;”P WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14063935040512 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚80706726 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚4310018942131967 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚38352015 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚4297833730506752 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14065280430080 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 257056788 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚18137792397824 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14041214062335 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 129488495 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚38352020 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14063935040512 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚4311897639373824 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚4311899010936832 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚4311897639373824 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 129478522 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚42354441 trainer/global_step‚14949 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1178531894332928 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1178531491705344 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 _step‚1230 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 257046809 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚4295977728069632 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640Ê1@¡ ;™P TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 129488495 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚38352020 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14063935040512 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚4311897639373824 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚4311899010936832 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 BDeviceStatsMonitor.on_train_batch_stÄg®Ö~art/active_bytes.all.freed‚4311897639373824 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 129478522 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚42354441 trainer/global_step‚14949 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1178531894332928 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1178531491705344 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 _step‚1230 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 257046809 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚4295977728069632 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14063935040512 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚80706726 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚4310018942131967 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚38352015 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚4297833730506752 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14065280430080 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 257056788 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚18137792397824 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14041214062335Î ¢F?4;/ _wandbruntime‚24752  _runtime‚24752=ƒY†$'‘;žN WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 trainer/global_step‚14949 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚4298111731236864 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚38354558 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 257073486 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 257073486 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 129496987 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 _step‚1231 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 257063493 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚4298111731236864 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚4298111705063424 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 127576493 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚42357199 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14066205472768 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚4296255609847808 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 127576499 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚38354563 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚4312176565139456 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚80711757 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚42357454 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚18139020134912 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14064860076032 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1178610573941248 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚4298111705063424 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚4296255584419840 % _timestamp‚1.7843145680585432e+09 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚4312177936709632 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚4310297747376501 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0DøæV)'’;£N OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 129496987 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚260 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 257063493 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1160471553806336 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚4312176565139456 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚80711757 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚42357454 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚18139020134912 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14064860076032 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1178610573941248 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚4298111705063424 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚4296255584419840 % _timestamp‚1.7843145680585432e+09 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚4312177936709632 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚4310297747376501 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 trainer/global_step‚14949 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚4298111731236864 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚38354558 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 257073486 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 257073486 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 129496987 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 _step‚1231 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 257063493 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚4298111731236864 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚4298111705063424 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 127576493 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚42357199 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14066205472768 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚4296255609847808 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 127576499 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚38354563 B—;‚(:³ ÔõéÒ…–Îù' disk.nvme0n1.out‚154986.11669921875 disk.rbd0.out‚0 disk.dm-0.in‚ 57481.28125) proc.memory.percent‚17.54589080810547 disk.sda1.out‚0 disk.drbd2165.out‚0 disk.nvme0n1.in‚ 57481.28125 disk.rbd1.in‚ 0.53564453125 disk.sda.in‚ 108.921875 disk.sda.out‚ 1426.984375 disk.dm-1.in‚ 108.921875$ disk.dm-0.out‚154986.11669921875 network.recv‚ 6.167016e+06 disk.drbd2165.in‚0 disk.sda2.in‚0 disk.rbd1.out‚ 64.7880859375 disk.sda1.in‚0$ memory_percent‚83.07931423187256) proc.memory.availableMB‚ 2772.28515625 proc.cpu.threads‚17 network.sent‚ 2.8883043e+07 disk.sda3.in‚ 108.921875! disk.nvme0c0n1.in‚ 57481.28125) disk.nvme0c0n1.out‚154986.11669921875$ disk./.usageGB‚565.2811698913574 disk.sda3.out‚ 1426.984375 disk.rbd3.in‚0 disk.rbd3.out‚ 0.4951171875 disk.sda2.out‚0 proc.memory.rssMB‚ 2874.71875 cpu‚25.580666608828214( disk./.usagePercent‚37.9364936139023 disk.dm-1.out‚ 1426.984375& disk.rbd0.in‚1.759218604441423e+13’PìÀž˜;‚(:“ ãõéÒ¢¨éö gpu.0.gpu‚89.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚279.79' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚55.958000000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Æ«(H™;‚(:½ ãõéÒžÉù disk.sda3.out‚ 1427.23828125 disk.rbd3.out‚ 0.4951171875) disk.nvme0c0n1.out‚154994.34326171875 proc.memory.rssMB‚ 2874.71875) proc.memory.percent‚17.54589080810547 proc.cpu.threads‚17$ disk./.usageGB‚565.2879753112793 disk.sda1.out‚0 disk.sda3.in‚ 108.921875 disk.rbd3.in‚0' disk.nvme0n1.out‚154994.34326171875 disk.sda2.in‚0 network.sent‚ 2.8946599e+07 disk.dm-1.in‚ 108.921875& disk.rbd0.in‚1.759218604441423e+13 disk.sda.in‚ 108.921875 disk.sda2.out‚0 disk.rbd1.in‚ 0.53564453125( proc.memory.availableMB‚ 2772.0390625 disk.dm-1.out‚ 1427.23828125 disk.drbd2165.in‚0# disk.nvme0c0n1.in‚ 57596.3203125 cpu‚25.58053726205936 network.recv‚ 6.171476e+06 disk.rbd0.out‚0 disk.dm-0.in‚ 57596.3203125$ disk.dm-0.out‚154994.34326171875 disk.sda1.in‚0 disk.drbd2165.out‚0! disk.nvme0n1.in‚ 57596.3203125# memory_percent‚83.0808162689209 disk.sda.out‚ 1427.23828125 disk.rbd1.out‚ 64.7880859375) disk./.usagePercent‚37.93695033134338›Ý'‘š;‚(:† òõéÒ§€èö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚146.95' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚29.39 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Í[Éþ›;‚(:½ òõéÒɲù disk.rbd3.in‚0 disk.sda.out‚ 1427.4765625) proc.memory.availableMB‚ 2771.96484375 proc.cpu.threads‚17$ disk./.usageGB‚565.2947044372559 disk.sda2.out‚0 disk.dm-1.out‚ 1427.4765625 disk.sda1.out‚Í ÓUJ0) disk.nvme0c0n1.out‚155005.09912109375 proc.memory.rssMB‚ 2874.71875 disk.rbd1.in‚ 0.53564453125 disk.rbd0.out‚0 disk.dm-0.in‚ 57601.2578125$ memory_percent‚83.08126926422119 cpu‚25.580408387762407 disk.rbd1.out‚ 64.8505859375 disk.sda3.out‚ 1427.4765625 disk.drbd2165.in‚0 disk.dm-1.in‚ 108.921875 disk.drbd2165.out‚0 disk.rbd3.out‚ 0.4951171875# disk.nvme0c0n1.in‚ 57601.2578125 network.recv‚ 6.174863e+06 disk.sda.in‚ 108.921875) proc.memory.percent‚17.54589080810547 disk.sda2.in‚0$ disk.dm-0.out‚155005.09912109375 network.sent‚ 2.8951677e+07) disk./.usagePercent‚37.93740192863378! disk.nvme0n1.in‚ 57601.2578125' disk.nvme0n1.out‚155005.09912109375& disk.rbd0.in‚1.759218604441423e+13 disk.sda1.in‚0 disk.sda3.in‚ 108.921875½ö±V•œ;‚(:Š öéÒª¥Š÷ gpu.0.gpu‚100.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚339.701' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚67.9402 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¯ÌrI;‚(:¾ öéÒòçÒù# disk.dm-0.out‚155017.3564453125 disk.drbd2165.out‚0) disk./.usagePercent‚37.93776597134747 disk.rbd1.out‚ 64.8505859375 disk.dm-0.in‚57690.61328125 proc.memory.rssMB‚ 2874.71875 network.sent‚ 2.8958185e+07 disk.sda1.out‚0 disk.sda2.out‚0$ disk.nvme0c0n1.in‚57690.61328125" disk.nvme0n1.in‚57690.61328125& disk.nvme0n1.out‚155017.3564453125) proc.memory.availableMB‚ 2771.80078125 disk.rbd0.out‚0 disk.sda1.in‚0) proc.memory.percent‚17.54589080810547$ disk./.usageGB‚565.3001289367676 disk.sda.in‚ 108.921875 disk.sda.out‚ 1427.82421875 disk.dm-1.out‚ 1427.82421875& disk.rbd0.in‚1.759218604441423e+13 disk.sda3.out‚ 1427.82421875 disk.rbd3.in‚0$ memory_percent‚83.08227062225342 network.recv‚ 6.17817e+06 disk.drbd2165.in‚0 cpu‚25.58029887682668 proc.cpu.threads‚17 disk.rbd1.in‚ 0.53564453125 disk.sda3.in‚ 108.921875 disk.rbd3.out‚ 0.4951171875( disk.nvme0c0n1.out‚155017.3564453125 disk.sda2.in‚0 disk.dm-1.in‚ 108.921875çØ”ž;‚(:‰ öéÒ›Êßö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚290.327' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚58.0654 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Dý×ÐJŸ;‚(:¿ öéÒœê˜ù disk.sda1.out‚0) proc.memory.percent‚17.54589080810547 disk.drbd2165.out‚0' disk.nvme0n1.out‚155028.69873046875$ disk.dm-0.out‚155028.69873046875 disk.sda1.in‚0 disk.sda3.in‚ 108.921875 disk.dm-1.in‚ 108.921875 disk.rbd0.out‚0 disk.sda3.out‚ 1428.1484375 disk.drbd2165.in‚0 disk.sda2.in‚0 disk.rbd1.out‚ 64.9287109375 disk.sda.out‚ 1428.1484375& disk.rbd0.in‚1.759218604441423e+13) proc.memory.availableMB‚ 2771.78515625" disk.nvme0n1.in‚57699.61328125 disk.sda.in‚ 108.921875 disk.dm-1.out‚ 1428.1484375 disk.rbd3.in‚0 proc.cpu.threads‚17 network.sent‚ 2.8962008e+07 network.recv‚ 6.181524e+06) disk./.usagePercent‚37.93680261499613) disk.nvme0c0n1.out‚155028.69873046875 disk.sda2.out‚0 cpu‚25.580180942704928 disk.dm-0.in‚57699.61328125# disk./.usageGB‚565.285774230957 disk.rbd3.out‚ 0.4951171875$ disk.nvme0c0n1.in‚57699.61328125$ memory_percent‚83.08236598968506 proc.memory.rssMB‚ 2874.71875 disk.rbd1.in‚ 0.53564453125€"ãŸ4 ;/ _wandbruntime‚24829  _runtime‚24829fúçÌ¡;Æ ' lr-AdamW/pg1‚0.00010531061123064824 ' lr-AdamW/pg2‚0.00010531061123064824 trainer/global_step‚14999 % _timestamp‚1.7843146456909838e+09 _runtime‚24829.908227204 _step‚1233ËS²Ñ¢;Ë ' lr-AdamW/pg2‚0.00010531061123064824 trainer/global_step‚14999 % _timestamp‚1.7843146456909838e+09 _runtime‚24829.908227204 _step‚1233 ' lr-AdamW/pg1‚0.00010531061123064824Ñ t0Ñ4£;/ _wandbruntime‚24829  _runtime‚24829×-Y(¤;”P O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚4311733740838912 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14086106575121 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1164268387782656 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚42492444 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 257892840 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 127978593 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 129904274 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 257882861 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14087451175443 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚4309871816978432 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚4325843991810048 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 129914247 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 127978593 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14110250971136 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚42492200 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 127978587 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚80971365 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 129914247 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14111596360704 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚38479165 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1182465992752128 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1182465592221696 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 127978587 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚4325843991810048 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚4311733767012352 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14111596360704 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚38479170 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 _step‚1234 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚32439296 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚4325845363373056 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1164268019691520 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1164268387782656 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚42492444 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 257892840 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 127978593 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 129904274 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 257882861 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14087451175443 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚4309871816978432 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚4325843991810048 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 129914247 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 127978593 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14110250971136 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚42492200 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 127978587 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚80971365 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 129914247 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14111596360704 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚38479165 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1182465992752128 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1182465592221696 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 127978587 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚4325843991810048 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚4311733767012352 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14111596360704 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚38479170Ò ìÕìk4¦;/ _wandbruntime‚24831  _runtime‚24831x4¯ú-§;žN TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1164345874190336 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 127986793 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚38481708 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚42494948 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1182544675211264 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚80976901 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14112521403392 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚4326124289145856 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 127986799 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 129912752 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚24831.300257692 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 127986799 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚4326122917575680 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _step‚1235 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1182544272590848 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚4324238073398409 trainer/global_step‚14999 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14111176006656 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14088374641801 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 129922739 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚245 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 129912752 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 129922739 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚18198801020928 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚4312011741569024 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14111176006656 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 % _timestamp‚1.7843146470828567e+09 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚80976656 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚4326124289145856 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14112521403392 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 257899545 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 257909538 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 257909538 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚4312011767742464 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚38481713 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚240 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14087030041367 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820XPJÝ)'¨;£N VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚42495188 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚4312011741569024 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚80976656 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚4326124289145856 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14112521403392 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 257899545 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 257909538 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 257909538 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚4312011767742464 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚38481713 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚240 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14087030041367 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1164345874190336 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 127986793 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚38481708 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚42494948 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1182544675211264 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚80976901 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14112521403392 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚4326124289145856 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 127986799 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 129912752 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚24831.300257692 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 127986799 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚4326122917575680 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _step‚1235 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1182544272590848 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚4324238073398409 trainer/global_step‚14999 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14111176006656 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14088374641801 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 129922739 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚245 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 129912752 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 129922739 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚18198801020928 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚4312011741569024 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14111176006656 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 % _timestamp‚1.7843146470828567e+09 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 QDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 128388887 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14132343688229 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1168142340075520 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 130339999 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚4337898224147493 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 258728892 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚32439296 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚4339791715809280 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14157912291328 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚4339790344246272 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚42629720 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚4325633803517952 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14156566901760 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚24908.990916693 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 128388893 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 128388893 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚81236285 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚4323765905887232 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 128388887 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚42629965 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1186399679905792 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚4323765880459264 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 130330026 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚4339791715809280 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚4325633803517952 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0tÜi»;™P QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 128388887 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14132343688229 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1168142340075520 Rh ˆ+¶&BDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 130339999 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚4337898224147493 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 258728892 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚32439296 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚4339791715809280 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14157912291328 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚4339790344246272 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚42629720 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚4325633803517952 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14156566901760 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚24908.990916693 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 128388893 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 128388893 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚81236285 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚4323765905887232 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 128388887 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚42629965 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1186399679905792 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚4323765880459264 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 130330026 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚4339791715809280 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚4325633803517952 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14133688288551 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 258718913 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚81236035 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚245 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1186400080436224 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚4339790344246272 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚38606320 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚18257339830272 _step‚1238 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0Ö D.\¸4¼;/ _wandbruntime‚24910  _runtime‚24910·ëâ7#'½;N OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 128397093 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚4324043762237440 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚4325911804248064 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 128397099 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚4324043787665408 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚4340070641582080 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14134611754909 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚18258540673024 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚4340070641582080 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚81241305 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1168220194574336 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1168219826483200 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚4338177029391915 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚38608863 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚42632684 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14133267154475 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 258745590 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚42632447 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1186478767679488 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14158837334016 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚4325911778074624 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚81241547 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 400523264 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 130348491 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 130338504 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚4325911778074624 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14158837334016 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚4338178399420317 trainer/global_step‚15049 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 258745590 _step‚1239 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚38608858 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 128397099 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 130348491 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 128397093 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚4340069270011904 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚242 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 258735597 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14157491937280 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚32432128 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 258745590 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚42632447 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1186478767679488 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14158837334016 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚4325911778074624 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚81241547 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 400523264 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 130348491 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 130338504 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚4325911778074624 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14158837334016 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚4338178399420317 trainer/global_step‚15049 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 258745590 _step‚1239 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚38608858 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 128397099 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 130348491 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 128397093 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚4340069270011904 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚242 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 258735597 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14157491937280 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚32432128 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚4324043787665408 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚4340070641582080 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14134611754909 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚18258540673024 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚4340070641582080 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚81241305 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1168220194574336 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1168219826483200 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚4338177029391915 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648× °DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚38733465 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚4339533813850112 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚18317406742528 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚4353736696682496 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1190334067202048 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14178580801337 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚38733470 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚4337659969368064 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 130755778 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 128799187 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚258 $ _timestamp‚1.784314804551877e+09 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1190334469829632 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚4339533840023552 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚4339533813850112 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚4339533840023552 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 128799193 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14204228221952 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚4353736696682496 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1190334067202048 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14178580801337 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚38733470 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚4337659969368064 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 130755778 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 128799187 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚258 $ _timestamp‚1.784314804551877e+09 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1190334469829632 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚4339533840023552 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚4339533813850112 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚4339533840023552 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 128799193 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14204228221952 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚38733465 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚4339533813850112 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚18317406742528Ú ?Ý"ô”Ò;‚(:‰ µ÷éÒÝÈöø gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚287.327' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚57.4654 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ØÈ¥Ó;‚(:ý µ÷éÒÂÄñù disk.sda1.out‚0# proc.memory.availableMB‚2771.75) disk./.usagePercent‚37.88876817735415 disk.sda3.in‚ 108.921875 disk.drbd2165.in‚0 disk.rbd1.out‚ 65.3349609375 disk.rbd0.out‚0 disk.sda3.out‚ 1435.3046875( disk.nvme0c0n1.out‚155246.3603515625 disk.sda2.out‚0 disk.dm-0.in‚ 58066.484375) proc.memory.percent‚17.54589080810547 cpu‚25.579900445821583" disk.nvme0c0n1.in‚ 58066.484375 disk.sda.in‚ 108.921875 disk.dm-1.in‚ 108.921875 proc.cpu.threads‚17 network.recv‚ 6.220845e+06& disk.nvme0n1.out‚155246.3603515625$ disk./.usageGB‚564.5700263977051 disk.sda.out‚ 1435.3046875# disk.dm-0.out‚155246.3603515625 proc.memory.rssMB‚ 2874.71875 network.sent‚ 2.9133886e+07 disk.drbd2165.out‚0 disk.rbd1.in‚ 0.53564453125 disk.dm-1.out‚ 1435.3046875$ memory_percent‚83.08258056640625 disk.sda1.in‚0 disk.nvme0n1.in‚ 58066.484375 disk.sda2.in‚0& disk.rbd0.in‚1.759218604441423e+13‹¼R4Ô;/ _wandbruntime‚24990  _runtime‚24990¥EÓ¿Õ;žN [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚4339811814580224 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚4339811840753664 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1172094514958336 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 130774243 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 128807399 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚81506746 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚4339811814580224 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚4354015622448128 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 130774243 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚4354016994018304 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 128807393 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 259571649 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 128807399 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚4352117355413823 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14180848868017 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1172094146867200 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚4339811840753664 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚4337937851146240 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚18318596829184 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14179504267583 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚4354016994018304 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚38736013 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚42770738 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚18318631358464 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14205153264640 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚257 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 _step‚1243 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14203807867904 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1190413146316800 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1190412743696384 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 259581642 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136\BƒF)'Ö;£N L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 259581642 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚4339811814580224 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚4339811840753664 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1172094514958336 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 130774243 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 128807399 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚81506746 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚4339811814580224 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚4354015622448128 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 130774243 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚4354016994018304 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 128807393 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 259571649 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 128807399 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚4352117355413823 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14180848868017 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1172094146867200 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚4339811840753664 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚4337937851146240 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚18318596829184 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14179504267583 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚4354016994018304 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚38736013 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚42770738 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚18318631358464 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14205153264640 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚257 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 _step‚1243 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14203807867904 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1190413146316800 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1190412743696384Û Ž×4×;/ _wandbruntime‚24990  _runtime‚24990–ŸØ; trainer/global_step‚15099 % _timestamp‚1.7843148060250366e+09 4loss/FBt1/assignment_loss‚0.00042036973419903656 4loss/FBt2/assignment_loss‚3.0081576912754926e-07 0loss/FBt1/detection_loss‚0.45804842884153446 /loss/FBt2/detection_loss‚0.2190298441239662 epoch‚37 3loss/SRqqt1/assignment_loss‚0.08997800110418382 1loss/SRqqt1/detection_loss‚0.4656502837901096 2loss/SRqqt2/detection_loss‚0.40523312263014677 _step‚1244 1loss/FRt2/assignment_loss‚0.17845375757468374 0loss/FRt1/detection_loss‚0.43164559901691135 /loss/FRt2/detection_loss‚1.2080811179618245 _runtime‚24990.241974392 0loss/FRt1/assignment_loss‚0.9137536752109344 2loss/SRqqt2/assignment_loss‚0.1445238989806326 %loss/total_loss‚4.514818399784895ÜQæÙ;• %loss/total_loss‚4.514818399784895 trainer/global_step‚15099 % _timestamp‚1.7843148060250366e+09 4loss/FBt1/assignment_loss‚0.00042036973419903656 4loss/FBt2/assignment_loss‚3.0081576912754926e-07 0loss/FBt1/detection_loss‚0.45804842884153446 /loss/FBt2/detection_loss‚0.2190298441239662 epoch‚37 3loss/SRqqt1/assignment_loss‚0.08997800110418382 1loss/SRqqt1/detection_loss‚0.4656502837901096 2loss/SRqqt2/detection_loss‚0.40523312263014677 _step‚1244 1loss/FRt2/assignment_loss‚0.17845375757468374 0loss/FRt1/detection_loss‚0.43164559901691135 /loss/FRt2/detection_loss‚1.2080811179618245 _runtime‚24990.241974392 0loss/FRt1/assignment_loss‚0.9137536752109344 2loss/SRqqt2/assignment_loss‚0.1445238989806326Ü †i‰ÿÚ;‚(:’ Ä÷éÒ¢²äø gpu.0.gpu‚4.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚40.0 gpu.0.powerWatts‚109.757' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.951400000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0N!6¾Û;‚(:Œ Ä÷éÒçßäù disk.rbd1.out‚ 65.4130859375 disk.sda1.out‚0) disk./.usagePercent‚37.94410702196307$ disk./.usageGB‚565.3946151733398 disk.drbd2165.in‚0 disk.sda2.out‚0# disk.dm-0.out‚155773.3271484375 proc.memory.rssMB‚ 2874.71875 cpu‚25.580022598782424 disk.dm-1.in‚ 108.921875 disk.dm-0.in‚58071.41796875) proc.memory.availableMB‚ 2771.68359375 network.recv‚ 6.225273e+06 disk.sda3.in‚ 108.921875 disk.rbd1.in‚ 0.53564453125& disk.rbd0.in‚1.759218604441423e+13 disk.sda1.in‚0 disk.sda3.out‚ 1439.84765625 disk.drbd2165.out‚0( disk.nvme0c0n1.out‚155773.3271484375 disk.dm-1.out‚ 1439.84765625 disk.rbd0.out‚0) proc.memory.percent‚17.54589080810547 network.sent‚ 2.9186355e+07$ disk.nvme0c0n1.in‚58071.41796875& disk.nvme0n1.out‚155773.3271484375 disk.sda2.in‚0 disk.sda.in‚ 108.921875 disk.sda.out‚ 1439.84765625$ memory_percent‚83.08298587799072 proc.cpu.threads‚17" disk.nvme0n1.in‚58071.41796875òEÑžÜ;‚(:“ Ó÷éÒ‰ËÝö gpu.0.gpu‚98.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚291.095' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚58.21900000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0r…MGÝ;‚(:‹ Ó÷éÒÚòäø network.recv‚ 6.229231e+06$ disk./.usageGB‚567.0399398803711 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd1.out‚ 65.4130859375 disk.dm-1.in‚ 108.921875 disk.sda1.out‚0 disk.drbd2165.out‚0 network.sent‚ 2.9238999e+07$ disk.nvme0c0n1.in‚58071.62890625& disk.rbd0.in‚1.759218604441423e+13 disk.sda1.in‚0) disk./.usagePercent‚38.05452614357132" disk.nvme0n1.in‚58071.62890625 disk.rbd1.in‚ 0.53564453125 disk.sda3.out‚ 1441.59765625) proc.memory.percent‚17.54589080810547 cpu‚25.580106564056234 disk.rbd0.out‚0 disk.sda3.in‚ 108.921875 disk.sda.out‚ 1441.59765625 disk.drbd2165.in‚0 proc.cpu.threads‚17& disk.nvme0n1.out‚157009.4755859375 disk.sda.in‚ 108.921875 disk.dm-0.in‚58071.62890625 proc.memory.rssMB‚ 2874.71875( disk.nvme0c0n1.out‚157009.4755859375# disk.dm-0.out‚157009.4755859375$ memory_percent‚83.06021690368652( proc.memory.availableMB‚ 2775.4140625 disk.dm-1.out‚ 1441.59765625 qlÌŸÞ;‚(:” â÷éÒЄ»÷ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚289.503' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚57.900600000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0(‹¨Éß;‚(:‡ â÷éÒž®ù$ memory_percent‚82.98659324645996 disk.nvme0n1.in‚ 58208.296875( proc.memory.availableMB‚ 2787.4765625 disk.sda2.out‚0 network.sent‚ 2.9245474e+07 disk.dm-1.out‚ 1442.6171875& disk.rbd0.in‚1.759218604441423e+13 disk.drbd2165.out‚0" proc.memory.rssMB‚ 2874.7265625 network.recv‚ 6.232658e+06) disk./.usagePercent‚38.13982478188977 disk.sda1.out‚0) disk.nvme0c0n1.out‚157573.25732421875 disk.rbd1.in‚ 0.53564453125$ disk./.usageGB‚568.3109512329102 disk.rbd0.out‚0 disk.drbd2165.in‚0" disk.nvme0c0n1.in‚ 58208.296875' disk.nvme0n1.out‚157573.25732421875) proc.memory.percent‚17.54593849182129 cpu‚25.580177639484866 disk.sda.out‚ 1442.6171875 disk.sda2.in‚0 disk.sda.in‚ 108.921875 disk.dm-1.in‚ 108.921875 disk.sda1.in‚0 disk.sda3.in‚ 108.921875 proc.cpu.threads‚17 disk.dm-0.in‚ 58208.296875$ disk.dm-0.out‚157573.25732421875 disk.sda3.out‚ 1442.6171875 disk.rbd1.out‚ 65.4755859375†(¶UŸà;‚(:” ñ÷éÒÞúÚö gpu.0.gpu‚100.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚330.913' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚66.18260000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ðDá;‚(:‡ ñ÷éÒ¥î»ø disk.sda2.in‚0 disk.rbd1.out‚ 65.4755859375 disk.rbd0.out‚0 disk.sda1.out‚0 disk.drbd2165.in‚0$ memory_percent‚82.98664093017578& disk.rbd0.in‚1.759218604441423e+13 disk.dm-0.in‚ 58208.7421875) proc.memory.percent‚17.54593849182129 proc.cpu.threads‚17 disk.dm-1.out‚ 1443.4609375 network.sent‚ 2.9250502e+07) disk./.usagePercent‚38.28326401127187 disk.rbd1.in‚ 0.53564453125 disk.sda.out‚ 1443.4609375) disk.nvme0c0n1.out‚158930.44091796875& proc.memory.availableMB‚ 2787.46875# disk./.usageGB‚570.448299407959 cpu‚25.580310498731308 disk.drbd2165.out‚0 disk.sda2.out‚0 disk.sda.in‚ 108.921875 disk.dm-1.in‚ 108.921875$ disk.dm-0.out‚158930.44091796875 disk.sda1.in‚0" proc.memory.rssMB‚ 2874.7265625' disk.nvme0n1.out‚158930.44091796875 disk.sda3.in‚ 108.921875 network.recv‚ 6.235947e+06 disk.sda3.out‚ 1443.4609375# disk.nvme0c0n1.in‚ 58208.7421875! disk.nvme0n1.in‚ 58208.7421875¦ÊfO”â;‚(:‰ €øéÒƒ¥†÷ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚268.238' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.6476 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0È%4€ã;‚(:‰ €øéÒ¦™éù( disk.nvme0c0n1.out‚161903.3935546875 disk.dm-1.out‚ 1443.7421875 disk.drbd2165.out‚0" proc.memory.rssMB‚ 2874.7265625 network.sent‚ 2.9255527e+07 disk.rbd1.in‚ 0.53564453125 disk.sda.in‚ 108.921875 disk.dm-0.in‚58209.01953125$ memory_percent‚82.98814296722412 disk.sda2.in‚0 disk.sda.out‚ 1443.7421875 disk.sda3.in‚ 108.921875$ disk.nvme0c0n1.in‚58209.01953125) proc.memory.availableMB‚ 2787.22265625 cpu‚25.5803791806018 proc.cpu.threads‚17$ disk./.usageGB‚571.0306091308594 disk.sda1.out‚0 disk.sda2.out‚0 disk.rbd1.out‚ 65.5537109375 disk.dm-1.in‚ 108.921875 disk.sda1.in‚0 disk.sda3.out‚ 1443.7421875 disk.drbd2165.in‚0) disk./.usagePercent‚38.32234330536611& disk.rbd0.in‚1.759218604441423e+13 disk.rbd0.out‚0) proc.memory.percent‚17.54593849182129 network.recv‚ 6.239529e+06# disk.dm-0.out‚161903.3935546875" disk.nvme0n1.in‚58209.01953125& disk.nvme0n1.out‚161903.3935546875r ±4ä;/ _wandbruntime‚25068  _runtime‚25068”°QSÈå; _step‚1245 & lr-AdamW/pg1‚9.940529790652515e-05 & lr-AdamW/pg2‚9.940529790652515e-05 trainer/global_step‚15149 $ _timestamp‚1.784314884212683e+09 _runtime‚25068.42988667¹}<ÒÍæ;Ç _step‚1245 & lr-AdamW/pg1‚9.940529790652515e-05 & lr-AdamW/pg2‚9.940529790652515e-05 trainer/global_step‚15149 $ _timestamp‚1.784314884212683e+09 _runtime‚25068.42988667Ý †;44ç;/ _wandbruntime‚25068  _runtime‚25068¨³Óè;’P QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 129209487 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 260391017 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 129209493 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚38860620 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1194268567851008 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14250544152576 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚42906175 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚18377218916352 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14249198763008 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 129209487 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚259 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚4351554058276864 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1175890980843520 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 trainer/global_step‚15149 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14249198763008 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 131181530 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚4365780246219599 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 _step‚1246 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 137156300ô±Î 8 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚18377184379904 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14226162514767 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 131191503 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚38860615 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚4367684420681728 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚4367683049118720 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚4367684420681728 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 260400996 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚81766790 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚42906429 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚25068.43490216 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚4365778876191309 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚4367683049118720 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14250544152576 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚254 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚4353433850355712 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚81767049Q™B.(é;—P QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 129209487 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 260391017 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 129209493 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚38860620 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1194268567851008 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14250544152576 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚42906175 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚18377218916352 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14249198763008 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 129209487 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚259 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚4351554058276864 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1175890980843520 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 trainer/global_step‚15149 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14249198763008 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 131181530 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚4365780246219599 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 _step‚1246 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚18377184379904 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14226162514767 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 131191503 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚38860615 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚4367684420681728 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚4367683049118720 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚4367684420681728 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 260400996 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚81766790 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚42906429 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚25068.43490216 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚4365778876191309 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚4367683049118720 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14250544152576 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚254 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚4353433850355712 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚81767049Þ ƒo4ê;/ _wandbruntime‚25069  _runtime‚25069ß+#'ë;N _runtime‚25069.819371427 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 260407701 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1175968835342336 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚4353711877259264 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚4366059051464133 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚4366057681435731 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚18378385574912 _step‚1247 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚4353711851085824 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14251469195264 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14250123798528 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚42908921 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 129217699 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 131199995 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 129217699 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚81772079 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚4351831965483008 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 129217693 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 131199995 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚18378420104192 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14227085981125 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 260417694 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1175968467251200 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 260407701 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚4367963346454528 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚38863163 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 260417694 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 129217693 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14250123798528 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚42909168 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚4353711877259264 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 $ _timestamp‚1.784314885601966e+09 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚4367961974884352 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 131190008 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚38863158 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1194347255446528 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚4353711851085824 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚252 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 131190008 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚81772079 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚4351831965483008 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 129217693 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 131199995 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚18378420104192 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14227085981125 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 260417694 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1175968467251200 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 260407701 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚4367963346454528 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚38863163 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 260417694 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 129217693 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14250123798528 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚42909168 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚4353711877259264 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 $ _timestamp‚1.784314885601966e+09 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚4367961974884352 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 131190008 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚38863158 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1194347255446528 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚4353711851085824 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚252 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 131190008 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 129217699 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 131199995 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 129217699 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008ß 4Ѩt4í;/ _wandbruntime‚25069  _runtime‚25069ËTýêî;‘ 0loss/FRt1/assignment_loss‚0.9489165021325735 3loss/SRqqt1/assignment_loss‚0.07036175888651301 3loss/SRqqt2/assignment_loss‚0.11779541962119106 2loss/SRqqt1/detection_loss‚0.48613064566360176 %loss/total_loss‚4.592888509611317 % _timestamp‚1.7843148856880589e+09 1loss/FRt2/assignment_loss‚0.14899021090723533 3loss/FBt1/assignment_loss‚0.0013803949353125968 trainer/global_step‚15149 /loss/FRt2/detection_loss‚1.3600876753386744 1loss/SRqqt2/detection_loss‚0.5208854577605477 0loss/FBt2/detection_loss‚0.19794027352375654 4loss/FBt2/assignment_loss‚1.8122178722101978e-06 0loss/FRt1/detection_loss‚0.40662394712968153 0loss/FBt1/detection_loss‚0.33377441149435794 epoch‚37 _runtime‚25069.905000843 _step‚1248î‚Ðäï;– 3loss/SRqqt2/assignment_loss‚0.11779541962119106 2loss/SRqqt1/detection_loss‚0.48613064566360176 %loss/total_loss‚4.592888509611317 % _timestamp‚1.7843148856880589e+09 1loss/FRt2/assignment_loss‚0.14899021090723533 3loss/FBt1/assignment_loss‚0.0013803949353125968 trainer/global_step‚15149 /loss/FRt2/detection_loss‚1.3600876753386744 1loss/SRqqt2/detection_loss‚0.5208854577605477 0loss/FBt2/detection_loss‚0.19794027352375654 4loss/FBt2/assignment_loss‚1.8122178722101978e-06 0loss/FRt1/detection_loss‚0.40662394712968153 0loss/FBt1/detection_loss‚0.33377441149435794 epoch‚37 _runtime‚25069.905000843 _step‚1248 0loss/FRt1/assignment_loss‚0.9489165021325735 3loss/SRqqt1/assignment_loss‚0.07036175888651301à ¤hMžð;‚(:“ øéÒã’ûö gpu.0.gpu‚85.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚339.809' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚67.96180000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0$NCiñ;‚(:Œ øéÒªëÝù& disk.rbd0.in‚1.759218604441423e+13 disk.sda.in‚ 108.921875 cpu‚25.58041103058756$ memory_percent‚82.98847675323486$ disk./.usageGB‚571.3339538574219 disk.rbd1.out‚ 65.5537109375" proc.memory.rssMB‚ 2874.7265625 disk.sda.out‚ 1447.7890625 disk.sda3.in‚ 108.921875 disk.drbd2165.out‚0 disk.rbd1.in‚ 0.53564453125 disk.rbd0.out‚0 disk.sda1.out‚0( disk./.usagePercent‚38.3427010244889 disk.sda1.in‚0) proc.memory.percent‚17.54593849182129) proc.memory.availableMB‚ 2787.16796875 network.recv‚ 6.242953e+06 disk.dm-1.in‚ 108.921875 disk.dm-1.out‚ 1447.7890625 disk.dm-0.in‚58209.23828125$ disk.dm-0.out‚162480.14013671875) disk.nvme0c0n1.out‚162480.14013671875" disk.nvme0n1.in‚58209.23828125 disk.sda3.out‚ 1447.7890625 disk.drbd2165.in‚0' disk.nvme0n1.out‚162480.14013671875 disk.sda2.in‚0 proc.cpu.threads‚17 network.sent‚ 2.9259285e+07$ disk.nvme0c0n1.in‚58209.23828125 disk.sda2.out‚0RMÿèŸò;‚(:” žøéÒ½¼íö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚284.671' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚56.934200000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0·á …ó;‚(:÷ žøéÒþ¹ˆù disk.drbd2165.out‚0 proc.cpu.threads‚17 disk.sda.out‚1452.625 network.recv‚ 6.247523e+06& disk.rbd0.in‚1.759218604441423e+13$ disk.dm-0.out‚162586.20654296875 disk.sda3.out‚1452.625 disk.sda2.out‚0$ memory_percent‚82.98907279968262) proc.memory.percent‚17.54593849182129$ disk./.usageGB‚570.6664695739746 disk.rbd0.out‚0 disk.sda1.in‚0 disk.sda1.out‚0' disk.nvme0n1.out‚162586.20654296875" proc.memory.rssMB‚ 2874.7265625 cpu‚25.58036440671086) disk./.usagePercent‚38.29790559417014 disk.sda3.in‚ 108.921875! disk.nvme0c0n1.in‚ 58409.53125) disk.nvme0c0n1.out‚162586.20654296875 disk.dm-1.out‚1452.625 disk.dm-0.in‚ 58409.53125 disk.nvme0n1.in‚ 58409.53125 disk.sda2.in‚0( proc.memory.availableMB‚ 2787.0703125 network.sent‚ 2.9322553e+07 disk.sda.in‚ 108.921875 disk.dm-1.in‚ 108.921875 disk.drbd2165.in‚0 disk.rbd1.in‚ 0.53564453125 disk.rbd1.out‚ 65.6787109375P¹Ÿô;‚(:” ­øéÒµ©ëö gpu.0.gpu‚83.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚167.734' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚33.546800000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0aØeõ;‚(:… ­øéÒ鲩ú disk.sda3.in‚ 108.921875 disk.nvme0n1.in‚ 58409.734375 disk.dm-1.in‚ 108.921875% proc.memory.availableMB‚ 2787.3125) proc.memory.percent‚17.54593849182129 network.sent‚ 2.9328866e+07 disk.drbd2165.in‚0 disk.sda.in‚ 108.921875$ memory_percent‚82.98759460449219 proc.cpu.threads‚17 disk.sda1.in‚0 disk.drbd2165.out‚0 disk.rbd1.out‚ 65.6787109375 disk.sda.out‚ 1454.4140625" proc.memory.rssMB‚ 2874.7265625 disk.dm-1.out‚ 1454.4140625$ disk.dm-0.out‚162615.56787109375* disk./.usagePercent‚38.298196418729034' disk.nvme0n1.out‚162615.56787109375 disk.rbd1.in‚ 0.53564453125& disk.rbd0.in‚1.759218604441423e+13 disk.rbd0.out‚0 disk.dm-0.in‚ 58409.734375 disk.sda3.out‚ 1454.4140625 cpu‚25.580353587936322 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda2.out‚0 network.recv‚ 6.251008e+06$ disk./.usageGB‚570.6708030700684" disk.nvme0c0n1.in‚ 58409.734375) disk.nvme0c0n1.out‚162615.56787109375‰Iƒžö;‚(:“ ¼øéÒóàî÷ gpu.0.gpu‚90.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚294.335' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚58.86699999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‰#*N÷;‚(:Š ¼øéÒÔÁ›ù disk.sda.out‚ 1454.83203125 disk.dm-1.out‚ 1454.83203125$ disk.dm-0.out‚162623.72802734375 disk.sda1.out‚0 disk.sda3.in‚ 108.921875' disk.nvme0n1.out‚162623.72802734375( proc.memory.availableMB‚ 2787.3359375 disk.sda.in‚ 108.921875 disk.rbd0.out‚0$ memory_percent‚82.98745155334473 disk.dm-1.in‚ 108.921875 disk.sda1.in‚0) disk.nvme0c0n1.out‚162623.72802734375 disk.sda2.in‚0 disk.dm-0.in‚ 58409.953125" disk.nvme0c0n1.in‚ 58409.953125 disk.rbd1.out‚ 65.7568359375) proc.memory.percent‚17.54593849182129& disk.rbd0.in‚1.759218604441423e+13 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.rbd1.in‚ 0.53564453125 disk.sda3.out‚ 1454.83203125 cpu‚25.580327861653345) disk./.usagePercent‚38.29685826734756 disk.nvme0n1.in‚ 58409.953125 disk.sda2.out‚0" proc.memory.rssMB‚ 2874.7265625 network.sent‚ 2.9332741e+07 network.recv‚ 6.254079e+06$ disk./.usageGB‚570.6508636474609 proc.cpu.threads‚17á@=’ø;‚(:‡ ËøéÒË´„÷ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚276.31' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.262 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0þ à{ ù;‚(:þ ËøéÒé÷áù' disk.nvme0n1.out‚162633.61767578125& disk.rbd0.in‚1.759218604441423e+13$ memory_percent‚82.98745155334473 proc.cpu.threads‚17$ disk./.usageGB‚570.6586761474609 disk.dm-1.out‚ 1455.9140625( proc.memory.availableMB‚ 2787.3359375" proc.memory.rssMB‚ 2874.7265625 network.recv‚ 6.257404e+06 disk.sda2.out‚0 disk.rbd1.in‚ 0.53564453125 disk.sda.in‚ 108.921875 disk.dm-1.in‚ 108.921875 disk.sda1.in‚0 network.sent‚ 2.9339131e+07 disk.dm-0.in‚ 58410.375 disk.sda3.out‚ 1455.9140625 disk.drbd2165.out‚0 disk.sda.out‚ 1455.9140625 disk.sda1.out‚0 disk.sda3.in‚ 108.921875) proc.memory.percent‚17.54593849182129* disk./.usagePercent‚38.297382570777685 disk.nvme0c0n1.in‚ 58410.375 disk.sda2.in‚0 disk.rbd0.out‚0 cpu‚25.58033846077128 disk.nvme0n1.in‚ 58410.375 disk.drbd2165.in‚0) disk.nvme0c0n1.out‚162633.61767578125 disk.rbd1.out‚ 65.7568359375$ disk.dm-0.out‚162633.61767578125˜}4ú;/ _wandbruntime‚25148  _runtime‚251480@:“Éû;à & lr-AdamW/pg1‚9.746835946517806e-05 & lr-AdamW/pg2‚9.746835946517806e-05 trainer/global_step‚15199 $ _timestamp‚1.784314963941902e+09 _runtime‚25148.159008644 _step‚1249‹àAÎü;È & lr-AdamW/pg1‚9.746835946517806e-05 & lr-AdamW/pg2‚9.746835946517806e-05 trainer/global_step‚15199 $ _timestamp‚1.784314963941902e+09 _runtime‚25148.159008644 _step‚1249á &HV4ý;/ _wandbruntime‚25148  _runtime‚25148k¸jÄ(þ;”P VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚18437013595136 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚4367333886861312 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚4367333913034752 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚4381630773117952 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 131607282 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚4379719202213217 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚4381629401554944 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚43045540 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14271055027553 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 _runtime‚25148.162885463 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 129619793 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 129619787 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚4365448172613632 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚4381629401554944 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1179765669318656 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚18436979058688 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚4365448147185664 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 131617255 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚262 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚82033305 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 261227069 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14295514693632 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1198202280286208 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚257M;9º(ÿ;™P ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚4367333886861312 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14296860083200 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14272399627875 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 129619793 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 129619787 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚4365448172613632 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚4381629401554944 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1179765669318656 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚18436979058688 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚4365448147185664 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 131617255 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚262 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚82033305 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 261227069 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14295514693632 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1198202280286208 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚257 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚18437013595136 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚4367333886861312 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚4367333913034752 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚4381630773117952 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 131607282 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚4379719202213217 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚4381629401554944 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚43045540 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14271055027553 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 _runtime‚25148.162885463 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 JDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚250 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1198280958902272 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14297785125888 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1179842787635200 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14273323094233 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚82038828 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚82038573 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚4379999377486041 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚4379998007457639 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 131615760 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 131625747 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 trainer/global_step‚15199 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14271978493799 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚4367611887591424 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚25149.548647081 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 129627999 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 261253746 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 131615760 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚4367611887591424 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚255 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚4365726054391808 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 129627999 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚4381909698890752 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚4381908327320576 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1179843155726336 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1198281361522688 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚82038828 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚82038573 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚4379999377486041 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚4379998007457639 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 131615760 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 131625747 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 trainer/global_step‚15199 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14271978493799 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚4367611887591424 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚25149.548647081 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 129627999 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 261253746 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 131615760 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚4367611887591424 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚255 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚4365726054391808 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 129627999 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚4381909698890752 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚4381908327320576 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1179843155726336 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1198281361522688 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚250 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1198280958902272 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820ã îe’4ƒ</ _wandbruntime‚25149  _runtime‚25149©óõ‹„< 1loss/SRqqt1/detection_loss‚0.4255316554893621 /loss/FBt1/detection_loss‚0.3062696526394575 _runtime‚25149.634644637 _step‚1252 3loss/SRqqt1/assignment_loss‚0.06599965421171804 3loss/SRqqt2/assignment_loss‚0.11187164206737238 4loss/FBt2/assignment_loss‚1.0187780532319112e-05 /loss/FRt2/detection_loss‚1.2524705772681153 0loss/FBt2/detection_loss‚0.10591070389391924 %loss/total_loss‚4.163253594931461 % _timestamp‚1.7843149654176974e+09 0loss/FRt1/assignment_loss‚0.8784187036017849 1loss/FRt2/assignment_loss‚0.10858995073843354 3loss/FBt1/assignment_loss‚0.0003552831088389515 /loss/FRt1/detection_loss‚0.4394199142552422 trainer/global_step‚15199 0loss/SRqqt2/detection_loss‚0.468405669876684 epoch‚37}—ît…<’ 0loss/FRt1/assignment_loss‚0.8784187036017849 1loss/FRt2/assignment_loss‚0.10858995073843354 3loss/FBt1/assignment_loss‚0.0003552831088389515 /loss/FRt1/detection_loss‚0.4394199142552422 trainer/global_step‚15199 0loss/SRqqt2/detection_loss‚0.468405669876684 epoch‚37 1loss/SRqqt1/detection_loss‚0.4255316554893621 /loss/FBt1/detection_loss‚0.3062696526394575 _runtime‚25149.634644637 _step‚1252 3loss/SRqqt1/assignment_loss‚0.06599965421171804 3loss/SRqqt2/assignment_loss‚0.11187164206737238 4loss/FBt2/assignment_loss‚1.0187780532319112e-05 /loss/FRt2/detection_loss‚1.2524705772681153 0loss/FBt2/detection_loss‚0.10591070389391924 %loss/total_loss‚4.163253594931461 % _timestamp‚1.7843149654176974e+09ä ‚†Ù·’†<‚(:‡ ÚøéÒ¾âúö gpu.0.gpu‚1.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚36.0 gpu.0.powerWatts‚109.347' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.8694 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¼§×‡<‚(:‡ ÚøéÒë®øø network.recv‚ 6.260914e+06 disk.rbd1.out‚ 65.8193359375 disk.dm-1.in‚ 108.921875 disk.drbd2165.out‚0 proc.cpu.threads‚17 disk.rbd1.in‚ 0.53564453125 disk.sda3.out‚ 1456.078125! proc.memory.rssMB‚ 2876.046875' disk.nvme0n1.out‚162645.87353515625 disk.sda.in‚ 108.921875& disk.rbd0.in‚1.759218604441423e+13 disk.sda3.in‚ 108.921875! disk.nvme0n1.in‚ 58542.8515625 disk.rbd0.out‚0 disk.dm-0.in‚ 58542.8515625$ disk.dm-0.out‚162645.87353515625) disk./.usagePercent‚38.29799417277698 disk.sda2.out‚0 disk.dm-1.out‚ 1456.078125# disk.nvme0c0n1.in‚ 58542.8515625 network.sent‚ 2.9342888e+07 disk.sda1.in‚0) disk.nvme0c0n1.out‚162645.87353515625 cpu‚25.580679549614565 disk.sda2.in‚0 disk.sda1.out‚0) proc.memory.availableMB‚ 2786.11328125* proc.memory.percent‚17.553997039794922$ disk./.usageGB‚570.6677894592285 disk.sda.out‚ 1456.078125 disk.drbd2165.in‚0# memory_percent‚82.9949140548706›“쓈<‚(:ˆ éøéÒûâòö gpu.0.gpu‚100.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚338.59' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚67.718 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0`|'F‰<‚(: éøéÒ˜¥‹ù* proc.memory.percent‚17.553997039794922 proc.cpu.threads‚17 network.sent‚ 2.9407622e+07 network.recv‚ 6.265007e+06 disk.dm-1.in‚ 108.921875 disk.rbd0.out‚0 disk.dm-0.in‚ 58543.0703125) disk.nvme0c0n1.out‚162800.76904296875! disk.nvme0n1.in‚ 58543.0703125 disk.sda2.out‚0 disk.sda.in‚ 108.921875 disk.sda3.out‚ 1456.63671875 disk.dm-1.out‚ 1456.63671875 disk.rbd1.out‚ 65.8193359375 cpu‚25.58174260843274) disk./.usagePercent‚38.31665814605324 disk.sda.out‚ 1456.63671875! proc.memory.rssMB‚ 2876.046875$ disk.dm-0.out‚162800.76904296875$ memory_percent‚82.99357891082764) proc.memory.availableMB‚ 2786.33203125$ disk./.usageGB‚570.9458961486816& disk.rbd0.in‚1.759218604441423e+13 disk.sda1.in‚0# disk.nvme0c0n1.in‚ 58543.0703125 disk.sda2.in‚0 disk.sda1.out‚0 disk.sda3.in‚ 108.921875 disk.drbd2165.in‚0 disk.drbd2165.out‚0' disk.nvme0n1.out‚162800.76904296875 disk.rbd1.in‚ 0.53564453125”°;™4Š</ _wandbruntime‚25177  _runtime‚25177§g)ÿ%‹<| 8"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.753411591053009 particle/accuracy_1_of_4‚1 particle/accuracy_2_of_5‚1 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.0934707522392273 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7633780241012573 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.021162431687116623 jet/accuracy_5_of_5‚0 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.010854862630367279 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 % _timestamp‚1.7843149929391499e+09 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6108551025390625 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6804110407829285 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6562716960906982 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6431759595870972 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9977789521217346 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.14943817257881165 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.14151127636432648 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9786529541015625 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.1439308226108551 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.14340832829475403 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.1965404748916626 ) Purity/2FRt*SRqqt2FBt/FRt_purity‚0.75 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 /particle/accuracy_4_of_4‚0.9380409717559814 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3506416082382202 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 )jet/accuracy_2_of_3‚0.502091646194458 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.625 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9768028259277344 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4580136239528656 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.7731481790542603 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9732645153999329 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7141912579536438 +particle/specificity‚0.8987252116203308 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7679229974746704 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 jet/accuracy_1_of_5‚1 epoch‚37 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.12253600358963013 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.8657407164573669 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7305685877799988 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.8653846383094788 6 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.987596333026886 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5170029997825623 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9895655512809753 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.46296295523643494 particle/accuracy_4_of_5‚1 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6565514206886292 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 - Purity/0FRt2SRqqt*FBt/FBt_purity‚0.765625 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.7115647196769714 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5950636267662048 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7306252717971802 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.13881297409534454 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.17061159014701843 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 ,jet/accuracy_3_of_4‚0.052480440586805344 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9711654186248779 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.025475740432739258 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.26851850748062134 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9977789521217346 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.11490022391080856 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.5851654410362244 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9906229376792908 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 jet/accuracy_4_of_5‚0 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.12696202099323273 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 *jet/accuracy_1_of_4‚0.9577260613441467 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.15569789707660675 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 /particle/accuracy_3_of_4‚0.9974937438964844 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6086320281028748 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8138091564178467 particle/accuracy_1_of_5‚1 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7349399328231812 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6608846187591553 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6853005290031433 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 + Purity/2FRt0SRqqt1FBt/FRt_purity‚0.5625 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.42500001192092896 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 (particle/accuracy‚0.8339688181877136 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.26923078298568726 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 _runtime‚25177.157017449 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.13767828047275543 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6086320281028748 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.03031207248568535 jet/accuracy_3_of_5‚1 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.21296219527721405 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9923869371414185 6 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.985582709312439 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.4772716164588928 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.11006099730730057 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8888888955116272 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.8991398811340332 8"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.729175329208374 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.021162431687116623 +jet/accuracy_3_of_3‚0.04770654812455177 *jet/accuracy_2_of_4‚0.7077386379241943 particle/accuracy_1_of_3‚1 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.2239583283662796 *loss/val_total_loss‚11.246015924907391 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6227851510047913 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9936847686767578 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9841936826705933 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4517049193382263 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.4634312689304352 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8138091564178467 +jet/accuracy_1_of_1‚0.28594955801963806 *jet/accuracy_1_of_3‚0.9037830829620361 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.5939081907272339 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9907525181770325 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6234862208366394 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9858113527297974 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.2306094467639923 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.2877596318721771 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 - Purity/0FRt2SRqqt1FBt/FBt_purity‚0.765625 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6500488519668579 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6493346691131592 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 particle/accuracy_2_of_2‚1 jet/accuracy_2_of_5‚1 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.02478834241628647 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 /"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.515625 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9847761988639832 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8396823406219482 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9864163994789124 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.7731481790542603 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 particle/accuracy_1_of_1‚1 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 6 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.976565420627594 :"PurA 8ity/0FRt1SRqqt1FBt/event_purity‚0.35604748129844666 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.025113513693213463 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7083333134651184 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.025198545306921005 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7202948331832886 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6923291683197021 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.2222222238779068 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.15625931322574615 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.15047381818294525 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.23297002911567688 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.26851850748062134 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.1397281140089035 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9888888597488403 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.6367793679237366 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9916297197341919 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5232858657836914 ,jet/accuracy_4_of_4‚0.013899499550461769 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6916136741638184 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.14340832829475403 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7317567467689514 particle/accuracy_3_of_5‚1 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.8500000238418579 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7083333134651184 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5070833563804626 particle/accuracy_3_of_3‚1 particle/accuracy_5_of_5‚1 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.2735522985458374 &particle/f_score‚0.607050895690918 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.12818048894405365 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.15692360699176788 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9874140620231628 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6359090805053711 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.02463115192949772 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.17770099639892578 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7142857313156128 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.6969010233879089 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.15701892971992493 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7857142686843872 _step‚1253 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2661578059196472 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.10546200722455978 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.2222222238779068 particle/accuracy_2_of_4‚1 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.7028534412384033 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6335890889167786 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.20898090302944183 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8097565770149231 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9137029051780701 +particle/sensitivity‚0.5972029566764832 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.20839084684848785 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9948258996009827 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.5743953585624695 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 particle/accuracy_1_of_2‚1 6validation_average_jet_accuracy‚0.3289946913719177 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 *jet/accuracy_1_of_2‚0.7274407744407654 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9983765482902527 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.35544082522392273 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.024991337209939957 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.5115740895271301 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6372931003570557 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6715525388717651 8"Purity/*FRt0SRqqt1FBt/event_purity‚0.817646861076355 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 trainer/global_step‚15199 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6681786179542542 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.6343725919723511 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.10499999672174454 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 particle/accuracy_2_of_3‚1 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6912681460380554 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.8500000238418579 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.012992595322430134 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.10950373113155365 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.005593937821686268 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 *jet/accuracy_2_of_2‚0.1261938065290451 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.26923078298568726 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9936507940292358 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6711301803588867ÄNÀ>Œ<•| 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6565514206886292 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 - Purity/0FRt2SRqqt*FBt/FBt_purity‚0.765625 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.7115647196769714 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5950636267662048 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7306252717971802 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.13881297409534454 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.17061159014701843 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 ,jet/accuracy_3_of_4‚0.052480440586805344 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9711654186248779 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.025475740432739258 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.26851850748062134 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9977789521217346 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.11490022391080856 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.5851654410362244 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9906229376792908 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 jet/accuracy_4_of_5‚0 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.12696202099323273 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 *jet/accuracy_1_of_4‚0.9577260613441467 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.15569789707660675 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 /particle/accuracy_3_of_4‚0.9974937438964844 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6086320281028748 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8138091564178467 particle/accuracy_1_of_5‚1 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7349399328231812 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6608846187591553 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6853005290031433 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 + Purity/2FRt0SRqqt1FBt/FRt_purity‚0.5625 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.42500001192092896 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 (particle/accuracy‚0.8339688181877136 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.26923078298568726 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 _runtime‚25177.157017449 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.13767828047275543 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6086320281028748 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.03031207248568535 jet/accuracy_3_of_5‚1 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.21296219527721405 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9923869371414185 6 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.985582709312439 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.4772716164588928 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.11006099730730057 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8888888955116272 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.8991398811340332 8"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.729175329208374 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.021162431687116623 +jet/accuracy_3_of_3‚0.04770654812455177 *jet/accuracy_2_of_4‚0.7077386379241943 particle/accuracy_1_of_3‚1 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.2239583283662796 *loss/val_total_loss‚11.246015924907391 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6227851510047913 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9936847686767578 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9841936826705933 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4517049193382263 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.4634312689304352 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8138091564178467 +jet/accuracy_1_of_1‚0.28594955801963806 *jet/accuracy_1_of_3‚0.9037830829620361 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.5939081907272339 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9907525181770325 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6234862208366394 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9858113527297974 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.2306094467639923 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.2877596318721771 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 - Purity/0FRt2SRqqt1FBt/FBt_purity‚0.765625 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6500488519668579 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6493346691131592 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 particle/accuracy_2_of_2‚1 jet/accuracy_2_of_5‚1 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.02478834241628647 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 /"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.515625 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9847761988639832 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8396823406219482 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9864163994789124 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.7731481790542603 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 particle/accuracy_1_of_1‚1 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 6 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.976565420627594 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.35604748129844666 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.025113513693213463 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7083333134651184 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.025198545306921005 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7202948331832886 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6923291683197021 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.2222222238779068 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.15625931322574615 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.15047381818294525 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.23297002911567688 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.26851850748062134 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.1397281140089035 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9888888597488403 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.6367793679237366 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9916297197341919 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5232858657836914 ,jet/accuracy_4_of_4‚0.013899499550461769 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6916136741638184 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.14340832829475403 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7317567467689514 particle/accuracy_3_of_5‚1 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.8500000238418579 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7083333134651184 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5070833563804626 particle/accuracy_3_of_3‚1 particle/accuracy_5_of_5‚1 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.2735522985458374 &particle/f_score‚0.607050895690918 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.12818048894405365 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.15692360699176788 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9874140620231628 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6359090805053711 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.02463115192949772 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.17770099639892578 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7142857313156128 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.6969010233879089 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.15701892971992493 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7857142686843872 _step‚1253 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2661578059196472 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.10546200722455978 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.2222222238779068 particle/accuracy_2_of_4‚1 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.7028534412384033 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6335890889167786 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.20898090302944183 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8097565770149231 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9137029051780701 +particle/sensitivity‚0.5972029566764832 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.20839084684848785 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9948258996009827 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.5743953585624695 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 particle/accuracy_1_of_2‚1 6validation_average_jet_accuracy‚0.3289946913719177 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 *jet/accuracy_1_of_2‚0.7274407744407654 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9983765482902527 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.35544082522392273 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.024991337209939957 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.5115740895271301 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6372931003570557 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6715525388717651 8"Purity/*FRt0SRqqt1FBt/event_purity‚0.817646861076355 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 trainer/global_step‚15199 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6681786179542542 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.6343725919723511 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.10499999672174454 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 particle/accuracy_2_of_3‚1 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6912681460380554 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.8500000238418579 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.012992595322430134 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.10950373113155365 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.005593937821686268 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 *jet/accuracy_2_of_2‚0.1261938065290451 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.26923078298568726 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9936507940292358 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6711301803588867 8"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.753411591053009 particle/accuracy_1_of_4‚1 particle/accuracy_2_of_5‚1 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.0934707522392273 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7633780241012573 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.021162431687116623 jet/accuracy_5_of_5‚0 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.010854862630367279 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 % _timestamp‚1.7843149929391499e+09 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6108551025390625 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6804110407829285 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6562716960906982 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6431759595870972 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9977789521217346 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.14943817257881165 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.14151127636432648 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9786529541015625 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.1439308226108551 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.14340832829475403 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.1965404748916626 ) Purity/2FRt*SRqqt2FBt/FRt_purity‚0.75 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 /particle/accuracy_4_of_4‚0.9380409717559814 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3506416082382202 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 )jet/accuracy_2_of_3‚0.502091646194458 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.625 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9768028259277344 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4580136239528656 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.7731481790542603 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9732645153999329 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7141912579536438 +particle/specificity‚0.8987252116203308 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7679229974746704 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 jet/accuracy_1_of_5‚1 epoch‚37 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.12253600358963013 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.8657407164573669 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7305685877799988 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.8653846383094788 6 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.987596333026886 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5170029997825623 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9895655512809753 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.46296295523643494 particle/accuracy_4_of_5‚1å =wÌ”<‚(:‰ øøéÒŠãƒ÷ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚257.757' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚51.5514 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¶Å¥GŽ<‚(: øøéÒöÕù disk.sda1.out‚0* proc.memory.percent‚17.127656936645508) disk./.usagePercent‚38.31504248250465! disk.nvme0n1.in‚ 58543.4921875 disk.rbd1.out‚ 65.8974609375 disk.dm-1.in‚ 108.921875' proc.memory.availableMB‚ 2856.484375 proc.cpu.threads‚17 disk.sda2.in‚0 disk.sda1.in‚0) disk.nvme0c0n1.out‚162943.98095703125 disk.sda.in‚ 108.921875 disk.rbd0.out‚0 disk.dm-0.in‚ 58543.4921875" proc.memory.rssMB‚ 2806.1953125 network.recv‚ 6.268266e+06 disk.drbd2165.out‚0$ memory_percent‚82.56540298461914# disk.nvme0c0n1.in‚ 58543.4921875 disk.rbd1.in‚ 0.53564453125 disk.dm-1.out‚ 1456.73828125 network.sent‚ 2.9412644e+07$ disk./.usageGB‚570.9218215942383 disk.drbd2165.in‚0 disk.sda2.out‚0$ disk.dm-0.out‚162943.98095703125 disk.sda3.in‚ 108.921875' disk.nvme0n1.out‚162943.98095703125& disk.rbd0.in‚1.759218604441423e+13 cpu‚25.582287783129434 disk.sda3.out‚ 1456.73828125 disk.sda.out‚ 1456.73828125›Û3”<‚(:‰ ‡ùéÒ¤åÛö gpu.0.gpu‚39.0 gpu.0.memory‚12, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚332.173' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚66.4346 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ô$ÞŸ<‚(: ‡ùéÒ¸Þ©ù cpu‚25.582308985914782& disk.rbd0.in‚1.759218604441423e+13$ disk.dm-0.out‚162964.41455078125 disk.sda1.in‚0 disk.sda3.in‚ 108.921875 disk.sda3.out‚ 1457.06640625$ disk./.usageGB‚570.9261436462402 disk.rbd1.out‚ 65.8974609375 disk.sda2.out‚0 disk.drbd2165.in‚0) proc.memory.availableMB‚ 2856.04296875! disk.nvme0n1.in‚ 58543.6953125 disk.sda2.in‚0 disk.rbd1.in‚ 0.53564453125 disk.rbd0.out‚0# disk.nvme0c0n1.in‚ 58543.6953125 proc.cpu.threads‚17 network.sent‚ 2.9469197e+07* disk./.usagePercent‚38.315332539040945 disk.sda.in‚ 108.921875) disk.nvme0c0n1.out‚162964.41455078125$ memory_percent‚82.56809711456299 disk.dm-0.in‚ 58543.6953125 disk.sda1.out‚0 disk.drbd2165.out‚0' disk.nvme0n1.out‚162964.41455078125 disk.sda.out‚ 1457.06640625" proc.memory.rssMB‚ 2806.1953125 network.recv‚ 6.27274e+06 disk.dm-1.in‚ 108.921875 disk.dm-1.out‚ 1457.06640625* proc.memory.percent‚17.1276569366455080Kz”‘<‚(:‰ –ùéÒ åùö gpu.0.gpu‚90.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚267.421' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.4842 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ñ h ’<‚(:‡ –ùéÒà÷´ù disk.drbd2165.out‚0# disk.nvme0c0n1.in‚ 58640.3203125 disk.dm-1.out‚ 1457.2265625$ memory_percent‚82.56809711456299 network.sent‚ 2.9475625e+07( disk./.usagePercent‚38.3157913045423 disk.sda1.in‚0 proc.cpu.threads‚17$ disk./.usageGB‚570.9329795837402( disk.nvme0c0n1.out‚162972.5791015625 network.recv‚ 6.276412e+06 disk.sda.in‚ 108.921875# disk.dm-0.out‚162972.5791015625 disk.sda3.out‚ 1457.2265625 disk.rbd1.out‚ 65.9599609375! disk.nvme0n1.in‚ 58640.3203125 disk.dm-1.in‚ 108.921875& disk.rbd0.in‚1.759218604441423e+13 disk.sda2.out‚0 disk.rbd0.out‚0 disk.dm-0.in‚ 58640.3203125 disk.sda1.out‚0) proc.memory.availableMB‚ 2856.04296875 disk.drbd2165.in‚0 disk.sda2.in‚0 disk.rbd1.in‚ 0.53564453125 disk.sda3.in‚ 108.921875" proc.memory.rssMB‚ 2806.1953125* proc.memory.percent‚17.127656936645508& disk.nvme0n1.out‚162972.5791015625 disk.sda.out‚ 1457.2265625 cpu‚25.58228955653272·Ì⣔“<‚(:‰ ¥ùéÒÁ·‹÷ gpu.0.gpu‚58.0 gpu.0.memory‚18, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚39.0 gpu.0.powerWatts‚110.656' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.1312 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¨Â·í”<‚(:Ž ¥ùéÒɆ¤ù" proc.memory.rssMB‚ 2806.1953125) disk./.usagePercent‚38.31626952661627 disk.sda.out‚ 1457.53515625 disk.rbd0.out‚0 disk.sda1.out‚0 disk.drbd2165.in‚0 cpu‚25.582260804527603 disk.rbd1.in‚ 0.53564453125& disk.rbd0.in‚1.759218604441423e+13 disk.dm-0.in‚ 58640.7421875 disk.sda3.in‚ 108.921875# disk.nvme0c0n1.in‚ 58640.7421875* proc.memory.percent‚17.127656936645508 disk.sda2.out‚0 disk.sda.in‚ 108.921875$ disk.dm-0.out‚162986.17236328125 disk.sda1.in‚0 disk.sda3.out‚ 1457.53515625! disk.nvme0n1.in‚ 58640.7421875 network.sent‚ 2.9480643e+07 disk.dm-1.in‚ 108.921875' disk.nvme0n1.out‚162986.17236328125( proc.memory.availableMB‚ 2855.7890625 proc.cpu.threads‚17$ disk./.usageGB‚570.9401054382324 disk.drbd2165.out‚0$ memory_percent‚82.56964683532715 network.recv‚ 6.279354e+06 disk.sda2.in‚0 disk.rbd1.out‚ 65.9599609375 disk.dm-1.out‚ 1457.53515625) disk.nvme0c0n1.out‚162986.17236328125|ƒpRž•<‚(:“ ´ùéқ݃÷ gpu.0.gpu‚99.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚294.095' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚58.81900000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0hÔ÷E–<‚(:‹ ´ùéÒ؈Ïù disk.dm-1.in‚ 108.921875 disk.sda1.in‚0 disk.sda3.out‚ 1457.7109375) disk.nvme0c0n1.out‚162999.01611328125* proc.memory.percent‚17.127656936645508 network.recv‚ 6.282973e+06 disk.sda2.out‚0$ disk.dm-0.out‚162999.01611328125 disk.sda3.in‚ 108.921875 proc.cpu.threads‚17) disk./.usagePercent‚38.31641980303886$ disk./.usageGB‚570.9423446655273& disk.rbd0.in‚1.759218604441423e+13 disk.drbd2165.in‚0 disk.rbd1.in‚ 0.53564453125 disk.rbd1.out‚ 66.1005859375 disk.sda.in‚ 108.921875 disk.dm-0.in‚ 58640.9453125 cpu‚25.582211233881594 disk.rbd0.out‚0 disk.sda1.out‚0 disk.sda.out‚ 1457.7109375 network.sent‚ 2.9485814e+07 disk.sda2.in‚0 disk.drbd2165.out‚0' disk.nvme0n1.out‚162999.01611328125$ memory_percent‚82.56993293762207 disk.dm-1.out‚ 1457.7109375! disk.nvme0n1.in‚ 58640.9453125# disk.nvme0c0n1.in‚ 58640.9453125( proc.memory.availableMB‚ 2855.7421875" proc.memory.rssMB‚ 2806.1953125ØO¥]4—</ _wandbruntime‚25255  _runtime‚25255lʘ<Ä & lr-AdamW/pg1‚9.554737887985728e-05 & lr-AdamW/pg2‚9.554737887985728e-05 trainer/global_step‚15249 % _timestamp‚1.7843150712097688e+09 _runtime‚25255.427013188 _step‚1254 üæŠÏ™<É trainer/global_step‚15249 % _timestamp‚1.7843150712097688e+09 _runtime‚25255.427013188 _step‚1254 & lr-AdamW/pg1‚9.554737887985728e-05 & lr-AdamW/pg2‚9.554737887985728e-05æ XýÀî4š</ _wandbruntime‚25255  _runtime‚25255ˆpaõ ›<“P UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 130162099 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚4399410949600768 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚4385061904801792 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚18504455013888 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1183687733858304 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚39123833 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 TED‚Ú§^$eviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚4399409578037760 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14349018625536 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚4385061930975232 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1202192188872192 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚4397492636327460 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚4397491266299170 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14347673235968 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚18504487453184 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚4385061930975232 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14347673235968 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚4399410949600768 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _step‚1255 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14323121272098 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 132092016 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 132101989 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚238 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 400530432 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14349018625536 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968;ppi(œ<˜P QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 130162099 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚233 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚4383168145027072 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚4399410949600768 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _step‚1255 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14323121272098 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 132092016 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 132101989 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚238 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 400530432 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14349018625536 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 130162099 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚4399410949600768 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚4385061904801792 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚18504455013888 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1183687733858304 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚39123833 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚4399409578037760 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14349018625536 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚4385061930975232 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1202192188872192 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚4397492636327460 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚4397491266299170 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14347673235968 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚18504487453184 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚4385061930975232 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14347673235968 JDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 262280786 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1202271292366336 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 _runtime‚25256.817383512 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 262270793 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚4385339905531904 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1202270891843072 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14325389338778 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14348598271488 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚4399689875373568 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 130170305 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14349943668224 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 132110481 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚4399688503803392 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚18505704009216 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚82345130 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚82344895 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚43218519 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚18505671577088 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚4397770071543592 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 130170299 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚230 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚4385339905531904 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 400523264 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14349943668224 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14348598271488 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚4385339931705344 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚32432128 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14324044738344 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚4399689875373568 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚43218749 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 130170305 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚39126376 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚4385339931705344 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 262270793 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚235 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚4397771441571994 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 262280786 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1183765588357120 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 132100494 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 130170299 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0¡YehDŸ<¢N A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 132110481 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 130170299 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 262280786 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1202271292366336 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 _runtime‚25256.817383512 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 262270793 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚4385339905531904 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1202270891843072 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14325389338778 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14348598271488 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚4399689875373568 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 130170305 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14349943668224 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 132110481 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚4399688503803392 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚18505704009216 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚82345130 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚82344895 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚43218519 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚18505671577088 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.currenÑ}ƒbät‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚4397770071543592 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 130170299 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚230 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚4385339905531904 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 400523264 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14349943668224 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14348598271488 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚4385339931705344 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚32432128 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14324044738344 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚4399689875373568 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚43218749 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 130170305 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚39126376 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚4385339931705344 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 262270793 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚235 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚4397771441571994 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 262280786 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1183765588357120 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 132100494 FŸ§<‚(:” áùéÒôþ‚÷ gpu.0.gpu‚98.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚269.838' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚53.967600000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0$É{¯¨<‚(:’ áùéÒÀäù disk.sda1.in‚0 disk.sda2.out‚0 disk.rbd1.in‚ 0.53564453125 cpu‚25.582213916928005) disk./.usagePercent‚38.31764275102993 disk.sda.in‚ 108.921875 disk.dm-1.out‚ 1461.10546875& disk.rbd0.in‚1.759218604441423e+13$ disk.dm-0.out‚163037.57177734375) disk.nvme0c0n1.out‚163037.57177734375 disk.drbd2165.in‚0' disk.nvme0n1.out‚163037.57177734375$ memory_percent‚82.57372379302979 network.recv‚ 6.294071e+06$ disk./.usageGB‚570.9605674743652 disk.sda3.out‚ 1461.10546875" disk.nvme0n1.in‚58750.08984375) proc.memory.availableMB‚ 2855.12109375 proc.cpu.threads‚17 disk.dm-0.in‚58750.08984375 disk.sda1.out‚0 disk.sda3.in‚ 108.921875$ disk.nvme0c0n1.in‚58750.08984375 disk.sda2.in‚0" proc.memory.rssMB‚ 2806.1953125 disk.sda.out‚ 1461.10546875 disk.rbd0.out‚0 disk.drbd2165.out‚0 disk.rbd1.out‚ 66.1630859375* proc.memory.percent‚17.127656936645508 network.sent‚ 2.9559182e+07 disk.dm-1.in‚ 108.921875Þfòž©<‚(:“ ðùéÒ–­öö gpu.0.gpu‚86.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚40.0 gpu.0.powerWatts‚286.799' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚57.35979999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0uˆª<‚(:‘ ðùéÒ®ˆ–ù" disk.nvme0n1.in‚58750.30859375 disk.sda2.in‚0 disk.rbd1.in‚ 0.53564453125 disk.dm-1.out‚ 1461.55859375& disk.rbd0.in‚1.759218604441423e+13$ memory_percent‚82.57222175598145 network.sent‚ 2.9563309e+07 network.recv‚ 6.296642e+06 disk.drbd2165.in‚0 disk.drbd2165.out‚0$ disk.nvme0c0n1.in‚58750.30859375 disk.sda.in‚ 108.921875$ disk.dm-0.out‚163045.60205078125 disk.sda1.in‚0$ disk./.usageGB‚570.9678153991699 disk.sda3.out‚ 1461.55859375) disk.nvme0c0n1.out‚163045.60205078125 disk.sda2.out‚0 disk.sda1.out‚0( proc.memory.availableMB‚ 2855.3671875 cpu‚25.58215559056095 disk.sda3.in‚ 108.921875 disk.sda.out‚ 1461.55859375 disk.dm-1.in‚ 108.921875 proc.cpu.threads‚17* disk./.usagePercent‚38.318129165344985 disk.rbd1.out‚ 66.2412109375 disk.rbd0.out‚0 disk.dm-0.in‚58750.30859375' disk.nvme0n1.out‚163045.60205078125" proc.memory.rssMB‚ 2806.1953125* proc.memory.percent‚17.127656936645508ÕÑé“«<‚(:ˆ ÿùéÒ¿‰µù gpu.0.gpu‚100.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚329.76' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚65.952 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0†*¬<‚(:’ ÿùéÒ¡™ñú disk.sda.out‚ 1461.83203125$ memory_percent‚82.57224559783936* proc.memory.percent‚17.127656936645508 network.recv‚ 6.299772e+06) disk./.usagePercent‚38.31858844286141$ disk./.usageGB‚570.9746589660645 disk.sda2.out‚0" disk.nvme0n1.in‚58750.73046875 disk.sda.in‚ 108.921875 disk.sda1.out‚0& disk.rbd0.in‚1.759218604441423e+13' disk.nvme0n1.out‚163064.66943359375) proc.memory.availableMB‚ 2855.36328125" proc.memory.rssMB‚ 2806.1953125 cpu‚25.582172396660393 disk.rbd0.out‚0 disk.sda3.in‚ 108.921875 disk.sda3.out‚ 1461.83203125$ disk.nvme0c0n1.in‚58750.73046875 disk.sda2.in‚0 disk.rbd1.in‚ 0.53564453125 proc.cpu.threads‚17$ disk.dm-0.out‚163064.66943359375 disk.rbd1.out‚ 66.2412109375 network.sent‚ 2.9568351e+07 disk.dm-1.in‚ 108.921875 disk.dm-1.out‚ 1461.83203125 disk.dm-0.in‚58750.73046875 disk.sda1.in‚0 disk.drbd2165.in‚0 disk.drbd2165.out‚0) disk.nvme0c0n1.out‚163064.66943359375ê1Kž­<‚(:“ ŽúéÒ–¹º÷ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚284.135' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚56.82699999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0JÍ@®<‚(: ŽúéÒƒ„Èú disk.dm-1.out‚ 1463.29296875 disk.sda1.out‚0) disk.nvme0c0n1.out‚163077.40673828125' disk.nvme0n1.out‚163077.40673828125 disk.sda2.in‚0 disk.rbd1.in‚ 0.53564453125 disk.sda.in‚ 108.921875 cpu‚25.582134349510692$ disk./.usageGB‚570.9790954589844 disk.rbd0.out‚0 disk.drbd2165.out‚0 network.recv‚ 6.303219e+06 disk.dm-0.in‚58796.99609375$ disk.nvme0c0n1.in‚58796.99609375$ disk.dm-0.out‚163077.40673828125 disk.dm-1.in‚ 108.921875& proc.memory.availableMB‚ 2855.53125" proc.memory.rssMB‚ 2806.1953125) disk./.usagePercent‚38.31888617962373& disk.rbd0.in‚1.759218604441423e+13 disk.sda1.in‚0" disk.nvme0n1.in‚58796.99609375 disk.sda2.out‚0 proc.cpu.threads‚17 network.sent‚ 2.9572115e+07 disk.rbd1.out‚ 66.3037109375 disk.sda3.out‚ 1463.29296875 disk.drbd2165.in‚0 disk.sda.out‚ 1463.29296875$ memory_percent‚82.57122039794922* proc.memory.percent‚17.127656936645508 disk.sda3.in‚ 108.921875í³á:4¯</ _wandbruntime‚25335  _runtime‚25335¾©cXÊ°<Ä & lr-AdamW/pg2‚9.364247953155597e-05 trainer/global_step‚15299 % _timestamp‚1.7843151512938817e+09 _runtime‚25335.511109094 _step‚1258 & lr-AdamW/pg1‚9.364247953155597e-05í×—zϱ<É & lr-AdamW/pg1‚9.364247953155597e-05 & lr-AdamW/pg2‚9.364247953155597e-05 trainer/global_step‚15299 % _timestamp‚1.7843151512938817e+09 _runtime‚25335.511109094 _step‚1258ê –pœ4²</ _wandbruntime‚25335  _runtime‚25335=/(³<”P JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚4413355930473984 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14393989166592 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14369358385206 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 130572399 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚43351535 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚39250983 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14393989166592 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚39250988 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚4398961941307392 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 _step‚1259 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 130572393 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 130572393 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚4411431592321078 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 263090161 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 _runtime‚25335.512397827 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 132517768 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚4411432962349368 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚18564670339584 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚43351282 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 132527741 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚4398961941307392 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚82602523 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚253 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚4413357302036992 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚4413357302036992ö¸âOÔ´<™P fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1187562422333440 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 % _timestamp‚1.7843151512950163e+09 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 263090161 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 _runtime‚25335.512397827 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 132517768 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚4411432962349368 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚18564670339584 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚43351282 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 132527741 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚4398961941307392 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚82602523 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚253 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 YJDeviceSt§ƒ‡KatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚4413357302036992 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚4413357302036992 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚4413355930473984 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14393989166592 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14369358385206 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 130572399 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚43351535 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚39250983 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14393989166592 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚39250988 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚4398961941307392 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 _step‚1259 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 130572393 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 130572393 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚4411431592321078 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 JDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚39253531 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 263106845 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚4397340115714048 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚4413636227809792 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14394914202112 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 132536233 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚4397340141142016 trainer/global_step‚15299 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚4413634856239616 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 263116838 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚4411711767593902 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚82607268 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 _step‚1260 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 263106845 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚43353990 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14394914202112 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 130580605 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚248 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 132526246 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1187639908741120 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚4399239968210944 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14370281851452 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚4413636227809792 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚4399239968210944 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14396259598848 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1206205370586624 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚4399239942037504 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚4399239942037504 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 132526246 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14396259598848 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚43353742 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14371626451886 _runtime‚25336.901614293 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 130580605 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 130580599 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1187639540649984 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚253 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚39253531 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 263106845 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚4397340115714048 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚4413636227809792 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14394914202112 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 132536233 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚4397340141142016 trainer/global_step‚15299 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚4413634856239616 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 263116838 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚4411711767593902 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚82607268 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 _step‚1260 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 263106845 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚43353990 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14394914202112 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 130580605 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚248 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 132526246 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1187639908741120 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚4399239968210944 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14370281851452 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚4413636227809792 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚4399239968210944 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14396259598848 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1206205370586624 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚4399239942037504 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚4399239942037504 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 132526246 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14396259598848 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚43353742 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14371626451886 _runtime‚25336.901614293 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 130580605 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 130580599 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1187639540649984 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚253 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 EDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚4425371918342986 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 130982693 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1191436742717440 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 132943520 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚4427302282910208 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚4412862003986432 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14440305097216 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1210060982548992 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 130982699 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚18624239831552 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚4427303654473216 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175ÉéSÅ(Ê<™P O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 132943520 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚4425373288371276 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 130982699 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚43486641 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1210060577824256 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36633600 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚280 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚39378138 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚18624203197952 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚4425371918342986 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 130982693 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1191436742717440 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 132943520 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚4427302282910208 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚4412862003986432 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14440305097216 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1210060982548992 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 130982699 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚18624239831552 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚4427303654473216 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175ï A0š½4Ë</ _wandbruntime‚25417  _runtime‚25417I"ã²$'Ì<žN A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1210139247215104 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 _step‚1264 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14417863564994 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 263952890 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 263952890 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 263942897 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 130990905 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 _runtime‚25417.283591136 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚4413140004716544 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚39380676 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚4425652093615810 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 130990899 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14442575529472 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1191513861033984 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14442575529472 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14441230132736 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 263942897 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 130990899 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚4413139978543104 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚4411234230050816 trainer/global_step‚15349 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 132961985 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚4413140004716544 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚266 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚82869745 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚4427581208675840 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 132951998 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚4413139978543104 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 132951998 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404717568 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚4413139978543104 LDDeviceStq‡iø!atsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚4411234230050816 trainer/global_step‚15349 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 132961985 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚4413140004716544 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚266 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚82869745 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚4427581208675840 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 132951998 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚4413139978543104 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 132951998 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404717568 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14417863564994 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 263952890 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 263952890 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 263942897 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 130990905 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 _runtime‚25417.283591136 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚4413140004716544 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚39380676 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚4425652093615810 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 130990899 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14442575529472 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1191513861033984 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0ð ½é4Î</ _wandbruntime‚25417  _runtime‚25417eìQÏ<ˆ 3loss/FBt1/assignment_loss‚5.219416903483958e-07 1loss/SRqqt1/detection_loss‚0.4813223362718395 /loss/FBt1/detection_loss‚0.3170121472017805 0loss/FRt1/assignment_loss‚0.9326994840915387 2loss/FBt2/assignment_loss‚1.54264457377857e-05 /loss/FRt2/detection_loss‚1.2138379742796004 0loss/FBt2/detection_loss‚0.08497302001595537 %loss/total_loss‚4.292358423060394 _step‚1265 1loss/FRt2/assignment_loss‚0.11986785557226926 /loss/FRt1/detection_loss‚0.4545898860916728 epoch‚38 trainer/global_step‚15349 # _timestamp‚1.78431523315008e+09 3loss/SRqqt1/assignment_loss‚0.09534038797429568 3loss/SRqqt2/assignment_loss‚0.09450860526113382 /loss/SRqqt2/detection_loss‚0.49819077791288 _runtime‚25417.367109705ï9zÐ< _runtime‚25417.367109705 3loss/FBt1/assignment_loss‚5.219416903483958e-07 1loss/SRqqt1/detection_loss‚0.4813223362718395 /loss/FBt1/detection_loss‚0.3170121472017805 0loss/FRt1/assignment_loss‚0.9326994840915387 2loss/FBt2/assignment_loss‚1.54264457377857e-05 /loss/FRt2/detection_loss‚1.2138379742796004 0loss/FBt2/detection_loss‚0.08497302001595537 %loss/total_loss‚4.292358423060394 _step‚1265 1loss/FRt2/assignment_loss‚0.11986785557226926 /loss/FRt1/detection_loss‚0.4545898860916728 epoch‚38 trainer/global_step‚15349 # _timestamp‚1.78431523315008e+09 3loss/SRqqt1/assignment_loss‚0.09534038797429568 3loss/SRqqt2/assignment_loss‚0.09450860526113382 /loss/SRqqt2/detection_loss‚0.49819077791288ñ ßÜqyŸÑ<‚(:” èúéÒð¹÷ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚265.314' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚53.062799999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0”WANÒ<‚(:Ž èúéÒì¾³ú network.sent‚ 2.9660887e+07* disk./.usagePercent‚38.321248105134636" disk.nvme0n1.in‚58999.53515625 disk.rbd1.out‚ 66.5224609375 disk.sda.in‚ 108.921875 disk.sda.out‚ 1467.32421875 disk.dm-1.in‚ 108.921875 disk.sda3.out‚ 1467.32421875) disk.nvme0c0n1.out‚163149.52392578125 disk.rbd0.out‚0$ disk.dm-0.out‚163149.52392578125' disk.nvme0n1.out‚163149.52392578125 disk.sda2.in‚0* proc.memory.percent‚17.127656936645508 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.drbd2165.out‚0$ disk.nvme0c0n1.in‚58999.53515625$ memory_percent‚82.57255554199219 cpu‚25.582030884676623 disk.sda3.in‚ 108.921875% proc.memory.availableMB‚ 2855.3125 disk.dm-1.out‚ 1467.32421875 disk.sda2.out‚0" proc.memory.rssMB‚ 2806.1953125# disk./.usageGB‚571.014289855957& disk.rbd0.in‚1.759218604441423e+13 disk.rbd1.in‚ 0.53564453125 proc.cpu.threads‚17 network.recv‚ 6.324063e+06 disk.dm-0.in‚58999.535156254odçžÓ<‚(:“ ÷úéÒØ‚÷ gpu.0.gpu‚83.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚332.801' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚66.56020000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¨eŒšÔ<‚(:ƒ ÷úéÒ’±õù disk.dm-1.in‚ 108.921875 disk.sda1.out‚0 disk.rbd1.in‚ 0.53564453125 disk.sda1.in‚0 disk.sda2.out‚0* proc.memory.percent‚17.127656936645508 disk.rbd0.out‚0 cpu‚25.582042461501867 disk.dm-1.out‚1467.625$ memory_percent‚82.57143497467041 disk.dm-0.in‚58999.75390625 disk.sda3.out‚1467.625) disk.nvme0c0n1.out‚163160.12841796875 disk.rbd1.out‚ 66.6162109375 disk.sda.in‚ 108.921875 network.recv‚ 6.329133e+06$ disk./.usageGB‚571.0212478637695' disk.nvme0n1.out‚163160.12841796875) proc.memory.availableMB‚ 2855.49609375" proc.memory.rssMB‚ 2806.1953125 proc.cpu.threads‚17 network.sent‚ 2.9725702e+07 disk.drbd2165.in‚0 disk.drbd2165.out‚0" disk.nvme0n1.in‚58999.75390625 disk.sda.out‚1467.625& disk.rbd0.in‚1.759218604441423e+13) disk./.usagePercent‚38.32171506287709$ disk.dm-0.out‚163160.12841796875 disk.sda3.in‚ 108.921875$ disk.nvme0c0n1.in‚58999.75390625 disk.sda2.in‚0ü“êe”Õ<‚(:‰ †ûéÒµÄíö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚268.262' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.6524 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÉÓ1Ö<‚(:’ †ûéÒ‡ŸØù$ disk.dm-0.out‚163173.31201171875 disk.sda3.in‚ 108.921875 disk.drbd2165.out‚0 disk.dm-0.in‚59000.19140625 disk.rbd1.in‚ 0.53564453125 cpu‚25.58200453957845 disk.sda2.in‚0 disk.rbd1.out‚ 66.6787109375 disk.sda.out‚ 1468.10546875" proc.memory.rssMB‚ 2806.1953125* disk./.usagePercent‚38.322159235948995) disk.nvme0c0n1.out‚163173.31201171875 disk.dm-1.in‚ 108.921875 disk.sda.in‚ 108.921875 network.sent‚ 2.9729531e+07 network.recv‚ 6.332787e+06$ disk./.usageGB‚571.0278663635254& disk.rbd0.in‚1.759218604441423e+13 disk.rbd0.out‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-1.out‚ 1468.10546875 disk.drbd2165.in‚0$ disk.nvme0c0n1.in‚59000.19140625" disk.nvme0n1.in‚59000.19140625 disk.sda2.out‚0 proc.cpu.threads‚17 disk.sda3.out‚ 1468.10546875' disk.nvme0n1.out‚163173.31201171875) proc.memory.availableMB‚ 2855.48828125$ memory_percent‚82.57148265838623* proc.memory.percent‚17.127656936645508ž¶®éŸ×<‚(:” •ûéÒŸ´€÷ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚40.0 gpu.0.powerWatts‚255.879' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚51.175799999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0€?b× Ø<‚(:‚ •ûéÒ‡³äù disk.rbd0.out‚0 disk.rbd1.out‚ 66.6787109375 disk.sda1.out‚0 disk.sda.in‚ 108.921875" disk.dm-0.out‚163184.451171875 disk.sda3.in‚ 108.921875' disk.nvme0c0n1.out‚163184.451171875 cpu‚25.58194625082624 disk.dm-1.in‚ 108.921875 disk.sda3.out‚ 1468.41796875 disk.drbd2165.out‚0 disk.nvme0n1.in‚ 59163.53125* proc.memory.percent‚17.127656936645508 proc.cpu.threads‚17 disk.sda1.in‚0 disk.dm-0.in‚ 59163.53125! disk.nvme0c0n1.in‚ 59163.53125% disk.nvme0n1.out‚163184.451171875 disk.sda2.out‚0) proc.memory.availableMB‚ 2855.23828125$ disk./.usageGB‚571.0278282165527 disk.rbd1.in‚ 0.53564453125$ memory_percent‚82.57300853729248" proc.memory.rssMB‚ 2806.1953125 network.sent‚ 2.9734586e+07& disk.rbd0.in‚1.759218604441423e+13 disk.drbd2165.in‚0 disk.sda2.in‚0 network.recv‚ 6.335894e+06) disk./.usagePercent‚38.32215667587365 disk.sda.out‚ 1468.41796875 disk.dm-1.out‚ 1468.41796875”zÙ<‚(:’ ¤ûéÒü¶ï÷ gpu.0.gpu‚99.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚331.35' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚66.27000000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 àiÙÚ<‚(:õ ¤ûéÒïÂù disk.nvme0n1.in‚59163.75& disk.rbd0.in‚1.759218604441423e+13) proc.memory.availableMB‚ 2856.11328125 disk.rbd1.out‚ 66.7412109375 disk.rbd0.out‚0 disk.nvme0c0n1.in‚59163.75 disk.sda2.out‚0 disk.sda.out‚ 1468.703125# disk.dm-0.out‚163193.3095703125" proc.memory.rssMB‚ 2806.1953125 cpu‚25.581957350178826 proc.cpu.threads‚17 disk.sda3.in‚ 108.921875 disk.sda2.in‚0 disk.sda.in‚ 108.921875 disk.dm-1.in‚ 108.921875$ disk./.usageGB‚571.0356597900391 disk.drbd2165.in‚0( disk.nvme0c0n1.out‚163193.3095703125& disk.nvme0n1.out‚163193.3095703125* proc.memory.percent‚17.127656936645508 disk.dm-0.in‚59163.75# memory_percent‚82.5676679611206 network.sent‚ 2.973962e+07) disk./.usagePercent‚38.32268225934145 disk.rbd1.in‚ 0.53564453125 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda3.out‚ 1468.703125 disk.dm-1.out‚ 1468.703125 network.recv‚ 6.339556e+06 disk.drbd2165.out‚0ãý>Ó4Û</ _wandbruntime‚25495  _runtime‚25495ÃOã ÊÜ<Ä _step‚1266 & lr-AdamW/pg1‚8.988141289778826e-05 & lr-AdamW/pg2‚8.988141289778826e-05 trainer/global_step‚15399 % _timestamp‚1.7843153115117743e+09 _runtime‚25495.729029831^ŠÏÝ<É & lr-AdamW/pg2‚8.988141289778826e-05 trainer/global_step‚15399 % _timestamp‚1.7843153115117743e+09 _runtime‚25495.729029831 _step‚1266 & lr-AdamW/pg1‚8.988141289778826e-05ò ¾Ì‡4Þ</ _wandbruntime‚25495  _runtime‚25495f-O(ß<“P ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚4426762014318592 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚4424850437181440 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚4426762040492032 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 264772244 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 264762265 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14461832611422 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚4441248635346432 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚4441250006909440 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 133379245 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14487966417408 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14463177211744 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14487966417408 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚43621122 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 131392993 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14486621027840 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 131392993 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 131392999 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 $ _timestamp‚1.784315311512752e+09 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 131392999 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 133369272 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1213995022024192 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚18683924386304 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚39505283 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚4439313614393184 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚4439312244364894 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚83126667 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚83126405 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1213994619396608 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 133379245 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1195311063101440 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚4441250006909440 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚4441248635346432 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚39505288 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚43621379 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚4424850411753472 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14486621027840 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚18683958922752 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚4426762040492032 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 _runtime‚25495.730090604 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192ÿA¯à<˜P GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 133379245 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1195311063101440 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚4441250006909440 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚4441248635346432 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚39505288 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚43621379 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚4424850411753472 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14486621027840 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚18683958922752 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚4426762040492032 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 _runtime‚25495.730090604 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚4426762014318592 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚4424850437181440 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚4426762040492032 N@DeviceStatsMonitor.on_train_batch_start/allocation.alÕ†ºŽol.allocated‚ 264772244 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 264762265 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14461832611422 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚4441248635346432 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚4441250006909440 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 133379245 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14487966417408 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14463177211744 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14487966417408 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚43621122 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 131392993 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14486621027840 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 131392993 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 131392999 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 $ _timestamp‚1.784315311512752e+09 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 131392999 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 133369272 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1213995022024192 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚18683924386304 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚39505283 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚4439313614393184 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚4439312244364894 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚83126667 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚83126405 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1213994619396608 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175ó ý¦O4á</ _wandbruntime‚25497  _runtime‚25497†$'â<žN `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1195388181417984 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 131401199 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14462756077668 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚18685113424384 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚4441527561112064 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14488891460096 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 264788942 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 133377750 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚4441528932682240 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 131401205 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚4427040015048704 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 131401199 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚4427040041222144 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚83131676 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1214073697462784 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 264778949 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚4427040041222144 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚83131924 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚18685147953664 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚4425128318959616 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 133387737 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚39507831 % _timestamp‚1.7843153129006994e+09¢¡D)'ã<£N TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚4441528932682240 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 131401205 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚4427040015048704 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 131401199 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚4427040041222144 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚83131676 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1214073697462784 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 264778949 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚4427040041222144 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚83131924 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚18685147953664 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚4425128318959616 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 133387737 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚39507831 % _timestamp‚1.7843153129006994e+09 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1195388181417984 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 131401199 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14462756077668 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚18685113424384 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚4441527561112064 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14488891460096 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 264788942 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 133377750 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736ô XÌ4ä</ _wandbruntime‚25497  _runtime‚25497cÐøiå< /loss/FRt2/detection_loss‚1.1927142583152623 _step‚1269 0loss/FRt1/assignment_loss‚0.8729533075493098 0loss/FRt2/assignment_loss‚0.0863201574544714 2loss/SRqqt1/assignment_loss‚0.0776259761406541 3loss/FBt2/assignment_loss‚0.0005533376467288619 1loss/SRqqt2/detection_loss‚0.4421323741684497 0loss/FBt1/detection_loss‚0.31341927121586477 trainer/global_step‚15399 % _timestamp‚1.7843153129874067e+09 4loss/FBt1/assignment_loss‚3.7391595539632435e-05 2loss/SRqqt1/detection_loss‚0.49347406182552683 /loss/FBt2/detection_loss‚0.1770065167713691 _runtime‚25497.20434098 %loss/total_loss‚4.187198414998512 epoch‚38 3loss/SRqqt2/assignment_loss‚0.11657545969998177 0loss/FRt1/detection_loss‚0.41438630261535514ˆÈ.»æ<’ 0loss/FRt2/assignment_loss‚0.0863201574544714 2loss/SRqqt1/assignment_loss‚0.0776259761406541 3loss/FBt2/assignment_loss‚0.0005533376467288619 1loss/SRqqt2/detection_loss‚0.4421323741684497 0loss/FBt1/detection_loss‚0.31341927121586477 trainer/global_step‚15399 % _timestamp‚1.7843153129874067e+09 4loss/FBt1/assignment_loss‚3.7391595539632435e-05 2loss/SRqqt1/detection_loss‚0.49347406182552683 /loss/FBt2/detection_loss‚0.1770065167713691 _runtime‚25497.20434098 %loss/total_loss‚4.187198414998512 epoch‚38 3loss/SRqqt2/assignment_loss‚0.11657545969998177 0loss/FRt1/detection_loss‚0.41438630261535514 /loss/FRt2/detection_loss‚1.1927142583152623 _step‚1269 0loss/FRt1/assignment_loss‚0.8729533075493098õ Y´­ÚŸç<‚(:” ³ûéÒƒ­Üö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚274.182' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚54.836400000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚06´œ\è<‚(:‰ ³ûéÒÐÙÛø# disk.dm-0.out‚163208.9658203125* disk./.usagePercent‚38.323142304880484$ disk./.usageGB‚571.0425148010254 disk.dm-1.in‚ 108.921875 disk.drbd2165.out‚0 proc.cpu.threads‚17 disk.sda.out‚ 1468.94921875 disk.dm-0.in‚ 59164.171875 disk.sda3.out‚ 1468.94921875" disk.nvme0c0n1.in‚ 59164.171875) proc.memory.availableMB‚ 2855.84765625" proc.memory.rssMB‚ 2806.1953125 disk.drbd2165.in‚0( disk.nvme0c0n1.out‚163208.9658203125 disk.sda2.out‚0 disk.rbd1.in‚ 0.53564453125 disk.rbd1.out‚ 66.7412109375* proc.memory.percent‚17.127656936645508 network.sent‚ 2.9745984e+07 network.recv‚ 6.342981e+06 disk.sda3.in‚ 108.921875& disk.nvme0n1.out‚163208.9658203125 disk.dm-1.out‚ 1468.94921875 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.in‚0# memory_percent‚82.5692892074585 cpu‚25.581891301156443& disk.rbd0.in‚1.759218604441423e+13 disk.nvme0n1.in‚ 59164.171875 disk.sda.in‚ 108.921875 disk.rbd0.out‚0ŠëíŸé<‚(:” ÂûéÒÀ‰À÷ gpu.0.gpu‚32.0 gpu.0.memory‚10, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚37.0 gpu.0.powerWatts‚109.997' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.999399999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¦i6Øê<‚(:ƒ ÂûéÒ¤©–ú) proc.memory.availableMB‚ 2855.91796875" proc.memory.rssMB‚ 2806.1953125 network.recv‚ 6.347603e+06 disk.rbd0.out‚0 disk.sda3.out‚ 1469.12890625 cpu‚25.581849943286457 proc.cpu.threads‚17 network.sent‚ 2.9809299e+07& disk.rbd0.in‚1.759218604441423e+13 disk.sda3.in‚ 108.921875 disk.nvme0n1.in‚ 59164.375) disk./.usagePercent‚38.32344669783869 disk.sda.out‚ 1469.12890625 disk.sda1.in‚0 disk.sda2.out‚0 disk.sda.in‚ 108.921875 disk.dm-0.in‚ 59164.375$ disk.dm-0.out‚163217.39208984375 disk.drbd2165.out‚0 disk.rbd1.out‚ 66.8037109375 disk.sda1.out‚0' disk.nvme0n1.out‚163217.39208984375$ memory_percent‚82.56886005401611* proc.memory.percent‚17.127656936645508$ disk./.usageGB‚571.0470504760742 disk.rbd1.in‚ 0.53564453125 disk.dm-1.out‚ 1469.12890625 disk.nvme0c0n1.in‚ 59164.375) disk.nvme0c0n1.out‚163217.39208984375 disk.sda2.in‚0 disk.dm-1.in‚ 108.921875 disk.drbd2165.in‚0Ž™Ð‚žë<‚(:“ ÑûéÒ‡ñ÷ gpu.0.gpu‚99.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚285.243' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚57.04859999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ur¬€ì<‚(: ÑûéÒô‹Èù network.recv‚ 6.350733e+06 disk.sda2.in‚0 disk.rbd1.in‚ 0.53564453125 disk.rbd1.out‚ 66.8037109375 disk.dm-1.out‚ 1469.4140625 disk.sda3.out‚ 1469.4140625$ disk./.usageGB‚571.0540771484375 disk.sda2.out‚0 disk.sda.out‚ 1469.4140625 disk.dm-1.in‚ 108.921875( proc.memory.availableMB‚ 2856.1640625" proc.memory.rssMB‚ 2806.1953125 proc.cpu.threads‚17 network.sent‚ 2.9813027e+07& disk.rbd0.in‚1.759218604441423e+13 disk.sda1.in‚0 disk.drbd2165.out‚0* proc.memory.percent‚17.127656936645508$ disk.nvme0c0n1.in‚59336.67578125 cpu‚25.58185206927447' disk.nvme0n1.out‚163229.19970703125 disk.rbd0.out‚0) disk./.usagePercent‚38.32391826371676) disk.nvme0c0n1.out‚163229.19970703125 disk.sda.in‚ 108.921875 disk.dm-0.in‚59336.67578125$ disk.dm-0.out‚163229.19970703125 disk.sda3.in‚ 108.921875 disk.drbd2165.in‚0$ memory_percent‚82.56735801696777" disk.nvme0n1.in‚59336.67578125 disk.sda1.out‚0¸é|”í<‚(:‰ àûéÒÄçëö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚293.271' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚58.6542 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Vx3¤žî<‚(:‹ àûéÒíóÊù$ disk.dm-0.out‚163252.42138671875" proc.memory.rssMB‚ 2806.1953125 network.sent‚ 2.981805e+07 network.recv‚ 6.354018e½ x+06 disk.rbd1.out‚ 66.8818359375& disk.rbd0.in‚1.759218604441423e+13 disk.sda1.out‚0 cpu‚25.581813780241287) disk.nvme0c0n1.out‚163252.42138671875' disk.nvme0n1.out‚163252.42138671875 disk.rbd1.in‚ 0.53564453125 disk.dm-0.in‚ 59337.265625 proc.cpu.threads‚17 disk.sda2.out‚0 disk.dm-1.out‚ 1471.57421875$ memory_percent‚82.56738185882568) disk./.usagePercent‚38.32466426967154 disk.drbd2165.out‚0 disk.nvme0n1.in‚ 59337.265625 disk.sda2.in‚0 disk.rbd0.out‚0 disk.dm-1.in‚ 108.921875 disk.sda1.in‚0 disk.sda3.in‚ 108.921875 disk.sda.out‚ 1471.57421875) proc.memory.availableMB‚ 2856.16015625* proc.memory.percent‚17.127656936645508 disk.drbd2165.in‚0" disk.nvme0c0n1.in‚ 59337.265625 disk.sda3.out‚ 1471.57421875$ disk./.usageGB‚571.0651931762695 disk.sda.in‚ 108.921875, –”ï<‚(:‰ ïûéÒñ’çö gpu.0.gpu‚99.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚346.406' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚69.2812 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚07‘¤_ ð<‚(:€ ïûéÒæ„–ù disk.sda.out‚ 1475.078125 disk.dm-1.out‚ 1475.078125 disk.rbd0.out‚0 disk.sda3.out‚ 1475.078125 disk.sda2.in‚0$ memory_percent‚82.56902694702148 disk.sda1.out‚0* proc.memory.percent‚17.127656936645508 proc.cpu.threads‚17( disk./.usagePercent‚38.3259448193578 disk.dm-1.in‚ 108.921875 disk.sda1.in‚0! disk.nvme0c0n1.in‚ 59337.46875' disk.nvme0n1.out‚163272.03369140625& disk.rbd0.in‚1.759218604441423e+13 disk.sda2.out‚0 disk.rbd1.out‚ 66.8818359375$ disk./.usageGB‚571.0842742919922 disk.dm-0.in‚ 59337.46875$ disk.dm-0.out‚163272.03369140625) disk.nvme0c0n1.out‚163272.03369140625' proc.memory.availableMB‚ 2855.890625 cpu‚25.581825311728505 disk.sda3.in‚ 108.921875 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.sda.in‚ 108.921875 network.sent‚ 2.9823211e+07 network.recv‚ 6.357109e+06 disk.nvme0n1.in‚ 59337.46875 disk.rbd1.in‚ 0.53564453125" proc.memory.rssMB‚ 2806.1953125úâHÊ”ñ<‚(:‰ þûéÒ›º‚÷ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚259.349' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚51.8698 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0yÔõò<‚(:„ þûéÒÑæ¶ù disk.sda.out‚ 1476.96484375 disk.sda3.in‚ 108.921875* proc.memory.percent‚17.127656936645508 disk.sda2.out‚0 network.recv‚ 6.360529e+06) disk./.usagePercent‚38.32542691611604 disk.sda.in‚ 108.921875 disk.dm-1.out‚ 1476.96484375 disk.drbd2165.in‚0 disk.drbd2165.out‚0 proc.cpu.threads‚17) disk.nvme0c0n1.out‚163299.29736328125 disk.rbd1.out‚ 67.0068359375& disk.rbd0.in‚1.759218604441423e+13 disk.sda2.in‚0 disk.dm-1.in‚ 108.921875 disk.rbd0.out‚0 disk.nvme0n1.in‚ 59337.6875# memory_percent‚82.5690507888794 disk.rbd1.in‚ 0.53564453125$ disk./.usageGB‚571.0765571594238$ disk.dm-0.out‚163299.29736328125 cpu‚25.58177682540779 network.sent‚ 2.9829688e+07 disk.sda1.out‚0 disk.sda3.out‚ 1476.96484375 disk.nvme0c0n1.in‚ 59337.6875) proc.memory.availableMB‚ 2855.88671875 disk.sda1.in‚0 disk.dm-0.in‚ 59337.6875' disk.nvme0n1.out‚163299.29736328125" proc.memory.rssMB‚ 2806.1953125%¬Ð4ó</ _wandbruntime‚25575  _runtime‚25575rÆô<À _runtime‚25575.42902663 _step‚1270 % lr-AdamW/pg1‚8.80254871786035e-05 % lr-AdamW/pg2‚8.80254871786035e-05 trainer/global_step‚15449 $ _timestamp‚1.784315391208734e+09O¸½ÖËõ<Å _runtime‚25575.42902663 _step‚1270 % lr-AdamW/pg1‚8.80254871786035e-05 % lr-AdamW/pg2‚8.80254871786035e-05 trainer/global_step‚15449 $ _timestamp‚1.784315391208734e+09ö x3„4ö</ _wandbruntime‚25575  _runtime‚25575’;H1(÷<”P M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 133795024 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚43755685 _step‚1271 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14508069724530 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚4455196359345664 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 131803299 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚18743430054400 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚18743395517952 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 131803299 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 133804997 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 _runtime‚25575.429960513 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚4455194987782656 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 265608296 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚4440662050824192 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 131803293 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14534282348032 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚83388123 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14532936958464 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚4440662076997632 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚4440662076997632 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 133804997 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14532936958464 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 133795024 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1217928813539840 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚4455194987782656 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚4440662050824192 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14534282348032 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1199185015394304 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14509414324852 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚4455194987782656 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚4440662050824192 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14534282348032 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1199185015394304 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14509414324852 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚4455194987782656 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 265608296 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚4440662050824192 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 131803293 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14534282348032 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚83388123 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14532936958464 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚4440662076997632 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚4440662076997632 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 133804997 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14532936958464 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 133795024 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1217928813539840 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚83393103 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚4455475285118464 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚18744587030016 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 131811505 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚4440940077727744 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚4453532745659626 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1218007088832000 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚4440940077727744 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14535207390720 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1199262501801984 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚39634976 _step‚1272 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 265624994 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14508993190776 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 265615001 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 131811499 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1199262869893120 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14533861993984 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚43758376 RCDeviceStatsMonitor.on_train_batch¬qÔÕž_end/inactive_split_bytes.all.peak‚ 1341808128 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1218007491452416 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 trainer/global_step‚15449 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 265624994 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚83393357 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 $ _timestamp‚1.784315392599314e+09 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 133813489 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚18744621559296 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14533861993984 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 131811499 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0Èa&('û<¢N UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 131811505 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚254 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚4455475285118464 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14535207390720 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚4455473913548288 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚4455475285118464 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚18744587030016 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 131811505 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚4440940077727744 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚4453532745659626 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1218007088832000 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚4440940077727744 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14535207390720 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1199262501801984 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚39634976 _step‚1272 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 265624994 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14508993190776 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 265615001 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 131811499 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1199262869893120 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14533861993984 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚43758376 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1218007491452416 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 trainer/global_step‚15449 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 265624994 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚83393357 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 $ _timestamp‚1.784315392599314e+09 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 133813489 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚18744621559296 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14533861993984 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 131811499 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚83393103 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 % _timestamp‚1.7843154708199763e+09 _runtime‚25655.037418731 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚4454562087329792 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚43891418 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚43891152 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚271 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1221862411096576 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 134220776 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚4452638589571072 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 132213599 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚4469141340218880 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚4454562087329792 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995[e¸Ž=™P A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 132213599 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚4469141340218880 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚4454562087329792 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 134230749 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1203059335778304 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚4469142711781888 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚83650735 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14580598278656 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 134230749 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚4469142711781888 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 266434369 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14579252889088 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚83651006 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚4452638614999040 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚4469141340218880 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14580598278656 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14554306837638 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 134220776 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14555651437960 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 % _timestamp‚1.7843154708199763e+09 _runtime‚25655.037418731 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚4454562087329792 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚43891418 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚43891152 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚271 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1221862411096576 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 134220776 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚4452638589571072û `Î4=/ _wandbruntime‚25656  _runtime‚25656!€ta$'=žN H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 266461046 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚4454840114233344 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 132221799 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚43894019 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 132221799 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚4454840088059904 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14580177924608 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1221941091654144 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14555230303884 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚4467473071681534 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 132221805 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚252 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚18804303997440 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 trainer/global_step‚15499 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 132221805 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1221941494274560 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14581523321344 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1203137190277120 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7843154722063384e+09 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14580177924608 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14556574904318 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚39762126 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 134239241 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 266461046 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚4469421637554688 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚4469420265984512 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚4454840088059904 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1203136822185984 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚83655893 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚4452916496777216 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚18804269468160 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚1276[™ê¸)'‘=£N TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚39762131 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 266451053 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7843154722063384e+09 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14580177924608 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14556574904318 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚39762126 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 134239241 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 266461046 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚4469421637554688 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚4469420265984512 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚4454840088059904 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1203136822185984 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚83655893 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚4452916496777216 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚18804269468160 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚1276 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 266461046 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚4454840114233344 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 132221799 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚43894019 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 132221799 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚4454840088059904 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14580177924608 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1221941091654144 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14555230303884 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚4467473071681534 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 132221805 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚252 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚18804303997440 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 trainer/global_step‚15499 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 132221805 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1221941494274560 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14581523321344 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1203137190277120 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚4483087692655104 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚83912322 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚4468462123835392 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14625568819712 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚4483089064218112 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚44025589 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14626914209280 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 132623893 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1206933656162304 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 132623893 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚4466532678479872 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚18862901062144 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚4466532703907840 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 134656501 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14625568819712 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚4468462150008832 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚39886733 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚4483089064218112 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚39886738 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 267270421 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14600543950746 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _runtime‚25734.856025521 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1225796925315584 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚18862866525696 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚260 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚255 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚4481133222430618 % _timestamp‚1.7843155506357071e+09 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚4483089064218112 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚44025589 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14626914209280 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 132623893 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1206933656162304 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 132623893 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚4466532678479872 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚18862901062144 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚4466532703907840 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 134656501 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14625568819712 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚4468462150008832 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚39886733 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚4483089064218112 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚39886738 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 267270421 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14600543950746 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _runtime‚25734.856025521 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1225796925315584 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚18862866525696 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚260 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚255 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚4481133222430618 % _timestamp‚1.7843155506357071e+09 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚4483087692655104 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚83912322 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚4468462123835392 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14625568819712 JDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚4468740150738944 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 132632099 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚4468740150738944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 267297098 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14601467416992 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚39889276 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7843155520247588e+09 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 132632105 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 132632105 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚4483367989990912 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14626493855232 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚83917840 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚243 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚18864093648384 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚4483367989990912 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1225875201689088 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚44028559 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚25736.242190206 _step‚1280 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1225875604309504 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1207011142569984 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚4481413397703442 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 132632099 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 267287105 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚18864059119104 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 134664993 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚4468740124565504 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 trainer/global_step‚15549 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14602812017426 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚4483366618420736 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚4466810585686016 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚83917592›D‡)'§=£N ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14627839251968 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1225875201689088 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚44028559 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚25736.242190206 _step‚1280 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1225875604309504 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1207011142569984 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚4481413397703442 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 132632099 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 267287105 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚18864059119104 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 134664993 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚4468740124565504 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 trainer/global_step‚15549 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14602812017426 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚4483366618420736 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚4466810585686016 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚83917592 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚44028316 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚4481412027675040 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚4468740150738944 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 132632099 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚4468740150738944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 267297098 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14601467416992 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚39889276 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7843155520247588e+09 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 132632105 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 132632105 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚4483367989990912 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14626493855232 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚83917840 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚243 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚18864093648384 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚4483367989990912 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 F w픳=‚(:‰ ßýéÒ·ªòö gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚286.263' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚57.2526 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Iý¾Ç´=‚(:Š ßýéÒЂú disk.dm-1.in‚ 108.921875 disk.dm-0.in‚ 60023.2734375# disk.nvme0c0n1.in‚ 60023.2734375! disk.nvme0n1.in‚ 60023.2734375$ memory_percent‚81.68003559112549 network.recv‚ 6.4143e+06 disk.sda3.out‚ 1508.00390625 disk.drbd2165.out‚0& disk.rbd0.in‚1.759218604441423e+13 disk.sda2.in‚0) proc.memory.availableMB‚ 3001.54296875 proc.cpu.threads‚17" proc.memory.rssMB‚ 2806.1953125) disk./.usagePercent‚38.33718815824657 disk.sda.in‚ 108.921875 disk.dm-1.out‚ 1508.00390625 disk.rbd0.out‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda3.in‚ 108.921875( disk.nvme0c0n1.out‚165207.4501953125 disk.sda.out‚ 1508.00390625 disk.rbd1.out‚ 67.5380859375* proc.memory.percent‚17.127656936645508 cpu‚25.581682486792324 network.sent‚ 3.0077648e+07$ disk./.usageGB‚571.2518081665039 disk.rbd1.in‚ 0.53564453125# disk.dm-0.out‚165207.4501953125 disk.drbd2165.in‚0& disk.nvme0n1.out‚165207.4501953125 disk.sda2.out‚0­DØ4µ=/ _wandbruntime‚25814  _runtime‚258146´mÊʶ=Ä % _timestamp‚1.7843156303807096e+09 _runtime‚25814.597810871 _step‚1282 & lr-AdamW/pg1‚8.255756762894378e-05 & lr-AdamW/pg2‚8.255756762894378e-05 trainer/global_step‚15599G##Ï·=É _step‚1282 & lr-AdamW/pg1‚8.255756762894378e-05 & lr-AdamW/pg2‚8.255756762894378e-05 trainer/global_step‚15599 % _timestamp‚1.7843156303807096e+09 _runtime‚25814.597810871‚ -ö½>4¸=/ _wandbruntime‚25814  _runtime‚25814îûDªÂ"¹=”P VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚4482362186514432 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚18922702959104 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚4482362186514432 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚247 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1229730679505408 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14671884750336 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚40013888 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 133034199 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚4482362160340992 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14673230139904 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚4497034045091328 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚4495073548452526 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 133034193 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚4480426792816640 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14646781063854 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚84173573 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 P@DeviceStatsMonitor.on_train_bÛ®‹Xatch_start/requested_bytes.all.peak‚ 47499471119 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚15599 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1210807976546304 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 % _timestamp‚1.7843156303814108e+09 _runtime‚25814.598687556 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚4497034045091328 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚252 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 135072280 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352éÌ(º=™P MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 268106473 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1210808344637440 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚44159937 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚40013888 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 133034199 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚4482362160340992 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14673230139904 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚4497034045091328 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚4495073548452526 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 133034193 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚4480426792816640 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14646781063854 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚84173573 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚15599 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1210807976546304 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 % _timestamp‚1.7843156303814108e+09 _runtime‚25814.598687556 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚4497034045091328 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚252 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 135072280 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚4482362186514432 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚18922702959104 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚4482362186514432 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚247 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1229730679505408 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14671884750336 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚84178781 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚18923894741504 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚239 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1229809357695488 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚4482640161071104 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 133042399 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 268123157 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚40016431 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1210885462953984 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 _runtime‚25815.983398016 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 133042399 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 _step‚1284 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 133042405 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 135080758 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚4480704649166848 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚4495352353696948 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚4497312970856960 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14674155182592 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14672809785856 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚44162594 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1210885831045120 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚44162355 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚4495353723725350 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14649049130534 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚4482640187244544 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 268133150 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 % _timestamp‚1.7843156317659905e+09 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚4497314342427136 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚18923929270784 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚4497314342427136 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚4482640187244544 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 135090745 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995²ÏÞ³a%¿=£N QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 135080758 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚4480704649166848 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚4495352353696948 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚4497312970856960 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14674155182592 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14672809785856 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚44162594 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1210885831045120 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚44162355 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚4495353723725350 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14649049130534 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚4482640187244544 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 268133150 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 % _timestamp‚1.7843156317659905e+09 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚4497314342427136 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚18923929270784 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚4497314342427136 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚4482640187244544 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 135090745 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚84178781 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚18923894741504 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚239 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1229809357695488 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚4482640161071104 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 133042399 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 268123157 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚40016431 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1210885462953984 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 _runtime‚25815.983398016 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 E?DeviceStatsMonitor.on_train_batch_end/o£ŽÈversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 133042399 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 _step‚1284 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 133042405„ @ùº4À=/ _wandbruntime‚25816  _runtime‚25816;.+QÁ=‹ /loss/FBt1/detection_loss‚0.2810069115093754 %loss/total_loss‚4.225180484899621 epoch‚38 _runtime‚25816.06932954 _step‚1285 0loss/FRt1/assignment_loss‚0.9146200089882581 2loss/FBt1/assignment_loss‚0.000299325350366421 0loss/FRt1/detection_loss‚0.41671338891891857 .loss/FRt2/detection_loss‚1.264865050230055 0loss/FBt2/detection_loss‚0.18976506993955675 trainer/global_step‚15599 % _timestamp‚1.7843156318523984e+09 4loss/SRqqt1/assignment_loss‚0.044646973641997906 2loss/SRqqt2/assignment_loss‚0.0925375196798549 3loss/FBt2/assignment_loss‚0.0001855090559475743 1loss/FRt2/assignment_loss‚0.12422679202544257 1loss/SRqqt1/detection_loss‚0.4664017435017753 1loss/SRqqt2/detection_loss‚0.4299121920580721é/¾Â= 4loss/SRqqt1/assignment_loss‚0.044646973641997906 2loss/SRqqt2/assignment_loss‚0.0925375196798549 3loss/FBt2/assignment_loss‚0.0001855090559475743 1loss/FRt2/assignment_loss‚0.12422679202544257 1loss/SRqqt1/detection_loss‚0.4664017435017753 1loss/SRqqt2/detection_loss‚0.4299121920580721 /loss/FBt1/detection_loss‚0.2810069115093754 %loss/total_loss‚4.225180484899621 epoch‚38 _runtime‚25816.06932954 _step‚1285 0loss/FRt1/assignment_loss‚0.9146200089882581 2loss/FBt1/assignment_loss‚0.000299325350366421 0loss/FRt1/detection_loss‚0.41671338891891857 .loss/FRt2/detection_loss‚1.264865050230055 0loss/FBt2/detection_loss‚0.18976506993955675 trainer/global_step‚15599 % _timestamp‚1.7843156318523984e+09… JmXf’Ã=‚(:‡ ýýéÒÊ©†÷ gpu.0.gpu‚86.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚341.69' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚68.338 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Q¢ª Ä=‚(: ýýéÒù«¾ú disk.drbd2165.in‚0 disk.sda2.in‚0" proc.memory.rssMB‚ 2806.1953125 disk.sda.out‚ 1515.49609375 disk.sda3.in‚ 108.921875$ disk./.usageGB‚571.2367668151855 disk.sda.in‚ 108.921875 network.recv‚ 6.422066e+06 disk.rbd0.out‚0# disk.nvme0c0n1.in‚ 60023.9140625 disk.rbd1.out‚ 67.6005859375) proc.memory.availableMB‚ 3001.38671875 network.sent‚ 3.0135188e+07) disk.nvme0c0n1.out‚165249.91259765625! disk.nvme0n1.in‚ 60023.9140625 disk.rbd1.in‚ 0.53564453125 disk.dm-0.in‚ 60023.9140625 disk.sda1.out‚0 disk.sda3.out‚ 1515.49609375 disk.drbd2165.out‚0 proc.cpu.threads‚17) disk./.usagePercent‚38.33617872053906$ disk.dm-0.out‚165249.91259765625 disk.sda1.in‚0' disk.nvme0n1.out‚165249.91259765625# memory_percent‚81.6809892654419 disk.dm-1.out‚ 1515.49609375 disk.sda2.out‚0* proc.memory.percent‚17.127656936645508 cpu‚25.58261196856701 disk.dm-1.in‚ 108.921875& disk.rbd0.in‚1.759218604441423e+13ÉBêÒ4Å=/ _wandbruntime‚25844  _runtime‚25844ŽŸðB,>Æ=¦| 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6559059619903564 6 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.949999988079071 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 8"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.607296347618103 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6898014545440674 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 /particle/accuracy_4_of_4‚0.9261784553527832 particle/accuracy_4_of_5‚1 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.7731481790542603 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7335060834884644 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.05000000074505806 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9854008555412292 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 6"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7341348528862 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 particle/accuracy_3_of_5‚1 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.19703102111816406 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.5853179693222046 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.14537881314754486 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 *jet/accuracy_1_of_3‚0.9033348560333252 particle/accuracy_2_of_5‚1 _runtime‚25844.02319809 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.028509799391031265 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3543817400932312 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.17927834391593933 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.5847731828689575 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3441808223724365 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6788536906242371 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9753750562667847 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 trainer/global_step‚15599 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.12652386724948883 particle/accuracy_1_of_2‚1 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6551498770713806 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.02690204046666622 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.01762283593416214 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.023376796394586563 *jet/accuracy_2_of_3‚0.4629932641983032 +particle/specificity‚0.8975564241409302 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6699726581573486 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.023702427744865417 particle/accuracy_5_of_5‚1 8"Purity/0FRt*SRqqt*FBt/event_purity‚0.630804717540741 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.26923078298568726 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9910508990287781 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.7233560085296631 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7142857313156128 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5093117952346802 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.21439221501350403 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7174036502838135 & Purity/1FRt1SRqqt2FBt/FRt_purity‚1 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.09666667133569717 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.44907405972480774 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9910483360290527 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.10494992882013321 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7614783644676208 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 (particle/accuracy‚0.8302990198135376 +particle/sensitivity‚0.5843823552131653 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 *jet/accuracy_1_of_4‚0.9636698365211487 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6260207295417786 ,jet/accuracy_4_of_4‚0.013899499550461769 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6084710955619812 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.13391530513763428 /particle/accuracy_3_of_4‚0.9974937438964844 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.12899990379810333 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.13448691368103027 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6445714235305786 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.26923078298568726 jet/accuracy_3_of_5‚1 _step‚1286 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.1462116539478302 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9875399470329285 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6419393420219421 particle/accuracy_3_of_3‚1 particle/accuracy_2_of_4‚1 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.10939726233482361 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.5678421854972839 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9937976002693176 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.02737058699131012 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9843295812606812 /"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.515625 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.023376796394586563 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 'particle/f_score‚0.5966013669967651 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6677883267402649 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6547585725784302 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8428861498832703 - Purity/0FRt2SRqqt1FBt/FBt_purity‚0.765625 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.2222222238779068 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.13295264542102814 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.05000000074505806 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.22724519670009613 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.7906805872917175 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.8500000238418579 jet/accuracy_2_of_5‚1 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.8657407164573669 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7288985252380371 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9966966509819031 jet/accuracy_5_of_5‚0 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.16041193902492523 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.07919338345527649 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9863317608833313 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9869948625564575 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 *jet/accuracy_1_of_2‚0.7299647927284241 jet/accuracy_1_of_5‚1 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8237279057502747 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.8653846383094788 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7126357555389404 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9888888597488403 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.009395823813974857 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 particle/accuracy_1_of_1‚1 - Purity/0FRt2SRqqt*FBt/FBt_purity‚0.765625 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.8500000238418579 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.14569181203842163 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.15371040999889374 particle/accuracy_2_of_3‚1 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.2083333283662796 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.13318851590156555 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7517384886741638 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9686591625213623 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.12092451751232147 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4577867090702057 )jet/accuracy_2_of_4‚0.696069061756134 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9888569116592407 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5182866454124451 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.09918437898159027 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 particle/accuracy_1_of_5‚1 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.4825170636177063 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.5568174123764038 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.26851850748062134 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.13624794781208038 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.13701030611991882 *jet/accuracy_1_of_1‚0.2826123535633087 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.25753188133239746 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5942835807800293 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.26851850748062134 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9966966509819031 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9746650457382202 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.10170407593250275 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.6573128700256348 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.1352020502090454 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9940201044082642 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.21111734211444855 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.2222222238779068 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8888888955116272 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.18999643623828888 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 ) Purity/2FRt*SRqqt2FBt/FRt_purity‚0.75 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7857142686843872 epoch‚38 6 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.992642879486084 particle/accuracy_2_of_2‚1 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9790536761283875 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6325346827507019 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.42500001192092896 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9018235802650452 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 8"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.756965696811676 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7323410511016846 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8237279057502747 particle/accuracy_1_of_4‚1 *loss/val_total_loss‚11.644770467318743 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0025322004221379757 7validation_average_jet_accuracy‚0.32487156987190247 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.21518269181251526 6 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.992594301700592 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9862642884254456 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.027842339128255844 6 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.997686505317688 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 8"Purity/0FRt1SRqqt0FBt/event_purity‚0.607296347618103 jet/accuracy_4_of_5‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6747989058494568 +jet/accuracy_2_of_2‚0.11507894098758698 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.45923519134521484 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9950113892555237 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.13295264542102814 8"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.734826922416687 +jet/accuracy_3_of_3‚0.04003160446882248 particle/accuracy_1_of_3‚1 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7268518209457397 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6340521574020386 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6437145471572876 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.46836525201797485 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7268518209457397 +jet/accuracy_3_of_4‚0.04323367029428482 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.5115740895271301 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9113219976425171 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.4879167079925537 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.625 , Purity/2FRt0SRqqt1FBt/FRt_purity‚0.53125 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9776607751846313 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.11597537994384766 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 % _timestamp‚1.7843156598053353e+09 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.12740398943424225 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.7731481790542603 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6896136403083801 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6214675903320312 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.7725844383239746 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9893961548805237 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.7356387376785278 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.026093069463968277 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6212562322616577 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6841820478439331 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.004285652656108141 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.12176713347434998 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7466553449630737 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.17207598686218262mÔXó”3Ç=«| 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.14537881314754486 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 *jet/accuracy_1_of_3‚0.9033348560333252 particle/accuracy_2_of_5‚1 _runtime‚25844.02319809 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.028509799391031265 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3543817400932312 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.17927834391593933 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.5847731828689575 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3441808223724365 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6788536906242371 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9753750562667847 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 trainer/global_step‚15599 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.12652386724948883 particle/accuracy_1_of_2‚1 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6551498770713806 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.02690204046666622 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.01762283593416214 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.023376796394586563 *jet/accuracy_2_of_3‚0.4629932641983032 +particle/specificity‚0.8975564241409302 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6699726581573486 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.023702427744865417 particle/accuracy_5_of_5‚1 8"Purity/0FRt*SRqqt*FBt/event_purity‚0.630804717540741 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.26923078298568726 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9910508990287781 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.7233560085296631 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7142857313156128 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5093117952346802 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.21439221501350403 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7174036502838135 & Purity/1FRt1SRqqt2FBt/FRt_purity‚1 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.09666667133569717 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.44907405972480774 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9910483360290527 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.10494992882013321 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7614783644676208 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 (particle/accuracy‚0.8302990198135376 +particle/sensitivity‚0.5843823552131653 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 *jet/accuracy_1_of_4‚0.9636698365211487 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6260207295417786 ,jet/accuracy_4_of_4‚0.013899499550461769 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6084710955619812 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.13391530513763428 /particle/accuracy_3_of_4‚0.9974937438964844 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.12899990379810333 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.13448691368103027 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6445714235305786 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.26923078298568726 jet/accuracy_3_of_5‚1 _step‚1286 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.1462116539478302 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9875399470329285 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6419393420219421 particle/accuracy_3_of_3‚1 particle/accuracy_2_of_4‚1 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.10939726233482361 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.5678421854972839 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9937976002693176 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.02737058699131012 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9843295812606812 /"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.515625 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.023376796394586563 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 'particle/f_score‚0.5966013669967651 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6677883267402649 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6547585725784302 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8428861498832703 - Purity/0FRt2SRqqt1FBt/FBt_purity‚0.765625 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.2222222238779068 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.13295264542102814 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.05000000074505806 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.22724519670009613 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.7906805872917175 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.8500000238418579 jet/accuracy_2_of_5‚1 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.8657407164573669 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7288985252380371 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9966966509819031 jet/accuracy_5_of_5‚0 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.16041193902492523 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.07919338345527649 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9863317608833313 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9869948625564575 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 *jet/accuracy_1_of_2‚0.7299647927284241 jet/accuracy_1_of_5‚1 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8237279057502747 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.8653846383094788 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7126357555389404 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9888888597488403 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.009395823813974857 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 particle/accuracy_1_of_1‚1 - Purity/0FRt2SRqqt*FBt/FBt_purity‚0.765625 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.8500000238418579 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.14569181203842163 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.15371040999889374 particle/accuracy_2_of_3‚1 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.2083333283662796 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.13318851590156555 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7517384886741638 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9686591625213623 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.12092451751232147 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4577867090702057 )jet/accuracy_2_of_4‚0.696069061756134 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9888569116592407 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5182866454124451 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.09918437898159027 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 particle/accuracy_1_of_5‚1 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.4825170636177063 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.5568174123764038 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.26851850748062134 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.13624794781208038 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.13701030611991882 *jet/accuracy_1_of_1‚0.2826123535633087 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.25753188133239746 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5942835807800293 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.26851850748062134 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9966966509819031 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9746650457382202 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.10170407593250275 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.6573128700256348 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.1352020502090454 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9940201044082642 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.21111734211444855 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.2222222238779068 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8888888955116272 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.18999643623828888 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 ) Purity/2FRt*SRqqt2FBt/FRt_purity‚0.75 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7857142686843872 epoch‚38 6 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.992642879486084 particle/accuracy_2_of_2‚1 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9790536761283875 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6325346827507019 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.42500001192092896 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9018235802650452 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 8"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.756965696811676 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7323410511016846 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8237279057502747 particle/accuracy_1_of_4‚1 *loss/val_total_loss‚11.644770467318743 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0025322004221379757 7validation_average_jet_accuracy‚0.32487156987190247 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.21518269181251526 6 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.992594301700592 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9862642884254456 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.027842339128255844 6 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.997686505317688 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 8"Purity/0FRt1SRqqt0FBt/event_purity‚0.607296347618103 jet/accuracy_4_of_5‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6747989058494568 +jet/accuracy_2_of_2‚0.11507894098758698 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.45923519134521484 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9950113892555237 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.13295264542102814 8"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.734826922416687 +jet/accuracy_3_of_3‚0.04003160446882248 particle/accuracy_1_of_3‚1 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7268518209457397 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6340521574020386 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6437145471572876 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.46836525201797485 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7268518209457397 +jet/accuracy_3_of_4‚0.04323367029428482 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.5115740895271301 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9113219976425171 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.4879167079925537 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.625 , Purity/2FRt0SRqqt1FBt/FRt_purity‚0.53125 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9776607751846313 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 8 Purity/1FRt1SRqÿ6(X qt*FBt/FRt_purity‚0.11597537994384766 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 % _timestamp‚1.7843156598053353e+09 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.12740398943424225 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.7731481790542603 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6896136403083801 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6214675903320312 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.7725844383239746 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9893961548805237 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.7356387376785278 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.026093069463968277 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6212562322616577 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6841820478439331 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.004285652656108141 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.12176713347434998 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7466553449630737 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.17207598686218262 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6559059619903564 6 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.949999988079071 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 8"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.607296347618103 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6898014545440674 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 /particle/accuracy_4_of_4‚0.9261784553527832 particle/accuracy_4_of_5‚1 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.7731481790542603 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7335060834884644 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.05000000074505806 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9854008555412292 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 6"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7341348528862 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 particle/accuracy_3_of_5‚1 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.19703102111816406 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.5853179693222046† rY:žÈ=‚(:“ ŒþéҚŊ÷ gpu.0.gpu‚99.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚291.631' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚58.32619999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0« É=‚(: ŒþéÒç”’ú# disk.dm-0.out‚165269.2255859375" disk.nvme0n1.in‚60222.93359375 disk.rbd1.in‚ 0.53564453125 disk.sda.in‚ 108.921875& disk.rbd0.in‚1.759218604441423e+13 disk.sda1.out‚0 disk.sda3.in‚ 108.921875) proc.memory.availableMB‚ 2937.05078125 network.sent‚ 3.0187987e+07 disk.rbd0.out‚0 disk.sda1.in‚0 disk.drbd2165.in‚0( disk.nvme0c0n1.out‚165269.2255859375$ memory_percent‚82.07366466522217 network.recv‚ 6.426606e+06$ disk./.usageGB‚571.2411079406738 disk.drbd2165.out‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-1.in‚ 108.921875 disk.dm-1.out‚ 1517.44140625# proc.memory.rssMB‚ 2869.77734375* disk./.usagePercent‚38.336470057113026$ disk.nvme0c0n1.in‚60222.93359375 disk.rbd1.out‚ 67.7255859375 disk.sda.out‚ 1517.44140625) proc.memory.percent‚17.51573085784912 proc.cpu.threads‚17 disk.dm-0.in‚60222.93359375& disk.nvme0n1.out‚165269.2255859375 disk.sda3.out‚ 1517.44140625 cpu‚25.583656847678128 ‡x”Ê=‚(:‰ ›þéÒǘŽ÷ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚266.886' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.3772 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ï@°AË=‚(:‰ ›þéÒãÄŸú$ disk.nvme0n1.out‚165292.12109375 disk.rbd1.in‚ 0.53564453125 disk.rbd1.out‚ 67.7255859375 disk.drbd2165.in‚0& disk.rbd0.in‚1.759218604441423e+13 disk.sda1.out‚0 proc.cpu.threads‚17) proc.memory.availableMB‚ 2937.27734375 disk.sda3.out‚ 1520.44921875$ disk.nvme0c0n1.in‚60223.15234375 disk.sda2.out‚0 disk.sda.out‚ 1520.44921875# proc.memory.rssMB‚ 2869.77734375) proc.memory.percent‚17.51573085784912! disk.dm-0.out‚165292.12109375 disk.sda1.in‚0 disk.sda3.in‚ 108.921875" disk.nvme0n1.in‚60223.15234375 disk.dm-1.out‚ 1520.44921875$ memory_percent‚82.07228183746338 cpu‚25.583627469133315 network.sent‚ 3.0245995e+07$ disk./.usageGB‚571.2496223449707 disk.rbd0.out‚0 disk.drbd2165.out‚0 disk.sda.in‚ 108.921875 disk.dm-1.in‚ 108.921875 network.recv‚ 6.430667e+06 disk.dm-0.in‚60223.15234375& disk.nvme0c0n1.out‚165292.12109375 disk.sda2.in‚0) disk./.usagePercent‚38.33704146592945¦. ”Ì=‚(:‰ ªþéÒœ©Šù gpu.0.gpu‚32.0 gpu.0.memory‚10, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚320.961' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚64.1922 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Å>€ Í=‚(: ªþéÒ¥‘žú) disk.nvme0c0n1.out‚165309.07275390625' disk.nvme0n1.out‚165309.07275390625$ memory_percent‚82.07228183746338 disk.rbd1.out‚ 67.8505859375 proc.cpu.threads‚17 network.sent‚ 3.0251058e+07 network.recv‚ 6.434098e+06 disk.dm-0.in‚60223.57421875 disk.sda1.in‚0 disk.sda3.out‚ 1522.1796875$ disk.nvme0c0n1.in‚60223.57421875 disk.rbd1.in‚ 0.53564453125) proc.memory.percent‚17.51573085784912 disk.dm-1.in‚ 108.921875 disk.dm-1.out‚ 1522.1796875& disk.rbd0.in‚1.759218604441423e+13 disk.rbd0.out‚0 disk.sda3.in‚ 108.921875 disk.drbd2165.out‚0" disk.nvme0n1.in‚60223.57421875) disk./.usagePercent‚38.33756090521643$ disk.dm-0.out‚165309.07275390625 disk.drbd2165.in‚0 disk.sda2.in‚0 disk.sda2.out‚0# proc.memory.rssMB‚ 2869.77734375 cpu‚25.583608028697086 disk.sda.in‚ 108.921875 disk.sda.out‚ 1522.1796875 disk.sda1.out‚0) proc.memory.availableMB‚ 2937.27734375$ disk./.usageGB‚571.2573623657227¶†/»ŸÎ=‚(:” ¹þéÒó¿ƒ÷ gpu.0.gpu‚90.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚262.342' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚52.468399999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ô„ØsÏ=‚(: ¹þéÒ¬½Áù disk.drbd2165.in‚0 disk.drbd2165.out‚0$ disk./.usageGB‚571.2575492858887) disk.nvme0c0n1.out‚165322.15185546875 disk.sda2.in‚0 disk.sda.out‚ 1523.89453125 disk.dm-0.in‚60223.77734375 disk.sda1.in‚0 network.sent‚ 3.0256074e+07* disk./.usagePercent‚38.337573449585605 disk.rbd1.out‚ 67.8505859375& disk.rbd0.in‚1.759218604441423e+13 disk.sda1.out‚0 disk.sda3.in‚ 108.921875$ disk.nvme0c0n1.in‚60223.77734375' disk.nvme0n1.out‚165322.15185546875 cpu‚25.583561081227025 disk.sda2.out‚0 disk.sda.in‚ 108.921875 proc.cpu.threads‚17 disk.dm-1.out‚ 1523.89453125$ disk.dm-0.out‚165322.15185546875" disk.nvme0n1.in‚60223.77734375 disk.sda3.out‚ 1523.89453125 network.recv‚ 6.437225e+06& proc.memory.availableMB‚ 2936.78125) proc.memory.percent‚17.51573085784912 disk.rbd1.in‚ 0.53564453125 disk.dm-1.in‚ 108.921875 disk.rbd0.out‚0$ memory_percent‚82.07530975341797# proc.memory.rssMB‚ 2869.77734375m(3Z”Ð=‚(:‰ ÈþéÒسŽ÷ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚288.439' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚57.6878 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0pA$ƒ Ñ=‚(:þ ÈþéÒï²ÿù disk.sda2.in‚0 disk.nvme0n1.in‚ 60323.9375 disk.rbd1.in‚ 0.53564453125 disk.rbd1.out‚ 67.9130859375 disk.sda1.in‚0 disk.nvme0c0n1.in‚ 60323.9375& disk.nvme0n1.out‚165330.0361328125 disk.sda2.out‚0 disk.sda1.out‚0 disk.sda3.in‚ 108.92578125 disk.sda3.out‚ 1524.96875 proc.cpu.threads‚17 network.sent‚ 3.0262523e+07 disk.dm-1.out‚ 1524.96875& disk.rbd0.in‚1.759218604441423e+13 disk.drbd2165.in‚0# proc.memory.rssMB‚ 2869.77734375) proc.memory.percent‚17.51573085784912 cpu‚25.58351176415412) disk./.usagePercent‚38.33803707923011$ disk./.usageGB‚571.2644577026367 disk.rbd0.out‚0# disk.dm-0.out‚165330.0361328125 disk.drbd2165.out‚0( disk.nvme0c0n1.out‚165330.0361328125 disk.sda.out‚ 1524.96875$ memory_percent‚82.07230567932129( proc.memory.availableMB‚ 2937.2734375 network.recv‚ 6.440509e+06 disk.sda.in‚ 108.92578125 disk.dm-1.in‚ 108.92578125 disk.dm-0.in‚ 60323.9375ÄŠ¨{Ò=‚(:‹ ×þéÒþ½¯ù disk.drbd2165.out‚0 disk.nvme0n1.in‚ 60324.359375 disk.sda2.in‚0 network.sent‚ 3.0266239e+07 disk.dm-1.in‚ 108.92578125 disk.rbd0.out‚0 disk.drbd2165.in‚0( disk.nvme0c0n1.out‚165347.5205078125# proc.memory.rssMB‚ 2869.77734375$ disk./.usageGB‚571.2711563110352 disk.rbd1.in‚ 0.53564453125 proc.cpu.threads‚17 disk.rbd1.out‚ 67.9130859375 network.recv‚ 6.443523e+06 disk.dm-1.out‚ 1525.3828125) proc.memory.percent‚17.51573085784912# disk.dm-0.out‚165347.5205078125 disk.sda1.out‚0 cpu‚25.583512931570404) disk./.usagePercent‚38.33848662846024 disk.sda3.in‚ 108.92578125 disk.sda3.out‚ 1525.3828125 disk.sda.out‚ 1525.3828125" disk.nvme0c0n1.in‚ 60324.359375& disk.nvme0n1.out‚165347.5205078125 disk.sda2.out‚0# memory_percent‚82.0723295211792 disk.sda.in‚ 108.92578125) proc.memory.availableMB‚ 2937.26953125& disk.rbd0.in‚1.759218604441423e+13 disk.dm-0.in‚ 60324.359375 disk.sda1.in‚0úú»Ó=‚(:„ ×þéÒÝÄí‰ gpu.0.gpu‚83.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚46.0 gpu.0.powerWatts‚337.0' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚67.4 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0·md¬4Ô=/ _wandbruntime‚25922  _runtime‚25922!bŽÉÕ=à $ _timestamp‚1.784315738698747e+09 _runtime‚25922.915926893 _step‚1287 & lr-AdamW/pg1‚8.076860386502981e-05 & lr-AdamW/pg2‚8.076860386502981e-05 trainer/global_step‚15649|ldRÎÖ=È & lr-AdamW/pg1‚8.076860386502981e-05 & lr-AdamW/pg2‚8.076860386502981e-05 trainer/global_step‚15649 $ _timestamp‚1.784315738698747e+09 _runtime‚25922.915926893 _step‚1287‡ CV)4×=/ _wandbruntime‚25922  _runtime‚25922ëË(Ø=“P S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚4514814221574144 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚40149951 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14725388682240 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚4512845612538479 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 269133513 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14725388682240 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 135566987 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 _step‚1288 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚18990399755264 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚242 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚4498146790658048 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚44332886 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1214730409177088 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚4500090204454912 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚84483079 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚4498146765230080 $ _timestamp‚1.784315738699642e+09 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚4514815593137152 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14700191908721 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 135566987 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚4500090178281472 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 400530432 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 135557014 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚4514814221574144 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14724043292672 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚84482837 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 133576499 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚4500090178281472 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14724043292672 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚4500090204454912 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 133576505 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1233721209462784 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚44333123 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚237 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 133576499 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚18990432194560 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 269143492 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 _runtime‚25922.917075811`–³(Ù=˜P GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 135557014 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1233720808932352 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚40149951 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14725388682240 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚4512845612538479 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 269133513 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14725388682240 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 135566987 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 _step‚1288 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚18990399755264 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚242 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚4498146790658048 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚44332886 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1214730409177088 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚4500090204454912 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚84483079 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚4498146765230080 $ _timestamp‚1.784315738699642e+09 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚4514815593137152 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14700191908721 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 135566987 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚4500090178281472 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 400530432 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 135557014 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚4514814221574144 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14724043292672 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚84482837 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 133576499 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚4500090178281472 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14724043292672 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚4500090204454912 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 133576505 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1233721209462784 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚44333123 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚237 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 133576499 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚18990432194560 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 269143492 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 _runtime‚25922.917075811 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚4514814221574144 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 JDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 269160190 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚4513124417782901 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 133584711 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚40152494 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚84488139 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1214808263675904 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚25924.30626731 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 133584705 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚4500368179011584 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚238âEŽü('Ü=¢N I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚4513124417782901 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 133584711 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚40152494 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚84488139 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1214808263675904 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚25924.30626731 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 133584705 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚4500368179011584 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚238 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚40152499 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚44335645 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚84488377 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 135565492 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 269160190‰ ½Fž4Ý=/ _wandbruntime‚25924  _runtime‚25924¸nÅÞ=‹ _runtime‚25924.389627174 _step‚1290 1loss/FRt2/assignment_loss‚0.13734006430399323 .loss/FRt2/detection_loss‚1.213278637960765 4loss/SRqqt1/assignment_loss‚0.052453302307397115 3loss/FBt2/assignment_loss‚8.642384586262811e-07 2loss/SRqqt1/detection_loss‚0.34579401210110045 %loss/total_loss‚4.138494953573866 trainer/global_step‚15649 0loss/FRt1/assignment_loss‚0.8709764223005692 3loss/SRqqt2/assignment_loss‚0.13581713875174184 3loss/FBt1/assignment_loss‚4.815649507301194e-06 /loss/FRt1/detection_loss‚0.4172860721353244 1loss/SRqqt2/detection_loss‚0.4587742477135868 /loss/FBt1/detection_loss‚0.4086330904848832 0loss/FBt2/detection_loss‚0.09813628562653857 epoch‚39 " _timestamp‚1.7843157401727e+09Ø:ß= 1loss/FRt2/assignment_loss‚0.13734006430399323 .loss/FRt2/detection_loss‚1.213278637960765 4loss/SRqqt1/assignment_loss‚0.052453302307397115 3loss/FBt2/assignment_loss‚8.642384586262811e-07 2loss/SRqqt1/detection_loss‚0.34579401210110045 %loss/total_loss‚4.138494953573866 trainer/global_step‚15649 0loss/FRt1/assignment_loss‚0.8709764223005692 3loss/SRqqt2/assignment_loss‚0.13581713875174184 3loss/FBt1/assignment_loss‚4.815649507301194e-06 /loss/FRt1/detection_loss‚0.4172860721353244 1loss/SRqqt2/detection_loss‚0.4587742477135868 /loss/FBt1/detection_loss‚0.4086330904848832 0loss/FBt2/detection_loss‚0.09813628562653857 epoch‚39 " _timestamp‚1.7843157401727e+09 _runtime‚25924.389627174 _step‚1290Š ð=HŸà=‚(:” æþéÒæÕÐö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚258.298' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚51.659600000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Xº C á=‚(:‚ æþéÒ•ºúù disk.drbd2165.in‚0& disk.rbd0.in‚1.759218604441423e+13) proc.memory.percent‚17.51573085784912 disk.sda1.in‚0# disk./.usageGB‚571.275634765625 disk.rbd1.in‚ 0.53564453125 disk.rbd1.out‚ 67.9755859375 disk.sda.in‚ 108.92578125# proc.memory.rssMB‚ 2869.77734375 cpu‚25.583473106854665 network.sent‚ 3.0272158e+07 network.recv‚ 6.446958e+06 disk.sda3.out‚ 1525.640625 disk.drbd2165.out‚0 disk.nvme0n1.in‚ 60324.5625) disk.nvme0c0n1.out‚165356.42431640625 disk.sda2.out‚0 proc.cpu.threads‚17) disk./.usagePercent‚38.33878718130544' disk.nvme0n1.out‚165356.42431640625 disk.sda2.in‚0 disk.dm-1.in‚ 108.92578125 disk.dm-1.out‚ 1525.640625& proc.memory.availableMB‚ 2937.03125 disk.nvme0c0n1.in‚ 60324.5625 disk.dm-0.in‚ 60324.5625 disk.sda.out‚ 1525.640625 disk.rbd0.out‚0$ disk.dm-0.out‚165356.42431640625 disk.sda1.out‚0$ memory_percent‚82.07378387451172 disk.sda3.in‚ 108.9257812553U§žâ=‚(:“ õþéÒ¨·äö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚297.575' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚59.51499999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‘g ãã=‚(:Ž õþéÒ¸´¹ù disk.sda1.out‚0' disk.nvme0n1.out‚165367.25537109375 cpu‚25.583454807262324 disk.drbd2165.out‚0 network.sent‚ 3.0334675e+07) disk./.usagePercent‚38.33925029893488& disk.rbd0.in‚1.759218604441423e+13 disk.rbd0.out‚0 disk.dm-1.out‚ 1525.98828125 disk.sda1.in‚0( proc.memory.availableMB‚ 2936.9453125$ disk.dm-0.out‚165367.25537109375 disk.sda3.in‚ 108.92578125! disk.nvme0c0n1.in‚ 60324.78125) disk.nvme0c0n1.out‚165367.25537109375 disk.sda2.out‚0# proc.memory.rssMB‚ 2869.77734375 proc.cpu.threads‚17 network.recv‚ 6.451271e+06 disk.sda3.out‚ 1525.98828125 disk.drbd2165.in‚0 disk.sda2.in‚0$ disk./.usageGB‚571.2825355529785 disk.sda.in‚ 108.92578125 disk.sda.out‚ 1525.98828125 disk.dm-0.in‚ 60324.78125) proc.memory.percent‚17.51573085784912 disk.dm-1.in‚ 108.92578125 disk.nvme0n1.in‚ 60324.78125 disk.rbd1.in‚ 0.53564453125 disk.rbd1.out‚ 67.9755859375$ memory_percent‚82.07430839538574Ž´†`•ä=‚(:Š „ÿéÒ€¤÷ gpu.0.gpu‚100.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚46.0 gpu.0.powerWatts‚333.378' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚66.6756 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0§L{å=‚(:” „ÿéÒ› òø) disk./.usagePercent‚38.33972365686569 disk.sda.in‚ 108.92578125 disk.rbd0.out‚0$ disk.nvme0c0n1.in‚60524.41015625 disk.sda2.in‚0) proc.memory.availableMB‚ 2936.68359375 disk.drbd2165.out‚0 disk.sda2.out‚0 disk.rbd1.in‚ 0.53564453125# proc.memory.rssMB‚ 2869.77734375 disk.dm-1.in‚ 108.92578125 disk.dm-1.out‚ 1526.4296875 disk.rbd1.out‚ 68.0380859375) proc.memory.percent‚17.51573085784912 proc.cpu.threads‚17$ disk./.usageGB‚571.2895889282227 disk.dm-0.in‚60524.41015625" disk.nvme0n1.in‚60524.41015625 cpu‚25.58342702657395& disk.rbd0.in‚1.759218604441423e+13 disk.sda1.in‚0 network.sent‚ 3.0341097e+07 disk.sda.out‚ 1526.4296875$ disk.dm-0.out‚165381.22802734375 disk.sda1.out‚0 disk.sda3.out‚ 1526.4296875 disk.drbd2165.in‚0 network.recv‚ 6.454958e+06) disk.nvme0c0n1.out‚165381.22802734375 disk.sda3.in‚ 108.92578125' disk.nvme0n1.out‚165381.22802734375$ memory_percent‚82.07590579986572K>@%Ÿæ=‚(:” “ÿéÒ˜ ¤÷ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚263.941' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚52.788199999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0<÷6ç=‚(: “ÿéÒÄäŸø) proc.memory.percent‚17.51573085784912 network.recv‚ 6.458218e+06" disk.nvme0n1.in‚60524.61328125' disk.nvme0n1.out‚165392.22509765625 disk.rbd1.in‚ 0.53564453125 disk.sda2.out‚0 disk.sda.in‚ 108.92578125 disk.dm-1.out‚ 1526.96875$ disk.dm-0.out‚165392.22509765625 disk.sda1.in‚0 disk.drbd2165.in‚0$ disk.nvme0c0n1.in‚60524.61328125 network.sent‚ 3.0346169e+07* disk./.usagePercent‚38.340026513778696 disk.rbd0.out‚0 disk.sda1.out‚0 disk.drbd2165.out‚0 disk.rbd1.out‚ 68.0380859375( proc.memory.availableMB‚ 2936.9296875 proc.cpu.threads‚17& disk.rbd0.in‚1.759218604441423e+13 disk.dm-0.in‚60524.61328125 disk.dm-1.in‚ 108.92578125$ memory_percent‚82.07440376281738) disk.nvme0c0n1.out‚165392.22509765625 disk.sda2.in‚0$ disk./.usageGB‚571.2941017150879 disk.sda3.in‚ 108.92578125# proc.memory.rssMB‚ 2869.77734375 disk.sda.out‚ 1526.96875 disk.sda3.out‚ 1526.96875 cpu‚25.5833706143715þ,žè=‚(:“ ¢ÿéÒº¯„÷ gpu.0.gpu‚84.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚46.0 gpu.0.powerWatts‚333.524' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚66.70479999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Øášé=‚(:’ ¢ÿéÒÝØ¥ù* disk./.usagePercent‚38.340728486437584 disk.sda3.in‚ 108.92578125 network.sent‚ 3.0351202e+07 disk.rbd0.out‚0$ disk.nvme0c0n1.in‚60524.83203125) proc.memory.availableMB‚ 2936.69140625( disk.nvme0c0n1.out‚165410.0283203125 network.recv‚ 6.461726e+06 disk.sda.in‚ 108.92578125 disk.sda1.in‚0 disk.sda3.out‚ 1528.9609375 disk.dm-0.in‚60524.83203125# memory_percent‚82.0758581161499# proc.memory.rssMB‚ 2869.77734375 disk.sda1.out‚0 disk.drbd2165.out‚0 cpu‚25.583387655974295" disk.nvme0n1.in‚60524.83203125 disk.dm-1.in‚ 108.92578125 proc.cpu.threads‚17& disk.nvme0n1.out‚165410.0283203125 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd1.out‚ 68.1005859375 disk.sda.out‚ 1528.9609375& disk.rbd0.in‚1.759218604441423e+13# disk.dm-0.out‚165410.0283203125 disk.drbd2165.in‚0$ disk./.usageGB‚571.3045616149902 disk.rbd1.in‚ 0.53564453125 disk.dm-1.out‚ 1528.9609375) proc.memory.percent‚17.51573085784912wðùf4ê=/ _wandbruntime‚26002  _runtime‚26002œ£˜Êë=Ä & lr-AdamW/pg1‚7.899667055123814e-05 & lr-AdamW/pg2‚7.899667055123814e-05 trainer/global_step‚15699 % _timestamp‚1.7843158185023513e+09 _runtime‚26002.719589631 _step‚12910P©_Ïì=É & lr-AdamW/pg1‚7.899667055123814e-05 & lr-AdamW/pg2‚7.899667055123814e-05 trainer/global_step‚15699 % _timestamp‚1.7843158185023513e+09 _runtime‚26002.719589631 _step‚1291‹ M Ò}4í=/ _wandbruntime‚26002  _runtime‚26002lŸ î=”P J[¬{llocated‚ 49071259648 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 135992739 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14771704612864 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 133986805 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 400530432 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚4513990240960512 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚44471138 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚245 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚4526785938560387 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚4513990214787072 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚84748244 % _timestamp‚1.7843158185065901e+09 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 133986799 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14770359223296 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚44470893 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1237655805942784 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1218604729561088 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚4528761945573376 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14770359223296 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 _runtime‚26002.724137056 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚84747994 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 _step‚1292 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚19050708290560 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 135982766 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14771704612864 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 269979544 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚40277101 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚32439296 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚4528760574010368 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚40277106 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚4528761945573376 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚19050675851264 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1218605097652224 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 269969565 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚250 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚4528760574010368 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚4526787308588677xeB\(ï=™P F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 269969565 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚250 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚4528760574010368 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚4526787308588677 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14770359223296 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚44470893 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1237655805942784 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1218604729561088 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚4528761945573376 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14770359223296 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 _runtime‚26002.724137056 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚84747994 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 _step‚1292 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚19050708290560 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 135982766 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14771704612864 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 269979544 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚40277101 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚32439296 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚4528760574010368 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚40277106 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚4528761945573376 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚19050675851264 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1218605097652224Œ Å3Ä4ð=/ _wandbruntime‚26004  _runtime‚26004±/2á$'ñ=žN GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚4529040871346176 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1218682215968768 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14771284258816 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚19051913910272 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚4527064743804809 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 135991244 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 133995011 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 133995011 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 133995005 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚44473631 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚84753521 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚4529040871346176 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚241 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚4514268215517184 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚4527066113833211 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚4529039499776000 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 136001231 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7843158199075904e+09 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1218682584059904 trainer/global_step‚15699 _step‚1293 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚4512318735917056 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 269996242 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚44473872 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 135991244 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14772629655552 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚246 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14746007887753 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚4514268241690624 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14771284258816 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 133995005 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚84753275ôïW‡ëò=£N H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 269996242 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1237734497970176 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚4512318761345024 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚40279644 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚19051879380992 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚40279649 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚4514268215517184 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚84753275 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚4529040871346176 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1218682215968768 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14771284258816 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚19051913910272 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚4527064743804809 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 135991244 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 133995011 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 133995011 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 133995005 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚44473631 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚84753521 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚4529040871346176 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚241 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚4514268215517184 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚4527066113833211 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚4529039499776000 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 E=DeviceStatsMonit,Ò•M>or.on_train_batch_end/inactive_split.all.peak‚572 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 136001231 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7843158199075904e+09 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1218682584059904 trainer/global_step‚15699 _step‚1293 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚4512318735917056 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 269996242 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚44473872 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 135991244 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14772629655552 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚246 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14746007887753 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚4514268241690624 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14771284258816 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 133995005 Q/ _wandbruntime‚26083  _runtime‚26083´@ÃÒÊ>Ä & lr-AdamW/pg1‚7.724188149553439e-05 & lr-AdamW/pg2‚7.724188149553439e-05 trainer/global_step‚15749 % _timestamp‚1.7843158987958407e+09 _runtime‚26083.013008215 _step‚1295-Í´+Ï‚>É & lr-AdamW/pg2‚7.724188149553439e-05 trainer/global_step‚15749 % _timestamp‚1.7843158987958407e+09 _runtime‚26083.013008215 _step‚1295 & lr-AdamW/pg1‚7.724188149553439e-05 q·Øù4ƒ>/ _wandbruntime‚26083  _runtime‚26083S M›(„>”P JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 136408518 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚4540726264582295 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚4527890277466112 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 _runtime‚26083.014223129 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 _step‚1296 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 270805617 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 134397105 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚4527890277466112 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚4540727634610585 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14818020543488 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1222479418036224 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14792666134937 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚44608267 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚40404256 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1241589515366400 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1222479049945088 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚19110465421312 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚40404251 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 270815596 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚260 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 % _timestamp‚1.7843158987966769e+09 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚4542706926446592 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14818020543488 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚19110499957760 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚4527890251292672 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚85012518 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 136418491 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 136418491 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175ƒ‘•0 …>™P JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 136408518 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚4540726264582295 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚4527890277466112 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 _runtime‚26083.014223129 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 _step‚1296 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 270805617 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 134397105 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚4527890277466112 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚4540727634610585 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14818020543488 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1222479418036224 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14792666134937 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚44608267 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚40404256 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1241589515366400 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1222479049945088 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚19110465421312 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097ÛVfgï G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚40404251 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 270815596 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚260 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 % _timestamp‚1.7843158987966769e+09 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚4542706926446592 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14818020543488 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚19110499957760 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚4527890251292672 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚85012518 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 136418491 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 136418491 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 ו|Z4†>/ _wandbruntime‚26084  _runtime‚26084Yõëì!'‡>›N TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚4528168278196224 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚85017770 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚4526212850253824 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚4542987223782400 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 136426983 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚44610976 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 136416996 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚40406794 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚19111694657536 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚19111660128256 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚4542987223782400 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚4528168278196224 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 134405305 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚85018024 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14792245000861 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚254 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1222556904443904 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 _runtime‚ 26084.4076042 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 270822301 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14818945586176 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚4542985852212224 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1241668599101440 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 136416996 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14817600189440 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚4528168252022784 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 $ _timestamp‚1.784315900190211e+09 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 136426983 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 270822301 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 134405311 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14818945586176 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _step‚1297 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚44611225 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 270832294 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 134405305 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚4528168252022784 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0vwþ4&'ˆ> N REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 270832294 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 134405305 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚4528168252022784 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚4528168278196224 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚85017770 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚4526212850253824 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚4542987223782400 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 136426983 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚44610976 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 136416996 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚40406794 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚19111694657536 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚19111660128256 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚4542987223782400 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚4528168278196224 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 134405305 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚85018024 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14792245000861 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚254 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1222556904443904 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 _runtime‚ 26084.4076042 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 270822301 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14818945586176 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚4542985852212224 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1241668599101440 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 136416996 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14817600189440 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚4528168252022784 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 $ _timestamp‚1.784315900190211e+09 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 136426983 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 270822301 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 134405311 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14818945586176 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _step‚1297 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚44611225 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 E/ _wandbruntime‚26084  _runtime‚26084+y˜=Š>Œ 0loss/FBt2/detection_loss‚0.14567202137749594 /loss/FRt1/assignment_loss‚0.852605194805964 /loss/FRt2/detection_loss‚1.2140336521163388 % _timestamp‚1.7843159002719805e+09 _runtime‚26084.488908286 %loss/total_loss‚4.083559843305582 3loss/FBt1/assignment_loss‚3.069363412474705e-05 /loss/FRt1/detection_loss‚0.4203130426915715 2loss/SRqqt2/detection_loss‚0.48088836205721014 0loss/FBt1/detection_loss‚0.12677404981753257 trainer/global_step‚15749 1loss/FRt2/assignment_loss‚0.13572159138592807 1loss/SRqqt1/detection_loss‚0.5255417079394034 _step‚1298 epoch‚39 2loss/SRqqt1/assignment_loss‚0.0696574802863746 2loss/SRqqt2/assignment_loss‚0.1095003201656814 3loss/FBt2/assignment_loss‚0.0028217270279569107ÓéO´‹>‘ %loss/total_loss‚4.083559843305582 3loss/FBt1/assignment_loss‚3.069363412474705e-05 /loss/FRt1/detection_loss‚0.4203130426915715 2loss/SRqqt2/detection_loss‚0.48088836205721014 0loss/FBt1/detection_loss‚0.12677404981753257 trainer/global_step‚15749 1loss/FRt2/assignment_loss‚0.13572159138592807 1loss/SRqqt1/detection_loss‚0.5255417079394034 _step‚1298 epoch‚39 2loss/SRqqt1/assignment_loss‚0.0696574802863746 2loss/SRqqt2/assignment_loss‚0.1095003201656814 3loss/FBt2/assignment_loss‚0.0028217270279569107 0loss/FBt2/detection_loss‚0.14567202137749594 /loss/FRt1/assignment_loss‚0.852605194805964 /loss/FRt2/detection_loss‚1.2140336521163388 % _timestamp‚1.7843159002719805e+09 _runtime‚26084.488908286’ îdö”Œ>‚(:‰ üÿéÒœ»ö gpu.0.gpu‚76.0 gpu.0.memory‚25, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚345.218' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚69.0436 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‚¤ï>‚(:‘ üÿéÒÑßæø disk.sda3.in‚ 108.92578125$ disk.nvme0c0n1.in‚60784.92578125 disk.sda2.out‚0 disk.sda1.out‚0 disk.drbd2165.out‚0 disk.dm-1.in‚ 108.92578125' proc.memory.availableMB‚ 2937.515625$ memory_percent‚82.07082748413086 proc.cpu.threads‚17& disk.nvme0n1.out‚165529.9306640625 disk.sda2.in‚0 disk.sda.out‚ 1545.6953125 cpu‚25.583283145113082 disk.dm-0.in‚60784.92578125 disk.rbd1.in‚ 0.53564453125 disk.rbd1.out‚ 68.4130859375# proc.memory.rssMB‚ 2869.77734375 disk.rbd0.out‚0 disk.sda1.in‚0 disk.sda3.out‚ 1545.6953125 disk.dm-1.out‚ 1545.6953125 disk.sda.in‚ 108.92578125) proc.memory.percent‚17.51573085784912" disk.nvme0n1.in‚60784.92578125$ disk./.usageGB‚571.3370704650879 disk.drbd2165.in‚0( disk.nvme0c0n1.out‚165529.9306640625 network.sent‚ 3.0439912e+07 network.recv‚ 6.483686e+06* disk./.usagePercent‚38.342910182644374& disk.rbd0.in‚1.759218604441423e+13# disk.dm-0.out‚165529.9306640625 œ žŽ>‚(:“ ‹€êÒøñ—ù gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚258.285' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚51.65700000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ü!v‰>‚(:† ‹€êÒü¶¨ú& disk.rbd0.in‚1.759218604441423e+13 disk.dm-0.in‚60987.39453125 network.recv‚ 6.488149e+06 proc.cpu.threads‚17* disk./.usagePercent‚38.343356915791624 disk.drbd2165.out‚0 disk.sda.in‚ 108.92578125 disk.sda1.out‚0 disk.sda3.in‚ 108.92578125 disk.rbd1.in‚ 0.53564453125 disk.dm-1.in‚ 108.92578125 disk.rbd0.out‚0 disk.dm-0.out‚165546.2890625( proc.memory.availableMB‚ 2937.4296875# proc.memory.rssMB‚ 2869.77734375 network.sent‚ 3.0503201e+07 disk.sda3.out‚ 1546.078125# disk.nvme0n1.out‚165546.2890625 disk.drbd2165.in‚0" disk.nvme0n1.in‚60987.39453125 disk.sda2.in‚0 disk.dm-1.out‚ 1546.078125) proc.memory.percent‚17.51573085784912 cpu‚25.583241803778108$ disk./.usageGB‚571.3437271118164 disk.sda1.in‚0$ disk.nvme0c0n1.in‚60987.39453125% disk.nvme0c0n1.out‚165546.2890625 disk.sda2.out‚0$ memory_percent‚82.07135200500488 disk.rbd1.out‚ 68.4130859375 disk.sda.out‚ 1546.078125“X—D”>‚(:‰ š€êÒºšóö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚296.933' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚59.3866 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0HÅõ ‘>‚(:Œ š€êÒ÷Ƈù disk.drbd2165.out‚0 network.sent‚ 3.050752e+07" disk.nvme0n1.in‚60987.59765625 disk.sda3.in‚ 108.92578125 disk.sda.in‚ 108.92578125" disk.dm-0.out‚165554.896484375$ memory_percent‚82.06987380981445 disk.sda3.out‚ 1546.2890625 cpu‚25.583253988598177$ disk./.usageGB‚571.3481979370117 disk.sda2.in‚0 disk.rbd1.in‚ 0.53564453125 disk.dm-1.out‚ 1546.2890625& disk.rbd0.in‚1.759218604441423e+13 disk.sda1.in‚0) proc.memory.percent‚17.51573085784912 network.recv‚ 6.492276e+06 disk.sda.out‚ 1546.2890625# proc.memory.rssMB‚ 2869.77734375 proc.cpu.threads‚17% disk.nvme0n1.out‚165554.896484375 disk.dm-0.in‚60987.59765625 disk.sda1.out‚0$ disk.nvme0c0n1.in‚60987.59765625' proc.memory.availableMB‚ 2937.671875) disk./.usagePercent‚38.34365695662176 disk.sda2.out‚0 disk.rbd1.out‚ 68.4755859375 disk.dm-1.in‚ 108.92578125 disk.rbd0.out‚0 disk.drbd2165.in‚0' disk.nvme0c0n1.out‚165554.896484375.ð•’>‚(:Š ©€êÒ•ïº÷ gpu.0.gpu‚100.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚348.292' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚69.6584 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ð§ßî“>‚(:” ©€êÒûÄãù& disk.rbd0.in‚1.759218604441423e+13 disk.sda1.in‚0$ memory_percent‚82.06989765167236 proc.cpu.threads‚17 network.recv‚ 6.50222e+06) disk./.usagePercent‚38.34397901409983 disk.dm-0.in‚ 60987.8359375 disk.sda3.out‚ 1546.53515625 disk.sda2.out‚0# disk.nvme0c0n1.in‚ 60987.8359375# proc.memory.rssMB‚ 2869.77734375 disk.drbd2165.out‚0 disk.rbd1.in‚ 0.53564453125 network.sent‚ 3.0516636e+07 disk.rbd0.out‚0 disk.sda1.out‚0 disk.drbd2165.in‚0 cpu‚25.583214320904375 disk.sda.out‚ 1546.53515625 disk.dm-1.in‚ 108.92578125! disk.nvme0n1.in‚ 60987.8359375' disk.nvme0n1.out‚165567.21435546875 disk.sda.in‚ 108.92578125 disk.sda2.in‚0 disk.rbd1.out‚ 68.4755859375) proc.memory.availableMB‚ 2937.66796875) proc.memory.percent‚17.51573085784912$ disk./.usageGB‚571.3529968261719 disk.dm-1.out‚ 1546.53515625$ disk.dm-0.out‚165567.21435546875 disk.sda3.in‚ 108.92578125) disk.nvme0c0n1.out‚165567.214355468751­Ó}ž”>‚(:“ ¸€êÒ¾Ãÿö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚281.143' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚56.22859999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0üCܦ•>‚(:Ž ¸€êÒÁ øù) proc.memory.percent‚17.51573085784912 disk.dm-1.out‚ 1546.703125 disk.sda3.in‚ 108.92578125( disk.nvme0c0n1.out‚165581.1416015625& disk.nvme0n1.out‚165581.1416015625 proc.cpu.threads‚17 network.sent‚ 3.0520355e+07 network.recv‚ 6.505648e+06 disk.sda.in‚ 108.92578125 disk.rbd0.out‚0 disk.dm-0.in‚60988.23828125 disk.sda1.in‚0 cpu‚25.58316631632705 disk.sda3.out‚ 1546.703125 disk.drbd2165.out‚0$ disk.nvme0c0n1.in‚60988.23828125) proc.memory.availableMB‚ 2937.41796875 disk.sda2.in‚0$ disk./.usageGB‚571.3596038818359 disk.sda1.out‚0 disk.drbd2165.in‚0" disk.nvme0n1.in‚60988.23828125) disk./.usagePercent‚38.34442241914913 disk.sda.out‚ 1546.703125& disk.rbd0.in‚1.759218604441423e+13# disk.dm-0.out‚165581.1416015625 disk.sda2.out‚0$ memory_percent‚82.07142353057861 disk.dm-1.in‚ 108.92578125 disk.rbd1.in‚ 0.53564453125 disk.rbd1.out‚ 68.5380859375# proc.memory.rssMB‚ 2869.77734375%G.“–>‚(:ˆ Ç€êÒ˜Áðö gpu.0.gpu‚12.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚329.033' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚65.8066 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0éô:Í—>‚(:Œ Ç€êÒÊžÑù disk.sda.out‚ 1546.98046875 disk.dm-0.in‚ 61122.6796875 disk.sda1.out‚0 disk.drbd2165.out‚0 proc.cpu.threads‚17 disk.sda2.in‚0( proc.memory.availableMB‚ 2937.4140625 cpu‚25.58317656103318 network.sent‚ 3.0525394e+07 disk.dm-1.out‚ 1546.98046875 network.recv‚ 6.508903e+06! disk.nvme0n1.in‚ 61122.6796875 disk.sda.in‚ 108.92578125 disk.sda1.in‚0 disk.drbd2165.in‚0% disk.nvme0n1.out‚165591.306640625 disk.rbd1.out‚ 68.5380859375" disk.dm-0.out‚165591.306640625# disk.nvme0c0n1.in‚ 61122.6796875) proc.memory.percent‚17.51573085784912$ disk./.usageGB‚571.3640899658203 disk.sda2.out‚0 disk.dm-1.in‚ 108.92578125& disk.rbd0.in‚1.759218604441423e+13 disk.sda3.out‚ 1546.98046875( disk./.usagePercent‚38.3447234840094 disk.rbd1.in‚ 0.53564453125 disk.rbd0.out‚0 disk.sda3.in‚ 108.92578125' disk.nvme0c0n1.out‚165591.306640625# proc.memory.rssMB‚ 2869.77734375$ memory_percent‚82.07144737243652NöÜA4˜>/ _wandbruntime‚26163  _runtime‚26163”BS™>Ä % _timestamp‚1.7843159787948728e+09 _runtime‚26163.012015744 _steHCÌLwp‚1299 & lr-AdamW/pg1‚7.550434940474066e-05 & lr-AdamW/pg2‚7.550434940474066e-05 trainer/global_step‚15799ÐFpÏš>É & lr-AdamW/pg1‚7.550434940474066e-05 & lr-AdamW/pg2‚7.550434940474066e-05 trainer/global_step‚15799 % _timestamp‚1.7843159787948728e+09 _runtime‚26163.012015744 _step‚1299“ ºäîÄ4›>/ _wandbruntime‚26163  _runtime‚26163Í3•Ü(œ>’P \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14864336474112 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚4556653278882816 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚19170338350080 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚4554666590604203 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 trainer/global_step‚15799 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚4556653278882816 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14864336474112 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14862991084544 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 271641669 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚44746268 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚4554667960632493 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 136834270 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1226353738420224 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚265 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚19170303813632 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14837558647723 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 271651648 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚4556654650445824 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚4556654650445824 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚44746008 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚4539829031956480 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 _runtime‚26163.013363769 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 134807405 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 136844243 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 136834270 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚4539829057384448 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚260 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚4541790287798272“Qôa(>—P NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 134807405 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 136844243 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 136834270 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚4539829057384448 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚260 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚4541790287798272 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14864336474112 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚4556653278882816 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚19170338350080 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚4554666590604203 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 trainer/global_step‚15799 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚4556653278882816 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14864336474112 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14862991084544 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 271641669 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚44746268 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚4554667960632493 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 136834270 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1226353738420224 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚265 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚19170303813632 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14837558647723 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 271651648 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚4556654650445824 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚4556654650445824 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚44746008 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚4539829031956480 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 _runtime‚26163.013363769” ÆJg4ž>/ _wandbruntime‚26164  _runtime‚26164¶é„î$'Ÿ>žN OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 136852735 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚40533944 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚4556933576218624 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 136842748 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 136852735 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚4556933576218624 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 271658353 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚19171504361472 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 271668346 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚4542068288528384 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚246 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚44748998 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚4554945395848625 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 _step‚1301 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 134815611 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 _runtime‚26164.397655445 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚4554946765877027 trainer/global_step‚15799 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14865261516800 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚32432128 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚241 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14839826714403 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚4556932204648448 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14863916120064 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1245602361098240 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚4540106913734656 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 271668346 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚40533949 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚85282701 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 271658353 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 134815611 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚44748757 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1226431224827904 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 % _timestamp‚1.7843159801802926e+09°/U)Á >£N K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 136842748 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 136852735 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚4556933576218624 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 271658353 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚19171504361472 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 271668346 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚4542068288528384 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 N>DeviceStatsMonitor.on_train_batch_end/allocatedd`­h _bytes.all.peak‚ 47545124352 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚246 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚44748998 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚4554945395848625 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 _step‚1301 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 134815611 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 _runtime‚26164.397655445 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚4554946765877027 trainer/global_step‚15799 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14865261516800 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚32432128 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚241 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14839826714403 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚4556932204648448 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14863916120064 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1245602361098240 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚4540106913734656 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 271668346 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚40533949 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚85282701 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 271658353 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 134815611 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚44748757 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1226431224827904 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 % _timestamp‚1.7843159801802926e+09 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 136852735 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚40533944 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚4556933576218624 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995• ö—?¥4¡>/ _wandbruntime‚26164  _runtime‚261649\6½¢>Œ 1loss/FRt2/assignment_loss‚0.09648033569419319 1loss/FBt1/assignment_loss‚0.00090624931696144 0loss/FRt1/detection_loss‚0.43497660680144634 /loss/FRt2/detection_loss‚1.1672864569499937 &loss/total_loss‚3.7375732777244504 epoch‚39 trainer/global_step‚15799 _runtime‚26164.48385846 3loss/SRqqt2/assignment_loss‚0.11311007706175975 4loss/FBt2/assignment_loss‚0.00022199066749998105 1loss/SRqqt1/detection_loss‚0.3926774530632151 1loss/SRqqt2/detection_loss‚0.4361285870179028 0loss/FBt1/detection_loss‚0.07359261158423508 _step‚1302 .loss/FRt1/assignment_loss‚0.84642540992038 3loss/SRqqt1/assignment_loss‚0.07593045828453139 0loss/FBt2/detection_loss‚0.09983704136233142 % _timestamp‚1.7843159802669103e+098¯¦n£>‘ 1loss/SRqqt2/detection_loss‚0.4361285870179028 0loss/FBt1/detection_loss‚0.07359261158423508 _step‚1302 .loss/FRt1/assignment_loss‚0.84642540992038 3loss/SRqqt1/assignment_loss‚0.07593045828453139 0loss/FBt2/detection_loss‚0.09983704136233142 % _timestamp‚1.7843159802669103e+09 1loss/FRt2/assignment_loss‚0.09648033569419319 1loss/FBt1/assignment_loss‚0.00090624931696144 0loss/FRt1/detection_loss‚0.43497660680144634 /loss/FRt2/detection_loss‚1.1672864569499937 &loss/total_loss‚3.7375732777244504 epoch‚39 trainer/global_step‚15799 _runtime‚26164.48385846 3loss/SRqqt2/assignment_loss‚0.11311007706175975 4loss/FBt2/assignment_loss‚0.00022199066749998105 1loss/SRqqt1/detection_loss‚0.3926774530632151– ²ÞQÎ’¤>‚(:‡ Ö€êÒ¯÷¬÷ gpu.0.gpu‚98.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚271.19' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.238 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‡‰¥>‚(: Ö€êÒ’ô¸ú* disk./.usagePercent‚38.345489458551846$ disk./.usageGB‚571.3755035400391 disk.sda3.out‚ 1547.2265625 disk.rbd1.out‚ 68.6005859375 disk.dm-1.in‚ 108.92578125 disk.rbd0.out‚0 disk.sda2.out‚0) proc.memory.percent‚17.51573085784912 disk.rbd1.in‚ 0.53564453125# disk.dm-0.out‚165598.7021484375 disk.sda3.in‚ 108.92578125( disk.nvme0c0n1.out‚165598.7021484375# memory_percent‚82.0702075958252 cpu‚25.58316549420457 disk.dm-0.in‚ 61123.1015625 network.sent‚ 3.0530452e+07 disk.dm-1.out‚ 1547.2265625 disk.sda1.in‚0 disk.drbd2165.out‚0# disk.nvme0c0n1.in‚ 61123.1015625( proc.memory.availableMB‚ 2937.6171875 disk.sda.out‚ 1547.2265625 disk.sda1.out‚0 network.recv‚ 6.512328e+06& disk.rbd0.in‚1.759218604441423e+13 disk.drbd2165.in‚0! disk.nvme0n1.in‚ 61123.1015625 disk.sda2.in‚0& disk.nvme0n1.out‚165598.7021484375# proc.memory.rssMB‚ 2869.77734375 proc.cpu.threads‚17 disk.sda.in‚ 108.92578125䉦”¦>‚(:‰ å€êÒù¾ïø gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚280.614' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚56.1228 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0lHiÚ§>‚(:Ž å€êÒµ½ú$ disk./.usageGB‚571.3798332214355# disk.dm-0.out‚165614.1884765625 disk.sda2.out‚0 disk.sda.in‚ 108.92578125 disk.sda3.in‚ 108.92578125 disk.sda2.in‚0) proc.memory.percent‚17.51573085784912 disk.sda1.out‚0 disk.sda3.out‚ 1547.66796875 network.sent‚ 3.0593736e+07 disk.drbd2165.out‚0# disk.nvme0c0n1.in‚ 61123.3203125 disk.dm-1.out‚ 1547.66796875 disk.dm-0.in‚ 61123.3203125! disk.nvme0n1.in‚ 61123.3203125& disk.nvme0n1.out‚165614.1884765625 disk.rbd1.in‚ 0.53564453125# proc.memory.rssMB‚ 2869.77734375) disk./.usagePercent‚38.34578002710321 disk.sda1.in‚0 disk.drbd2165.in‚0( disk.nvme0c0n1.out‚165614.1884765625 disk.rbd1.out‚ 68.6005859375& disk.rbd0.in‚1.759218604441423e+13 disk.rbd0.out‚0 network.recv‚ 6.516312e+06 disk.dm-1.in‚ 108.92578125( proc.memory.availableMB‚ 2937.0390625 proc.cpu.threads‚17 disk.sda.out‚ 1547.66796875# memory_percent‚82.0737361907959 cpu‚25.5830987612448—W£”¨>‚(:‰ ô€êÒ²Ó¨÷ gpu.0.gpu‚99.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚335.642' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚67.1284 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0“á-©>‚(: ô€êÒÏ¡Óú network.recv‚ 6.519972e+06% disk.nvme0n1.out‚165621.830078125 disk.drbd2165.out‚0 disk.sda.in‚ 108.92578125 disk.sda1.out‚0$ memory_percent‚82.07724094390869 disk.sda.out‚ 1547.83984375" disk.dm-0.out‚165621.830078125 disk.sda1.in‚0 disk.sda3.in‚ 108.92578125) proc.memory.availableMB‚ 2936.46484375 proc.cpu.threads‚17 network.sent‚ 3.0598754e+07 disk.rbd1.out‚ 68.7255859375 disk.rbd0.out‚0 disk.dm-0.in‚ 61123.5234375! disk.nvme0n1.in‚ 61123.5234375 disk.sda2.out‚0 disk.sda3.out‚ 1547.83984375' disk.nvme0c0n1.out‚165621.830078125# proc.memory.rssMB‚ 2869.77734375) proc.memory.percent‚17.51573085784912) disk./.usagePercent‚38.34608672412923 disk.dm-1.out‚ 1547.83984375 disk.sda2.in‚0 disk.drbd2165.in‚0# disk.nvme0c0n1.in‚ 61123.5234375$ disk./.usageGB‚571.3844032287598 disk.rbd1.in‚ 0.53564453125 disk.dm-1.in‚ 108.92578125& disk.rbd0.in‚1.759218604441423e+13 cpu‚25.583098535826693OÆ5Àžª>‚(:“ ƒêÒÀ²¬ø gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚279.55' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚55.910000000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ã¥h«>‚(:Œ ƒêÒ ²•ù) disk.nvme0c0n1.out‚165632.38916015625 disk.sda3.out‚ 1548.234375 network.recv‚ 6.523406e+06) disk./.usagePercent‚38.34670677437716' disk.nvme0n1.out‚165632.38916015625 disk.sda2.in‚0 disk.rbd1.in‚ 0.53564453125) proc.memory.availableMB‚ 2936.43359375" disk.nvme0c0n1.in‚ 61488.390625 disk.nvme0n1.in‚ 61488.390625 disk.sda2.out‚0 disk.dm-1.in‚ 108.92578125 disk.sda3.in‚ 108.92578125 disk.rbd1.out‚ 68.7255859375 disk.sda.out‚ 1548.234375$ disk.dm-0.out‚165632.38916015625 disk.sda1.in‚0 disk.drbd2165.out‚0# proc.memory.rssMB‚ 2869.77734375$ disk./.usageGB‚571.3936424255371 disk.sda.in‚ 108.92578125& disk.rbd0.in‚1.759218604441423e+13 disk.rbd0.out‚0 disk.dm-0.in‚ 61488.390625 disk.sda1.out‚0 cpu‚25.583111192520512 proc.cpu.threads‚17$ memory_percent‚82.07743167877197) proc.memory.percent‚17.51573085784912 network.sent‚ 3.0603873e+07 disk.dm-1.out‚ 1548.234375 disk.drbd2165.in‚0 ½Tsž¬>‚(:“ ’êÒ’±üö gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚279.255' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚55.85099999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0+Ù+Ç­>‚(: ’êÒ÷ô÷ù disk.drbd2165.out‚0 disk.sda2.in‚0 disk.sda.out‚ 1548.45703125 disk.nvme0n1.in‚ 61488.609375# disk.dm-0.out‚165647.1005859375 disk.drbd2165.in‚0( disk.nvme0c0n1.out‚165647.1005859375& disk.nvme0n1.out‚165647.1005859375 network.sent‚ 3.0610315e+07) disk./.usagePercent‚38.34710358605523 disk.sda.in‚ 108.92578125 disk.dm-1.in‚ 108.92578125 disk.sda2.out‚0) proc.memory.percent‚17.51573085784912 disk.rbd0.out‚0 disk.rbd1.in‚ 0.53564453125$ memory_percent‚82.07852840423584 network.recv‚ 6.526859e+06 disk.dm-1.out‚ 1548.45703125 disk.sda1.in‚0 disk.sda3.in‚ 108.92578125" disk.nvme0c0n1.in‚ 61488.609375 disk.rbd1.out‚ 68.7880859375$ disk./.usageGB‚571.3995552062988 disk.sda3.out‚ 1548.45703125# proc.memory.rssMB‚ 2869.77734375 proc.cpu.threads‚17) proc.memory.availableMB‚ 2936.25390625 cpu‚25.583100229388315& disk.rbd0.in‚1.759218604441423e+13 disk.dm-0.in‚ 61488.609375 disk.sda1.out‚0ŒûÆ4®>/ _wandbruntime‚26243  _runtime‚26243°¤:µÊ¯>Ä & lr-AdamW/pg1‚7.378418587729608e-05 & lr-AdamW/pg2‚7.378418587729608e-05 trainer/global_step‚15849 % _timestamp‚1.7843160588537805e+09 _runtime‚26243.070958753 _step‚1303Î,ÍÏ°>É & lr-AdamW/pg1‚7.378418587729608e-05 & lr-AdamW/pg2‚7.378418587729608e-05 trainer/global_step‚15849 % _timestamp‚1.7843160588537805e+09 _runtime‚26243.070958753 _step‚1303— ©/4±>/ _wandbruntime‚26243  _runtime‚26243m”\?(²>”P _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14885140361153 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 272487700 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 272477721 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 135217705 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚258 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚4553723120865280 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 135217699 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 137260022 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚85542617 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚4570599631319040 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚44884061 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚44883803 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1230227690713088 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14883795760831 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚4570601002882048 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14910652404736 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚40658551 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚4555690350477312 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚4555690324303872 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 135217699 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 137269995 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚19230106275840 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 137269995 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚19230071739392 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚4555690324303872 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 _step‚1304 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚4555690350477312 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1249457762452480 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚4570601002882048 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14910652404736 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚4570599631319040 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 137260022 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 % _timestamp‚1.7843160588549097e+09 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1249458165080064 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 135217705 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995Jý<%"³>™P JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚4568608286654401 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚4570599631319040 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 137260022 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 % _timestamp‚1.7843160588549097e+09 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1249458165080064 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 135217705 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14885140361153 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 272487700 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 272477721 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 135217705 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚258 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚4553723120865280 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 135217699 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 137260022 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚85542617 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚4570599631319040 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚44884061 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚44883803 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1230227690713088 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14883795760831 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚4570601002882048 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14910652404736 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚40658551 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 ^IDeviceStatsMonitor.on_train_batch_stS& ýart/active_bytes.large_pool.allocated‚4555690350477312 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚4555690324303872 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 135217699 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 137269995 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚19230106275840 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 137269995 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚19230071739392 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚4555690324303872 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 _step‚1304 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚4555690350477312 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1249457762452480 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚4570601002882048 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14910652404736 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 B/ _wandbruntime‚26244  _runtime‚26244{~Dž$'µ>žN XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚4570879928654848 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 135225911 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚4555968351207424 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚4570879928654848 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1249536443261952 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚19231300670464 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚243 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1249536845882368 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 272504398 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚40661094 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 135225911 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 135225905 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14884719227077 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 137278487 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚85547617 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 135225905 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14910232050688 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 137278487 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 272504398 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚44886766 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚4568887091898935 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚4555968325033984 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14911577447424 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14886063827511 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚4555968351207424 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 137268500 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚26244.460147172 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 272494405 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 137268500 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1230305545211904 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _step‚1305 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14911577447424 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 272494405 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1230305177120768 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚15849 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚4554001002643456 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7843160602426505e+09 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0™K±l)'¶>£N OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 135225911 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚4555968351207424 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚4570879928654848 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1249536443261952 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚19231300670464 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚243 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1249536845882368 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 272504398 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚40661094 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 135225911 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 135225905 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14884719227077 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 137278487 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚85547617 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 135225905 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14910232050688 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 137278487 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 272504398 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚44886766 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚4568887091898935 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚4555968325033984 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14911577447424 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14886063827511 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚4555968351207424 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 137268500 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚26244.460147172 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 272494405 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 137268500 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1230305545211904 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _step‚1305 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14911577447424 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 272494405 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1230305177120768 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚15849 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚4554001002643456 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7843160602426505e+09 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚4570879928654848 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0™ ™ 4·>/ _wandbruntime‚26244  _runtime‚26244LNû$¸>ˆ 0loss/FRt1/assignment_loss‚0.8733660816697302 1loss/FRt2/assignment_loss‚0.09009781927642427 .loss/FRt2/detection_loss‚1.134304864282279 0loss/SRqqt1/detection_loss‚0.498681898384342 _step‚1306 2loss/SRqqt1/assignment_loss‚0.1136735251977672 1loss/SRqqt2/assignment_loss‚0.111340494981638 1loss/SRqqt2/detection_loss‚0.4441491441056844 0loss/FBt2/detection_loss‚0.10033401697885282 % _timestamp‚1.7843160603274503e+09 _runtime‚26244.544372399 3loss/FBt2/assignment_loss‚0.0007049007674213095 /loss/FBt1/detection_loss‚0.3208159750201152 %loss/total_loss‚4.094554240526831 3loss/FBt1/assignment_loss‚3.870635355889879e-05 /loss/FRt1/detection_loss‚0.4070468135090184 epoch‚39 trainer/global_step‚15849|å)¹> 3loss/FBt2/assignment_loss‚0.0007049007674213095 /loss/FBt1/detection_loss‚0.3208159750201152 %loss/total_loss‚4.094554240526831 3loss/FBt1/assignment_loss‚3.870635355889879e-05 /loss/FRt1/detection_loss‚0.4070468135090184 epoch‚39 trainer/global_step‚15849 0loss/FRt1/assignment_loss‚0.8733660816697302 1loss/FRt2/assignment_loss‚0.09009781927642427 .loss/FRt2/detection_loss‚1.134304864282279 0loss/SRqqt1/detection_loss‚0.498681898384342 _step‚1306 2loss/SRqqt1/assignment_loss‚0.1136735251977672 1loss/SRqqt2/assignment_loss‚0.111340494981638 1loss/SRqqt2/detection_loss‚0.4441491441056844 0loss/FBt2/detection_loss‚0.10033401697885282 % _timestamp‚1.7843160603274503e+09 _runtime‚26244.544372399š $b鞺>‚(:“ ¡êÒµ’òö gpu.0.gpu‚80.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚336.465' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚67.29299999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ëÊŠº »>‚(:ÿ ¡êÒÁÂØù disk.rbd1.in‚ 0.53564453125 disk.dm-0.in‚ 61488.8125! disk.dm-0.out‚165660.41015625& disk.rbd0.in‚1.759218604441423e+13# proc.memory.rssMB‚ 2869.77734375) proc.memory.percent‚17.51573085784912 proc.cpu.threads‚17$ disk./.usageGB‚571.4061851501465 disk.sda3.out‚ 1548.6953125& disk.nvme0c0n1.out‚165660.41015625 disk.nvme0n1.in‚ 61488.8125 disk.sda1.in‚0( proc.memory.availableMB‚ 2936.3671875 cpu‚25.583129400546984 network.sent‚ 3.0614042e+07) disk./.usagePercent‚38.34754852714974 disk.dm-1.out‚ 1548.6953125 disk.rbd0.out‚0 disk.dm-1.in‚ 108.92578125 disk.drbd2165.in‚0$ disk.nvme0n1.out‚165660.41015625 disk.sda2.in‚0 disk.sda.in‚ 108.92578125 disk.sda.out‚ 1548.6953125$ memory_percent‚82.07783699035645 disk.drbd2165.out‚0 disk.nvme0c0n1.in‚ 61488.8125 disk.sda2.out‚0 disk.rbd1.out‚ 68.7880859375 disk.sda1.out‚0 network.recv‚ 6.530029e+06 disk.sda3.in‚ 108.92578125šøÎèž¼>‚(:“ °êÒÒÄŒ÷ gpu.0.gpu‚90.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚294.335' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚58.86699999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0º ý½>‚(:Œ °êÒŒ½Øù disk.rbd1.in‚ 0.53564453125 disk.rbd0.out‚0 disk.sda1.in‚0 disk.drbd2165.in‚0) disk./.usagePercent‚38.34800550059836( disk.nvme0c0n1.out‚165674.9150390625 disk.nvme0n1.in‚ 61489.234375 disk.sda.out‚ 1548.86328125 disk.sda3.out‚ 1548.86328125$ disk./.usageGB‚571.4129943847656 disk.sda2.out‚0 disk.rbd1.out‚ 68.8505859375 disk.sda.in‚ 108.92578125& disk.rbd0.in‚1.759218604441423e+13 disk.sda1.out‚0 cpu‚25.583101021807174" disk.nvme0c0n1.in‚ 61489.234375 disk.dm-0.in‚ 61489.234375# memory_percent‚82.0775032043457) proc.memory.percent‚17.51573085784912 network.recv‚ 6.534625e+06& disk.nvme0n1.out‚165674.9150390625 disk.sda2.in‚0# disk.dm-0.out‚165674.9150390625 disk.sda3.in‚ 108.92578125 disk.drbd2165.out‚0# proc.memory.rssMB‚ 2869.77734375 proc.cpu.threads‚17 network.sent‚ 3.0677334e+07 disk.dm-1.in‚ 108.92578125 disk.dm-1.out‚ 1548.86328125' proc.memory.availableMB‚ 2936.421875Õ2h¯”¾>‚(:‰ ¿êÒšõõö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚291.971' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚58.3942 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0N O^¿>‚(:“ ¿êÒ¡òÈù network.sent‚ 3.0682395e+07 disk.sda2.out‚0 disk.dm-1.in‚ 108.92578125# disk.dm-0.out‚165686.2880859375$ memory_percent‚82.07759857177734 disk.sda3.out‚ 1549.07421875( disk.nvme0c0n1.out‚165686.2880859375 disk.dm-1.out‚ 1549.07421875 network.recv‚ 6.538248e+06 disk.rbd1.out‚ 68.8505859375 disk.sda1.in‚0# proc.memory.rssMB‚ 2869.77734375$ disk./.usageGB‚571.4175148010254& disk.rbd0.in‚1.759218604441423e+13 disk.dm-0.in‚61548.75390625 cpu‚25.583082370984187& disk.nvme0n1.out‚165686.2880859375 disk.sda3.in‚ 108.92578125& proc.memory.availableMB‚ 2936.40625 proc.cpu.threads‚17 disk.sda2.in‚0 disk.sda.out‚ 1549.07421875 disk.rbd0.out‚0 disk.drbd2165.out‚0* disk./.usagePercent‚38.348308869526434" disk.nvme0n1.in‚61548.75390625 disk.rbd1.in‚ 0.53564453125 disk.sda.in‚ 108.92578125) proc.memory.percent‚17.51573085784912 disk.sda1.out‚0 disk.drbd2165.in‚0$ disk.nvme0c0n1.in‚61548.75390625}^†ç À>‚(:• ÎêÒ÷°Ÿ÷ gpu.0.gpu‚100.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚244.564' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚48.912800000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‹ö~kÁ>‚(: ÎêÒÄÙýø) proc.memory.percent‚17.51573085784912 disk.sda.in‚ 108.92578125 disk.dm-1.out‚ 1549.546875 disk.sda1.in‚0 proc.cpu.threads‚17 disk.dm-1.in‚ 108.92578125& disk.rbd0.in‚1.759218604441423e+13 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.sda2.in‚0 disk.sda1.out‚0 network.sent‚ 3.0687556e+07 disk.rbd0.out‚0# disk.dm-0.out‚165694.1240234375$ memory_percent‚82.07898139953613# proc.memory.rssMB‚ 2869.77734375* disk./.usagePercent‚38.3488¹RË82070395604$ disk./.usageGB‚571.4260559082031 disk.rbd1.out‚ 68.9130859375" disk.nvme0n1.in‚61708.85546875 disk.sda2.out‚0 disk.rbd1.in‚ 0.53564453125 disk.sda3.out‚ 1549.546875 cpu‚25.5830267117182 disk.sda3.in‚ 108.92578125$ disk.nvme0c0n1.in‚61708.85546875( disk.nvme0c0n1.out‚165694.1240234375& disk.nvme0n1.out‚165694.1240234375 network.recv‚ 6.541819e+06 disk.sda.out‚ 1549.546875 disk.dm-0.in‚61708.85546875( proc.memory.availableMB‚ 2936.1796875 /wžÂ>‚(:“ ÝêÒÑýøö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚255.638' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚51.12760000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0hâ Ã>‚(:þ ÝêÒÓ¹•ù disk.sda1.in‚0 disk.drbd2165.in‚0' disk.nvme0c0n1.out‚165711.576171875 disk.sda2.out‚0 network.recv‚ 6.545124e+06 disk.sda1.out‚0 disk.rbd1.in‚ 0.53564453125 disk.rbd1.out‚ 68.9130859375 disk.dm-1.in‚ 108.92578125 disk.dm-0.in‚61709.27734375 disk.sda3.out‚ 1550.171875$ disk.nvme0c0n1.in‚61709.27734375 network.sent‚ 3.0693964e+07 disk.sda3.in‚ 108.92578125% disk.nvme0n1.out‚165711.576171875# proc.memory.rssMB‚ 2869.77734375 cpu‚25.582988528331903 disk.drbd2165.out‚0 proc.memory.availableMB‚2936) proc.memory.percent‚17.51573085784912 disk.sda2.in‚0 memory_percent‚ 82.080078125) disk./.usagePercent‚38.34922332843874$ disk./.usageGB‚571.4311408996582 disk.sda.in‚ 108.92578125 disk.sda.out‚ 1550.171875 disk.dm-1.out‚ 1550.171875& disk.rbd0.in‚1.759218604441423e+13" disk.nvme0n1.in‚61709.27734375 disk.rbd0.out‚0" disk.dm-0.out‚165711.576171875 proc.cpu.threads‚17Éà|¬4Ä>/ _wandbruntime‚26323  _runtime‚26323TtPßÆÅ>À % lr-AdamW/pg1‚7.20815013960898e-05 % lr-AdamW/pg2‚7.20815013960898e-05 trainer/global_step‚15899 # _timestamp‚1.78431613895176e+09 _runtime‚26323.172046203 _step‚1307‚Ö™ÅËÆ>Å % lr-AdamW/pg1‚7.20815013960898e-05 % lr-AdamW/pg2‚7.20815013960898e-05 trainer/global_step‚15899 # _timestamp‚1.78431613895176e+09 _runtime‚26323.172046203 _step‚1307› =òN)4Ç>/ _wandbruntime‚26323  _runtime‚26323¾gý(È>”P G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 _runtime‚26323.176539014 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 135628005 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 137685774 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 % _timestamp‚1.7843161389592066e+09 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚4569590386982912 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 135628005 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚4582548612676309 trainer/global_step‚15899 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 135627999 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚45021206 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 273323752 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 137695747 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1234102379188224 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚40785706 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 135627999 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚4584545983755264 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14955622945792 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚4584547355318272 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚4567617209774080 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚19289835945984 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚32439296 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 137685774 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚4569590386982912 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚4569590360809472 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚85807159 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚85806907 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 400530432 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚252 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14930032873939 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14956968335360 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚45021453 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290ý ¡Ð(É>™P BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚40785706 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 135627999 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚4584545983755264 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14955622945792 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚4584547355318272 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚4567617209774080 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚19289835945984 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚32439296 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 137685774 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚4569590386982912 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚4569590360809472 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚85807159 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚85806907 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 400530432 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚252 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14930032873939 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14956968335360 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚45021453 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 _runtime‚26323.176539014 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 135628005 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 137685774 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 % _timestamp‚1.7843161389592066e+09 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚4569590386982912 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 135628005 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚4582548612676309 trainer/global_step‚15899 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 135627999 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚45021206 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 273323752 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 137695747 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1234102379188224œ FS¨m4Ê>/ _wandbruntime‚26324  _runtime‚263244µQT£%Ë>žN ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 273340450 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14956547981312 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 137704239 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚245 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 137694252 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚45024163 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1234179497504768 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 137704239 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚4584824909520896 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚15899 _runtime‚26324.561269347 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚4569868387713024 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚4582827417920843 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚4584826281091072 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚40788244 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚4567895116980224 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 _step‚1309 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 135636205 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 273330457 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚4569868361539584 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 135636211 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚4569868361539584 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚240 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1253470500912128 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚4567895091552256 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 137694252 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚4584826281091072 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚85812167 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚85812412 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1234179865595904 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚45023923 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚19291003407360 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14932300940619 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 273330457 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 135636205 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚40788249 L>DeviceStatsMonitor.on_train_batch_end/allocation.alT1Ol.allocated‚ 273340450 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14930956340185 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14957893378048 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504¸ö„’)'Ì>£N L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 273340450 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14930956340185 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14957893378048 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 273340450 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14956547981312 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 137704239 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚245 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 137694252 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚45024163 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1234179497504768 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 137704239 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚4584824909520896 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚15899 _runtime‚26324.561269347 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚4569868387713024 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚4582827417920843 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚4584826281091072 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚40788244 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚4567895116980224 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 _step‚1309 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 135636205 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 273330457 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚4569868361539584 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 135636211 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚4569868361539584 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚240 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1253470500912128 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚4567895091552256 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 137694252 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚4584826281091072 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚85812167 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚85812412 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1234179865595904 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚45023923 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚19291003407360 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14932300940619 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 273330457 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 135636205 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚40788249 |ƒƒv4Í>/ _wandbruntime‚26324  _runtime‚26324U¢Î> 1loss/SRqqt2/detection_loss‚0.4273990354865774 0loss/FBt1/detection_loss‚0.15902670204513925 0loss/FBt2/detection_loss‚0.15622203042847518 %loss/total_loss‚4.048967877386875 trainer/global_step‚15899 _step‚1310 2loss/SRqqt2/assignment_loss‚0.1394009428185828 3loss/FBt2/assignment_loss‚3.634145463572559e-07 % _timestamp‚1.7843161404303367e+09 _runtime‚26324.647271413 0loss/FRt1/assignment_loss‚0.8812850065951078 1loss/FRt2/assignment_loss‚0.09929705269925006 3loss/SRqqt1/assignment_loss‚0.06259199507621804 0loss/FRt1/detection_loss‚0.42051346719016724 2loss/SRqqt1/detection_loss‚0.45068318388795564 epoch‚39 3loss/FBt1/assignment_loss‚0.0013950414722785354 /loss/FRt2/detection_loss‚1.2511530562725777‹ª1Ï>” 0loss/FBt2/detection_loss‚0.15622203042847518 %loss/total_loss‚4.048967877386875 trainer/global_step‚15899 _step‚1310 2loss/SRqqt2/assignment_loss‚0.1394009428185828 3loss/FBt2/assignment_loss‚3.634145463572559e-07 % _timestamp‚1.7843161404303367e+09 _runtime‚26324.647271413 0loss/FRt1/assignment_loss‚0.8812850065951078 1loss/FRt2/assignment_loss‚0.09929705269925006 3loss/SRqqt1/assignment_loss‚0.06259199507621804 0loss/FRt1/detection_loss‚0.42051346719016724 2loss/SRqqt1/detection_loss‚0.45068318388795564 epoch‚39 3loss/FBt1/assignment_loss‚0.0013950414722785354 /loss/FRt2/detection_loss‚1.2511530562725777 1loss/SRqqt2/detection_loss‚0.4273990354865774 0loss/FBt1/detection_loss‚0.15902670204513925ž Í’wežÐ>‚(:“ ìêÒ¯¢õö gpu.0.gpu‚40.0 gpu.0.memory‚13, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚339.809' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚67.96180000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0MRBÖÑ>‚(:” ìêÒ͇Òù cpu‚25.582940320006763) disk./.usagePercent‚38.34967569375174 disk.sda1.in‚0 disk.drbd2165.in‚0& disk.nvme0n1.out‚165719.3349609375& disk.rbd0.in‚1.759218604441423e+13 disk.sda.out‚ 1552.01953125# disk.dm-0.out‚165719.3349609375 disk.sda3.in‚ 108.92578125# proc.memory.rssMB‚ 2869.77734375 disk.dm-1.out‚ 1552.01953125( disk.nvme0c0n1.out‚165719.3349609375) proc.memory.percent‚17.51573085784912 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd1.out‚ 68.9755859375 disk.dm-0.in‚61709.49609375$ memory_percent‚82.07237720489502) proc.memory.availableMB‚ 2937.26171875 network.recv‚ 6.549017e+06$ disk./.usageGB‚571.4378814697266 disk.sda3.out‚ 1552.01953125$ disk.nvme0c0n1.in‚61709.49609375" disk.nvme0n1.in‚61709.49609375 network.sent‚ 3.069912e+07 disk.rbd1.in‚ 0.53564453125 disk.sda.in‚ 108.92578125 disk.drbd2165.out‚0 disk.rbd0.out‚0 proc.cpu.threads‚17 disk.dm-1.in‚ 108.92578125 disk.sda1.out‚0Ûë&ŸÒ>‚(:” ûêÒ–™çö gpu.0.gpu‚90.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚280.116' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚56.023199999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0&X–á"Ó>‚(:— ûêÒ¶»ù) disk./.usagePercent‚38.35015365981817 disk.sda2.out‚0 disk.rbd1.out‚ 68.9755859375 network.sent‚ 3.0761356e+07 disk.sda.in‚ 108.92578125 disk.drbd2165.out‚0 disk.sda2.in‚0" disk.nvme0n1.in‚61709.91796875 disk.rbd0.out‚0 disk.sda.out‚ 1552.47265625 disk.dm-0.in‚61709.91796875$ memory_percent‚82.07087516784668 proc.cpu.threads‚17& disk.rbd0.in‚1.759218604441423e+13$ disk.dm-0.out‚165728.75732421875 disk.sda3.in‚ 108.92578125 disk.drbd2165.in‚0$ disk.nvme0c0n1.in‚61709.91796875' disk.nvme0n1.out‚165728.75732421875 network.recv‚ 6.553367e+06$ disk./.usageGB‚571.4450035095215 disk.dm-1.in‚ 108.92578125 disk.dm-1.out‚ 1552.47265625# proc.memory.rssMB‚ 2869.77734375) proc.memory.percent‚17.51573085784912 cpu‚25.582884007839503 disk.sda1.out‚0 disk.sda3.out‚ 1552.47265625) disk.nvme0c0n1.out‚165728.75732421875 disk.rbd1.in‚ 0.53564453125 disk.sda1.in‚0( proc.memory.availableMB‚ 2937.5078125Ö£z5ŸÔ>‚(:” Š‚êÒ™Žæö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚299.744' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚59.948800000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 E]°Õ>‚(:ü Š‚êÒäѤù cpu‚25.582875144766543 disk.drbd2165.in‚0 disk.dm-0.in‚62021.98828125 disk.sda1.out‚0 disk.sda3.in‚ 108.92578125 disk.rbd1.out‚ 69.1005859375 disk.dm-1.out‚1553 disk.rbd0.out‚0$ memory_percent‚82.07082748413086 network.sent‚ 3.0767706e+07) disk.nvme0c0n1.out‚165740.65283203125 disk.sda2.in‚0 disk.rbd1.in‚ 0.53564453125 disk.sda.out‚1553 disk.sda3.out‚1553 disk.dm-1.in‚ 108.92578125# proc.memory.rssMB‚ 2869.77734375 proc.cpu.threads‚17$ disk.nvme0c0n1.in‚62021.98828125 disk.sda2.out‚0$ disk.dm-0.out‚165740.65283203125) proc.memory.percent‚17.51573085784912* disk./.usagePercent‚38.350446532437346 disk.sda1.in‚0 disk.drbd2165.out‚0' disk.nvme0n1.out‚165740.65283203125' proc.memory.availableMB‚ 2937.515625 network.recv‚ 6.556829e+06 disk.sda.in‚ 108.92578125$ disk./.usageGB‚571.4493675231934" disk.nvme0n1.in‚62021.98828125& disk.rbd0.in‚1.759218604441423e+13mHÊ”Ö>‚(:‰ ™‚êÒ±³÷ö gpu.0.gpu‚81.0 gpu.0.memory‚25, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚338.056' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚67.6112 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ùãíÞ×>‚(: ™‚êÒœŸáù( proc.memory.availableMB‚ 2937.2421875 disk.dm-1.in‚ 108.92578125 disk.dm-1.out‚ 1553.3515625 disk.dm-0.in‚62022.20703125 disk.sda3.out‚ 1553.3515625 disk.drbd2165.out‚0 disk.sda2.out‚0 cpu‚25.582874476555467 disk.drbd2165.in‚0" disk.nvme0n1.in‚62022.20703125 network.sent‚ 3.077276e+07 disk.rbd1.out‚ 69.1005859375 disk.sda.in‚ 108.92578125( disk.nvme0c0n1.out‚165751.2470703125 disk.sda2.in‚0 proc.cpu.threads‚17$ disk./.usageGB‚571.4562377929688$ memory_percent‚82.07249641418457# proc.memory.rssMB‚ 2869.77734375) proc.memory.percent‚17.51573085784912& disk.rbd0.in‚1.759218604441423e+13# disk.dm-0.out‚165751.2470703125) disk./.usagePercent‚38.35090760200651 disk.rbd1.in‚ 0.53564453125 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda3.in‚ 108.92578125$ disk.nvme0c0n1.in‚62022.20703125 network.recv‚ 6.560079e+06 disk.sda.out‚ 1553.3515625 disk.rbd0.out‚0& disk.nvme0n1.out‚165751.2470703125#°‰.žØ>‚(:“ ¨‚êÒÖ¢÷ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚264.23' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚52.846000000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0íþ‚#Ù>‚(:˜ ¨‚êÒ—Öìú disk.dm-1.in‚ 108.92578125 network.sent‚ 3.0776519e+07) disk./.usagePercent‚38.35130287763938 disk.drbd2165.out‚0) disk.nvme0c0n1.out‚165763.06005859375 disk.sda.in‚ 108.92578125 disk.dm-0.in‚62022.62890625$ disk.nvme0c0n1.in‚62022.62890625# proc.memory.rssMB‚ 2869.77734375 disk.rbd1.in‚ 0.53564453125 disk.sda.out‚ 1553.49609375 disk.sda1.in‚0) proc.memory.percent‚17.51573085784912 proc.cpu.threads‚17 disk.sda3.out‚ 1553.49609375 disk.drbd2165.in‚0' disk.nvme0n1.out‚165763.06005859375 disk.sda2.out‚0 disk.dm-1.out‚ 1553.49609375& disk.rbd0.in‚1.759218604441423e+13 disk.rbd0.out‚0 disk.sda1.out‚0$ memory_percent‚82.06984996795654 cpu‚25.582844169687277 network.recv‚ 6.563753e+06 disk.sda3.in‚ 108.92578125" disk.nvme0n1.in‚62022.62890625 disk.sda2.in‚0) proc.memory.availableMB‚ 2937.67578125$ disk./.usageGB‚571.4621276855469 disk.rbd1.out‚ 69.1630859375$ disk.dm-0.out‚165763.06005859375å“÷”Ú>‚(:‰ ·‚êÒŠ•ùö gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚303.759' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚60.7518 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 θÛ>‚(:Š ·‚êÒ¥ÆÚù& disk.rbd0.in‚1.759218604441423e+13" disk.dm-0.out‚165771.146484375 disk.rbd1.in‚ 0.53564453125 disk.rbd1.out‚ 69.1630859375$ memory_percent‚82.07104206085205# proc.memory.rssMB‚ 2869.77734375 proc.cpu.threads‚17 network.sent‚ 3.0781551e+07' disk.nvme0c0n1.out‚165771.146484375) proc.memory.availableMB‚ 2937.48046875 disk.sda.in‚ 108.92578125 disk.dm-0.in‚62022.83203125 disk.sda1.in‚0 disk.drbd2165.in‚0$ disk.nvme0c0n1.in‚62022.83203125* disk./.usagePercent‚38.351609574665396 disk.dm-1.in‚ 108.92578125 disk.sda1.out‚0 network.recv‚ 6.566858e+06 disk.sda.out‚ 1553.78125 disk.sda3.out‚ 1553.78125" disk.nvme0n1.in‚62022.83203125 disk.sda2.out‚0 cpu‚25.582799374188376$ disk./.usageGB‚571.4666976928711 disk.dm-1.out‚ 1553.78125 disk.rbd0.out‚0 disk.sda3.in‚ 108.92578125 disk.drbd2165.out‚0 disk.sda2.in‚0% disk.nvme0n1.out‚165771.146484375) proc.memory.percent‚17.51573085784912ÆGZ14Ü>/ _wandbruntime‚26403  _runtime‚26403DÿsÊÝ>Ä _step‚1311 & lr-AdamW/pg1‚7.039640532136434e-05 & lr-AdamW/pg2‚7.039640532136434e-05 trainer/global_step‚15949 % _timestamp‚1.7843162192478201e+09 _runtime‚26403.465110514zËÏÞ>É & lr-AdamW/pg1‚7.039640532136434e-05 & lr-AdamW/pg2‚7.039640532136434e-05 trainer/global_step‚15949 % _timestamp‚1.7843162192478201e+09 _runtime‚26403.465110514 _step‚1311Ÿ 2Uh´4ß>/ _wandbruntime‚26403  _runtime‚26403¯ÏŸ‰(à>”P M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 138111526 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚45158182 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚4598493707754496 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚4598492336191488 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 136038305 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15001938876416 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚86071033 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚4581511324110848 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 136038299 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15003284265984 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚4581511298682880 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚4583490397315072 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚40912851 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚251 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚19349546400768 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14976269987047 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 136038305 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 138121499 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 _step‚1312 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1257326280509440 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚4583490397315072 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚86071289 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973¡€*|úá>™P M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 138111526 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚45158182 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚4598493707754496 JDeviceStatsMonitor.on_train_batch²óÅ%$_start/active_bytes.all.freed‚4598492336191488 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 136038305 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15001938876416 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚86071033 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚4581511324110848 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 136038299 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15003284265984 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚4581511298682880 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚4583490397315072 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚40912851 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚251 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚19349546400768 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14976269987047 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 136038305 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 138121499 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 _step‚1312 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1257326280509440 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚4583490397315072 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚86071289 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973  Dö’Î4â>/ _wandbruntime‚26404  _runtime‚26404Dlò"'ã>œN RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚4596766373914349 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚4581789180461056 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚250 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 274176502 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚45161190 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1238053817888768 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 138120004 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14977193453293 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 136046505 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚4598772633527296 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚45160945 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 274166509 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚4598772633527296 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1238054185979904 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚4596767743942751 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚86076339 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚4583768424218624 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 136046505 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 138120004 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚4598771261957120 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15004209308672 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚4583768424218624 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 $ _timestamp‚1.784316220640432e+09 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 138129991 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14978538053727 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚4581789205889024 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 274176502 Q¡N _runtime‚26404.85784357 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚15002863911936 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚19350777495552 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚4583768398045184 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 138129991 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚4583768398045184 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 136046505 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 138120004 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚4598771261957120 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15004209308672 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚4583768424218624 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 $ _timestamp‚1.784316220640432e+09 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 138129991 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14978538053727 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚4581789205889024 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 274176502 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 274176502 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚45161190 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1238053817888768 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 138120004 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14977193453293 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 136046505 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚4598772633527296 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚45160945 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 274166509 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚4598772633527296 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1238054185979904 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚4596767743942751 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚86076339 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚4583768424218624 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F/ _wandbruntime‚26404  _runtime‚26404|DÑÌæ>’ epoch‚39 _runtime‚26404.941079734 4loss/FBt2/assignment_loss‚3.5858950621505806e-07 3loss/FBt1/assignment_loss‚2.622596821784858e-07 2loss/SRqqt1/detection_loss‚0.46530500849097056 /loss/FBt1/detection_loss‚0.3013633247397138 trainer/global_step‚15949 % _timestamp‚1.7843162207241573e+09 0loss/FRt1/assignment_loss‚0.8885795467466281 1loss/FRt2/assignment_loss‚0.13274442355836455 3loss/SRqqt1/assignment_loss‚0.05390618353687203 0loss/FRt1/detection_loss‚0.41093185478874183 /loss/FRt2/detection_loss‚1.2339020399967602 _step‚1314 3loss/SRqqt2/assignment_loss‚0.12875822225026456 2loss/SRqqt2/detection_loss‚0.44066376247053257 0loss/FBt2/detection_loss‚0.09991516429213274 &loss/total_loss‚4.1560701517201695V̦ç>— _runtime‚26404.941079734 4loss/FBt2/assignment_loss‚3.5858950621505806e-07 3loss/FBt1/assignment_loss‚2.622596821784858e-07 2loss/SRqqt1/detection_loss‚0.46530500849097056 /loss/FBt1/detection_loss‚0.3013633247397138 trainer/global_step‚15949 % _timestamp‚1.7843162207241573e+09 0loss/FRt1/assignment_loss‚0.8885795467466281 1loss/FRt2/assignment_loss‚0.13274442355836455 3loss/SRqqt1/assignment_loss‚0.05390618353687203 0loss/FRt1/detection_loss‚0.41093185478874183 /loss/FRt2/detection_loss‚1.2339020399967602 _step‚1314 3loss/SRqqt2/assignment_loss‚0.12875822225026456 2loss/SRqqt2/detection_loss‚0.44066376247053257 0loss/FBt2/detection_loss‚0.09991516429213274 &loss/total_loss‚4.1560701517201695 epoch‚39¢ ËÒ“è>‚(:ˆ Æ‚êÒë¢Ý÷ gpu.0.gpu‚100.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚345.59' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚69.118 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‹!cé>‚(: Æ‚êÒýøÈû disk.dm-0.in‚62135.14453125" disk.nvme0n1.in‚62135.14453125 cpu‚25.582758184279083$ memory_percent‚82.07123279571533 network.recv‚ 6.570501e+06 disk.sda.in‚ 108.92578125 disk.sda1.out‚0 disk.rbd1.out‚ 69.2255859375& disk.rbd0.in‚1.759218604441423e+13 disk.sda1.in‚0( disk.nvme0c0n1.out‚165779.4755859375 disk.sda2.in‚0 network.sent‚ 3.078672e+07# disk./.usageGB‚571.476016998291$ disk.nvme0c0n1.in‚62135.14453125 disk.sda3.in‚ 108.92578125 disk.sda3.out‚ 1554.015625) proc.memory.percent‚17.51573085784912) disk./.usagePercent‚38.35223500107154 disk.dm-1.in‚ 108.92578125# disk.dm-0.out‚165779.4755859375 disk.drbd2165.out‚0& disk.nvme0n1.out‚165779.4755859375# proc.memory.rssMB‚ 2869.77734375 disk.rbd1.in‚ 0.53564453125 disk.sda.out‚ 1554.015625 disk.dm-1.out‚ 1554.015625 disk.rbd0.out‚0 disk.drbd2165.in‚0 disk.sda2.out‚0) proc.memory.availableMB‚ 2937.44921875 proc.cpu.threads‚177'.Rê>‚(:‰ Õ‚êÒ­Ýê÷ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚287.327' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.poæ {ZwerPercent‚57.4654 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0»ÜA ë>‚(:• Õ‚êÒ¶‘µø disk.sda.in‚ 108.92578125 disk.dm-0.in‚62135.36328125 disk.sda1.in‚0 disk.sda3.in‚ 108.92578125 disk.sda3.out‚ 1554.3203125# proc.memory.rssMB‚ 2869.77734375 network.recv‚ 6.574811e+06$ disk./.usageGB‚571.4804306030273 disk.dm-1.in‚ 108.92578125 disk.drbd2165.out‚0$ memory_percent‚82.07628726959229 cpu‚25.582755352641488 disk.sda1.out‚0 disk.rbd1.in‚ 0.53564453125 disk.sda.out‚ 1554.3203125 disk.dm-1.out‚ 1554.3203125 disk.rbd0.out‚0" disk.nvme0n1.in‚62135.36328125 proc.cpu.threads‚17) disk./.usagePercent‚38.35253120178866) proc.memory.availableMB‚ 2936.62109375 disk.sda2.out‚0& disk.rbd0.in‚1.759218604441423e+13 disk.drbd2165.in‚0) disk.nvme0c0n1.out‚165796.35107421875) proc.memory.percent‚17.51573085784912$ disk.dm-0.out‚165796.35107421875$ disk.nvme0c0n1.in‚62135.36328125' disk.nvme0n1.out‚165796.35107421875 network.sent‚ 3.0850053e+07 disk.sda2.in‚0 disk.rbd1.out‚ 69.2255859375(n ÆŸì>‚(:” ä‚êÒè¡çö gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚38.0 gpu.0.powerWatts‚109.997' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.999399999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ƒ+Eÿ"í>‚(:— ä‚êÒ˜ã³ù disk.drbd2165.in‚0 proc.cpu.threads‚17$ disk./.usageGB‚571.4847412109375 disk.sda2.out‚0$ memory_percent‚82.07809925079346 disk.rbd1.in‚ 0.53564453125 disk.dm-0.in‚62135.56640625 disk.drbd2165.out‚0# proc.memory.rssMB‚ 2869.77734375 network.recv‚ 6.578156e+06) disk.nvme0c0n1.out‚165805.02587890625 disk.sda2.in‚0 disk.dm-1.in‚ 108.92578125 disk.rbd1.out‚ 69.2880859375 disk.sda.in‚ 108.92578125$ disk.dm-0.out‚165805.02587890625 disk.sda1.out‚0) proc.memory.percent‚17.51573085784912 network.sent‚ 3.0856386e+07& disk.rbd0.in‚1.759218604441423e+13) proc.memory.availableMB‚ 2936.32421875 cpu‚25.58284812950208) disk./.usagePercent‚38.35282049030235$ disk.nvme0c0n1.in‚62135.56640625' disk.nvme0n1.out‚165805.02587890625 disk.rbd0.out‚0 disk.sda3.in‚ 108.92578125 disk.sda3.out‚ 1554.55078125" disk.nvme0n1.in‚62135.56640625 disk.sda.out‚ 1554.55078125 disk.dm-1.out‚ 1554.55078125 disk.sda1.in‚0Îÿë>Ÿî>‚(:” ó‚êÒÔ›õö gpu.0.gpu‚99.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚240.541' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚48.108200000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0“Ô%ï>‚(: ó‚êÒ‚›ôø disk.dm-0.in‚62135.98828125 disk.drbd2165.out‚0" disk.nvme0n1.in‚62135.98828125' disk.nvme0n1.out‚165817.14892578125 disk.sda2.in‚0 network.sent‚ 3.0861301e+07* disk./.usagePercent‚38.353335321453706 disk.dm-1.out‚ 1554.75390625 disk.sda.in‚ 108.92578125 disk.sda.out‚ 1554.75390625& disk.rbd0.in‚1.759218604441423e+13$ disk.dm-0.out‚165817.14892578125 disk.sda2.out‚0 disk.sda1.out‚0 disk.rbd1.in‚ 0.53564453125 disk.rbd1.out‚ 69.2880859375" proc.memory.availableMB‚2936.5 network.recv‚ 6.580967e+06$ disk.nvme0c0n1.in‚62135.98828125) proc.memory.percent‚17.51573085784912 disk.sda1.in‚0 disk.sda3.in‚ 108.92578125 disk.drbd2165.in‚0) disk.nvme0c0n1.out‚165817.14892578125 cpu‚25.58292492404367 disk.rbd0.out‚0 disk.sda3.out‚ 1554.75390625$ disk./.usageGB‚571.4924125671387# proc.memory.rssMB‚ 2869.77734375 proc.cpu.threads‚17# memory_percent‚82.0770263671875 disk.dm-1.in‚ 108.92578125 ¢üžð>‚(:“ ‚ƒêÒÞ»ø gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚272.619' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚54.52380000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0%½Åñ>‚(: ‚ƒêÒûˆ˜ù# disk.nvme0c0n1.in‚ 62283.1796875 disk.sda.out‚ 1555.046875 disk.sda3.in‚ 108.92578125) proc.memory.percent‚17.51573085784912 cpu‚25.582981203023895 disk.drbd2165.out‚0 disk.dm-1.out‚ 1555.046875 disk.sda3.out‚ 1555.046875) proc.memory.availableMB‚ 2936.25390625 disk.rbd1.out‚ 69.3505859375 network.recv‚ 6.584219e+06* disk./.usagePercent‚38.353633826238635' disk.nvme0n1.out‚165829.44580078125 disk.rbd1.in‚ 0.53564453125 disk.dm-1.in‚ 108.92578125 disk.sda2.in‚0$ disk./.usageGB‚571.4968605041504 disk.drbd2165.in‚0 disk.sda.in‚ 108.92578125 disk.dm-0.in‚ 62283.1796875$ disk.dm-0.out‚165829.44580078125$ memory_percent‚82.07852840423584# proc.memory.rssMB‚ 2869.77734375 proc.cpu.threads‚17 disk.sda2.out‚0& disk.rbd0.in‚1.759218604441423e+13 disk.rbd0.out‚0 disk.sda1.in‚0 disk.sda1.out‚0 network.sent‚ 3.0865689e+07) disk.nvme0c0n1.out‚165829.44580078125! disk.nvme0n1.in‚ 62283.1796875LH]4ò>/ _wandbruntime‚26483  _runtime‚26483µ¶R­Êó>Ä _runtime‚26483.209131401 _step‚1315 & lr-AdamW/pg1‚6.872900588369158e-05 & lr-AdamW/pg2‚6.872900588369158e-05 trainer/global_step‚15999 % _timestamp‚1.7843162989919584e+09–Ž¸Ïô>É & lr-AdamW/pg1‚6.872900588369158e-05 & lr-AdamW/pg2‚6.872900588369158e-05 trainer/global_step‚15999 % _timestamp‚1.7843162989919584e+09 _runtime‚26483.209131401 _step‚1315£ v˜TF4õ>/ _wandbruntime‚26483  _runtime‚26483ÃaÙð(ö>”P ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚4595405413019648 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 136448605 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚4597390433820672 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7843162989930546e+09 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚260 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1261260284002304 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚15048254807040 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15048254807040 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 138547251 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚86335835 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚45295569 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1241851019956224 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 _runtime‚26483.210520183 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15022507100155 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 trainer/global_step‚15999 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚19409229509632 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚4597390459994112 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚4610429264720125 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15023851700477 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚4612438688627712 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚41040001 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _step‚1316 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚4597390433820672 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 138547251 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15049600196608 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1261259881374720 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 138537278 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚4612440060190720 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚45295829 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚4597390459994112 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15049600196608 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 274995856 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 136448605 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚41040006 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1241850651865088 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚4595405387591680 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚19409264046080 B™P ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚4595405413019648 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 136448605 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚4597390433820672 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7843162989930546e+09 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚260 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1261260284002304 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚15048254807040 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15048254807040 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 138547251 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚86335835 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚45295569 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1241851019956224 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 _runtime‚26483.210520183 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15022507100155 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 trainer/global_step‚15999 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚19409229509632 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚4597390459994112 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚4610429264720125 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15023851700477 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚4612438688627712 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚41040001 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _step‚1316 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚4597390433820672 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 138547251 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15049600196608 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1261259881374720 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 138537278 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚4612440060190720 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚45295829 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚4597390459994112 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15049600196608 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 274995856 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 136448605 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚41040006 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1241850651865088 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚4595405387591680 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚19409264046080 B/ _wandbruntime‚26484  _runtime‚26484£HÖ°>ù>žN ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 275002561 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1241928138272768 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚4595683294797824 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚45298567 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 trainer/global_step‚15999 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚4597668460724224 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1261338556673024 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 138545756 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 136456805 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15050525239296 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚19410452929536 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚86341116 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚45298317 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15050525239296 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚4595683269369856 _runtime‚26484.597055088 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 136456811 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚4612718985963520 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚250 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚4610708069964659 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 136456811 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1261338959293440 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 275012554 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚15049179842560 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚86340861 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚4597668434550784 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 136456805 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 138555743 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 275012554 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚255 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚41042549 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15024775166835 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15049179842560 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 % _timestamp‚1.784316300379473:,æ4e+09 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚4597668434550784 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚41042544 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15023430566401 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F£N E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15023430566401 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 275002561 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1241928138272768 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚4595683294797824 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚45298567 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 trainer/global_step‚15999 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚4597668460724224 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1261338556673024 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 138545756 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 136456805 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15050525239296 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚19410452929536 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚86341116 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚45298317 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15050525239296 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚4595683269369856 _runtime‚26484.597055088 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 136456811 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚4612718985963520 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚250 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚4610708069964659 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 136456811 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1261338959293440 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 275012554 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚15049179842560 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚86340861 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚4597668434550784 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 136456805 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 138555743 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 275012554 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚255 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚41042549 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15024775166835 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15049179842560 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 % _timestamp‚1.7843163003794734e+09 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚4597668434550784 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚41042544 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008¥ ù+€‡4û>/ _wandbruntime‚26484  _runtime‚26484/Cg2ü> _step‚1318 epoch‚39 .loss/FRt1/assignment_loss‚0.91496541797836 4loss/SRqqt1/assignment_loss‚0.054162737898068684 3loss/SRqqt2/assignment_loss‚0.09408850315367605 4loss/FBt1/assignment_loss‚3.2218724985230783e-09 /loss/FRt2/detection_loss‚1.1148465590696792 1loss/FRt2/assignment_loss‚0.08182026790905242 4loss/FBt2/assignment_loss‚0.00014117060122677033 0loss/FBt2/detection_loss‚0.14545044881468663 trainer/global_step‚15999 /loss/FRt1/detection_loss‚0.4517660424438442 /loss/FBt1/detection_loss‚0.1766691010843469 %loss/total_loss‚3.880173043570733 % _timestamp‚1.7843163004667778e+09 2loss/SRqqt1/detection_loss‚0.41106675608691134 1loss/SRqqt2/detection_loss‚0.4351960353090083 _runtime‚26484.683697573¼3Qý>” /loss/FRt1/detection_loss‚0.4517660424438442 /loss/FBt1/detection_loss‚0.1766691010843469 %loss/total_loss‚3.880173043570733 % _timestamp‚1.7843163004667778e+09 2loss/SRqqt1/detection_loss‚0.41106675608691134 1loss/SRqqt2/detection_loss‚0.4351960353090083 _runtime‚26484.683697573 _step‚1318 epoch‚39 .loss/FRt1/assignment_loss‚0.91496541797836 4loss/SRqqt1/assignment_loss‚0.054162737898068684 3loss/SRqqt2/assignment_loss‚0.09408850315367605 4loss/FBt1/assignment_loss‚3.2218724985230783e-09 /loss/FRt2/detection_loss‚1.1148465590696792 1loss/FRt2/assignment_loss‚0.08182026790905242 4loss/FBt2/assignment_loss‚0.00014117060122677033 0loss/FBt2/detection_loss‚0.14545044881468663 trainer/global_step‚15999¦ VGý|Ÿþ>‚(:” ‘ƒêÒ¼»‘÷ gpu.0.gpu‚68.0 gpu.0.memory‚21, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚37.0 gpu.0.powerWatts‚109.588' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.917599999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0<ž%4 ÿ>‚(:‚ ‘ƒêÒ²¿¾ú' disk.nvme0c0n1.out‚165839.392578125! disk.nvme0n1.in‚ 62283.3828125 disk.sda2.out‚0 disk.dm-1.in‚ 108.92578125 network.sent‚ 3.0870763e+07 network.recv‚ 6.587565e+06* disk./.usagePercent‚38.354150705450266$ disk./.usageGB‚571.5045623779297& disk.rbd0.in‚1.759218604441423e+13 disk.rbd0.out‚0" disk.dm-0.out‚165839.392578125 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.rbd1.out‚ 69.3505859375 disk.sda.in‚ 108.92578125# proc.memory.rssMB‚ 2871.09765625 cpu‚25.58338424728844 proc.cpu.threads‚17' proc.memory.availableMB‚ 2934.765625 disk.drbd2165.out‚0 disk.sda1.in‚0 disk.sda3.in‚ 108.92578125% disk.nvme0n1.out‚165839.392578125 disk.sda2.in‚0 disk.rbd1.in‚ 0.53564453125# disk.nvme0c0n1.in‚ 62283.3828125 disk.sda.out‚ 1555.4375 disk.dm-1.out‚ 1555.4375$ memory_percent‚82.08761215209961 disk.sda3.out‚ 1555.4375 disk.dm-0.in‚ 62283.3828125* proc.memory.percent‚17.523789405822754FêÃÀ”€?‚(:‰  ƒêÒ²ï™ø gpu.0.gpu‚48.0 gpu.0.memory‚15, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚327.874' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚65.5748 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¤Á9?‚(:”  ƒêÒ¿¤úø& disk.rbd0.in‚1.759218604441423e+13' disk.nvme0n1.out‚165852.22705078125 cpu‚25.58441450596096 disk.dm-0.in‚ 62283.8046875 disk.sda1.in‚0# proc.memory.rssMB‚ 2865.48828125$ disk.dm-0.out‚165852.22705078125! disk.nvme0n1.in‚ 62283.8046875 disk.sda.in‚ 108.92578125 proc.cpu.threads‚17 network.recv‚ 6.592353e+06 disk.rbd0.out‚0 disk.drbd2165.in‚0# disk.nvme0c0n1.in‚ 62283.8046875 disk.sda2.in‚0 disk.rbd1.in‚ 0.53564453125$ memory_percent‚82.05211162567139) proc.memory.percent‚17.48955249786377) disk./.usagePercent‚38.35467219279751 disk.sda3.in‚ 108.92578125$ disk./.usageGB‚571.5123329162598 disk.sda1.out‚0 disk.sda3.out‚ 1555.77734375 disk.rbd1.out‚ 69.4755859375 network.sent‚ 3.0932808e+07 disk.dm-1.in‚ 108.92578125) disk.nvme0c0n1.out‚165852.22705078125) proc.memory.availableMB‚ 2940.58203125 disk.dm-1.out‚ 1555.77734375 disk.drbd2165.out‚0 disk.sda2.out‚0 disk.sda.out‚ 1555.77734375>žG½4‚?/ _wandbruntime‚26512  _runtime‚26512À©~®6ƒ?¡| 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9898974299430847 6 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.152212455868721 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6544378995895386 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6546354293823242 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.10471903532743454 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.1120699793100357 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 particle/accuracy_4_of_5‚1 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.47186246514320374 epoch‚39 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9910075068473816 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.26851850748062134 8"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.619714617729187 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.13259178400039673 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7381330728530884 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.26851850748062134 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7327769994735718 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6198474168777466 - Purity/0FRt2SRqqt1FBt/FBt_purity‚0.765625 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.1473248451948166 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 +jet/accuracy_3_of_4‚0.04323367029428482 ,jet/accuracy_4_of_4‚0.013899499550461769 _step‚1319 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.12437109649181366 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.21682170033454895 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.02883034385740757 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5922656059265137 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9746650457382202 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.1198415532708168 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6607917547225952 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9170029163360596 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9888888597488403 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3433492183685303 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.8500000238418579 6 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.988606870174408 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6451684236526489 *"Purity/2FRt*SRqqt2FBt/event_purity‚0.5 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.5714285969734192 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.2222222238779068 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.05000000074505806 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.2083333283662796 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9776531457901001 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.21272936463356018 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.02588968351483345 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8128896355628967 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.26923078298568726 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.6552720665931702 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 *jet/accuracy_1_of_4‚0.9589023590087891 particle/accuracy_1_of_4‚1 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7602112293243408 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 particle/accuracy_2_of_5‚1 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8455727696418762 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9782153367996216 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 8"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.728560745716095 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 +particle/sensitivity‚0.5874273180961609 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5114696621894836 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7280035614967346 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.19699430465698242 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0032324804924428463 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.26923078298568726 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.5909671187400818 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9941394329071045 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6058144569396973 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 6validation_average_jet_accuracy‚0.3246648907661438 6 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.132612407207489 +jet/accuracy_1_of_1‚0.28187286853790283 particle/accuracy_2_of_3‚1 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5095833539962769 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8888888955116272 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.8500000238418579 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4528804123401642 8"Purity/0FRt1SRqqt0FBt/event_purity‚0.604714035987854 7"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.64049232006073 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9862642884254456 particle/accuracy_5_of_5‚1 *loss/val_total_loss‚11.961924246833917 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.09554310888051987 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8128896355628967 +particle/specificity‚0.8956698179244995 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.12713034451007843 8"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.604714035987854 - Purity/0FRt2SRqqt*FBt/FBt_purity‚0.765625 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.42500001192092896 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.13238847255706787 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 5 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.98810213804245 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.018288319930434227 particle/accuracy_1_of_2‚1 jet/accuracy_3_of_5‚1 jet/accuracy_4_of_5‚0 particle/accuracy_1_of_5‚1 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9974524974822998 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.03115689381957054 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.18062302470207214 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.02242973819375038 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9047377705574036 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6825201511383057 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7138321995735168 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.023000027984380722 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.25724127888679504 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.13432510197162628 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6918506026268005 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.6875 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9873688220977783 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6701977252960205 6 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.949999988079071 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.23494884371757507 8"Purity/1FRt0SRqqt0FBt/event_purity‚0.132612407207489 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9951704740524292 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6781776547431946 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 % _timestamp‚1.7843163280691268e+09 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7142857313156128 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.7731481790542603 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.7783169746398926 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9916297197341919 & Purity/1FRt1SRqqt2FBt/FRt_purity‚1 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 (particle/accuracy‚0.8294698596000671 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6303220391273499 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9906229376792908 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.46296295523643494 /particle/accuracy_3_of_4‚0.9974937438964844 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6914072632789612 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.8653846383094788 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6337130069732666 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9974524974822998 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.0870402455329895 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 particle/accuracy_3_of_3‚1 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.1012720838189125 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.7138321995735168 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7546296119689941 +jet/accuracy_2_of_2‚0.11661581695079803 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.10738345235586166 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.029207993298768997 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.1327182799577713 particle/accuracy_1_of_3‚1 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3526081144809723 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6258916258811951 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9871701002120972 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.2222222238779068 *jet/accuracy_2_of_3‚0.4716508984565735 jet/accuracy_5_of_5‚0 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.14563557505607605 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7286032438278198 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9865805506706238 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6726912260055542 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.13001202046871185 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6320780515670776 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.17710906267166138 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.5691555738449097 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.005986332893371582 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 *jet/accuracy_1_of_2‚0.7304662466049194 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9928644895553589 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.5840353965759277 8"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.687150239944458 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.09666667133569717 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7463204264640808 *jet/accuracy_2_of_4‚0.6788923740386963 particle/accuracy_1_of_1‚1 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.5572916269302368 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.4861145615577698 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.44523411989212036 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.00958407111465931 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 +jet/accuracy_3_of_3‚0.04206186532974243 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.8657407164573669 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.13680140674114227 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7546296119689941 'particle/f_score‚0.5966808199882507 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 particle/accuracy_2_of_2‚1 trainer/global_step‚15999 _runtime‚26512.287093054 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9798296689987183 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.5254629850387573 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.1865912228822708 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7557624578475952 particle/accuracy_2_of_4‚1 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.030642732977867126 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.19900453090667725 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.02242973819375038 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7398526072502136 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9841936826705933 jet/accuracy_2©œéy_of_5‚1 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.7731481790542603 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9743400812149048 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9936507940292358 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6497675776481628 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.12092570960521698 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.7169312238693237 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 jet/accuracy_1_of_5‚1 6 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.998227596282959 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.1435360610485077 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 /particle/accuracy_4_of_4‚0.9335044622421265 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.05000000074505806 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 6 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.515622615814209 ."Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.53125 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.5644907355308533 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 )jet/accuracy_1_of_3‚0.903435468673706 particle/accuracy_3_of_5‚1 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.14499925076961517 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.22645430266857147 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.7924550175666809 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7238343954086304coš½,>„?¦| =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.09554310888051987 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8128896355628967 +particle/specificity‚0.8956698179244995 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.12713034451007843 8"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.604714035987854 - Purity/0FRt2SRqqt*FBt/FBt_purity‚0.765625 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.42500001192092896 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.13238847255706787 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 5 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.98810213804245 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.018288319930434227 particle/accuracy_1_of_2‚1 jet/accuracy_3_of_5‚1 jet/accuracy_4_of_5‚0 particle/accuracy_1_of_5‚1 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9974524974822998 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.03115689381957054 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.18062302470207214 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.02242973819375038 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9047377705574036 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6825201511383057 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7138321995735168 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.023000027984380722 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.25724127888679504 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.13432510197162628 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6918506026268005 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.6875 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9873688220977783 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6701977252960205 6 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.949999988079071 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.23494884371757507 8"Purity/1FRt0SRqqt0FBt/event_purity‚0.132612407207489 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9951704740524292 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6781776547431946 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 % _timestamp‚1.7843163280691268e+09 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7142857313156128 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.7731481790542603 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.7783169746398926 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9916297197341919 & Purity/1FRt1SRqqt2FBt/FRt_purity‚1 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 (particle/accuracy‚0.8294698596000671 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6303220391273499 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9906229376792908 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.46296295523643494 /particle/accuracy_3_of_4‚0.9974937438964844 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6914072632789612 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.8653846383094788 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6337130069732666 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9974524974822998 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.0870402455329895 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 particle/accuracy_3_of_3‚1 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.1012720838189125 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.7138321995735168 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7546296119689941 +jet/accuracy_2_of_2‚0.11661581695079803 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.10738345235586166 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.029207993298768997 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.1327182799577713 particle/accuracy_1_of_3‚1 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3526081144809723 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6258916258811951 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9871701002120972 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.2222222238779068 *jet/accuracy_2_of_3‚0.4716508984565735 jet/accuracy_5_of_5‚0 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.14563557505607605 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7286032438278198 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9865805506706238 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6726912260055542 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.13001202046871185 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6320780515670776 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.17710906267166138 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.5691555738449097 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.005986332893371582 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 *jet/accuracy_1_of_2‚0.7304662466049194 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9928644895553589 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.5840353965759277 8"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.687150239944458 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.09666667133569717 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7463204264640808 *jet/accuracy_2_of_4‚0.6788923740386963 particle/accuracy_1_of_1‚1 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.5572916269302368 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.4861145615577698 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.44523411989212036 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.00958407111465931 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 +jet/accuracy_3_of_3‚0.04206186532974243 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.8657407164573669 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.13680140674114227 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7546296119689941 'particle/f_score‚0.5966808199882507 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 particle/accuracy_2_of_2‚1 trainer/global_step‚15999 _runtime‚26512.287093054 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9798296689987183 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.5254629850387573 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.1865912228822708 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7557624578475952 particle/accuracy_2_of_4‚1 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.030642732977867126 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.19900453090667725 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.02242973819375038 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7398526072502136 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9841936826705933 jet/accuracy_2_of_5‚1 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.7731481790542603 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9743400812149048 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9936507940292358 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6497675776481628 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.12092570960521698 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.7169312238693237 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 jet/accuracy_1_of_5‚1 6 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.998227596282959 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.1435360610485077 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 /particle/accuracy_4_of_4‚0.9335044622421265 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.05000000074505806 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 6 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.515622615814209 ."Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.53125 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.5644907355308533 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 )jet/accuracy_1_of_3‚0.903435468673706 particle/accuracy_3_of_5‚1 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.14499925076961517 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.22645430266857147 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.7924550175666809 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7238343954086304 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9898974299430847 6 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.152212455868721 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6544378995895386 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6546354293823242 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.10471903532743454 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.1120699793100357 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 particle/accuracy_4_of_5‚1 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.47186246514320374 epoch‚39 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9910075068473816 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.26851850748062134 8"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.619714617729187 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.13259178400039673 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7381330728530884 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.26851850748062134 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7327769994735718 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6198474168777466 - Purity/0FRt2SRqqt1FBt/FBt_purity‚0.765625 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.1473248451948166 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 +jet/accuracy_3_of_4‚0.04323367029428482 ,jet/accuracy_4_of_4‚0.013899499550461769 _step‚1319 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.12437109649181366 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.21682170033454895 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.02883034385740757 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5922656059265137 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9746650457382202 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.1198415532708168 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6607917547225952 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9170029163360596 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9888888597488403 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3433492183685303 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.8500000238418579 6 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.988606870174408 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6451684236526489 *"Purity/2FRt*SRqqt2FBt/event_purity‚0.5 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.5714285969734192 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.2222222238779068 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.05000000074505806 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.2083333283662796 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9776531457901001 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.21272936463356018 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.02588968351483345 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8128896355628967 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.26923078298568726 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.6552720665931702 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 *jet/accuracy_1_of_4‚0.9589023590087891 particle/accuracy_1_of_4‚1 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7602112293243408 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 particle/accuracy_2_of_5‚1 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8455727696418762 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9782153367996216 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 8"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.728560745716095 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 +particle/sensitivity‚0.5874273180961609 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5114696621894836 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7280035614967346 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.19699430465698242 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0032324804924428463 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.26923078298568726 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.5909671187400818 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9941394329071045 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6058144569396973 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 6validation_average_jet_accuracy‚0.3246648907661438 6 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.132612407207489 +jet/accuracy_1_of_1‚0.28187286853790283 particle/accuracy_2_of_3‚1 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5095833539962769 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8888888955116272 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.8500000238418579 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4528804123401642 8"Purity/0FRt1SRqqt0FBt/event_purity‚0.604714035987854 7"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.64049232006073 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9862642884254456 particle/accuracy_5_of_5‚1 *loss/val_total_loss‚11.961924246833917§ å³ß;ž…?‚(:“ ¯ƒêÒ¡‘†÷ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚290.855' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚58.17100000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ŽoŠi †?‚(:• ¯ƒêÒ‚ãù network.sent‚ 3.0939228e+07 disk.sda2.out‚0& disk.rbd0.in‚1.759218604441423e+13 disk.sda1.in‚0 disk.drbd2165.out‚0# disk.nvme0c0n1.in‚ 62284.0234375! disk.nvme0n1.in‚ 62284.0234375' disk.nvme0n1.out‚165864.19287109375 disk.rbd0.out‚0$ disk.dm-0.out‚165864.19287109375 disk.sda3.out‚ 1556.05859375) disk.nvme0c0n1.out‚165864.19287109375( proc.memory.availableMB‚ 2940.2578125# proc.memory.rssMB‚ 2865.48828125 network.recv‚ 6.595713e+06 disk.dm-0.in‚ 62284.0234375 disk.sda3.in‚ 108.92578125$ memory_percent‚82.05409049987793 disk.rbd1.in‚ 0.53564453125 disk.sda.out‚ 1556.05859375$ disk./.usageGB‚571.5167503356934 disk.sda2.in‚0 disk.rbd1.out‚ 69.4755859375 disk.sda.in‚ 108.92578125 disk.dm-1.in‚ 108.92578125 disk.sda1.out‚0 cpu‚25.584949110523564 proc.cpu.threads‚17 disk.dm-1.out‚ 1556.05859375 disk.drbd2165.in‚0) proc.memory.percent‚17.48955249786377* disk./.usagePercent‚38.3549686495221645­8Ÿ‡?‚(:” ¾ƒêÒ«²òö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚134.841' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚26.968200000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0/AiCˆ?‚(: ¾ƒêÒ±Ùù$ disk.dm-0.out‚165873.20849609375 disk.sda1.in‚0 network.sent‚ 3.0995826e+07$ disk./.usageGB‚571.5260543823242 disk.sda.in‚ 108.92578125 disk.dm-0.in‚ 62470.90625 disk.dm-1.out‚ 1556.34765625 proc.cpu.threads‚17! disk.nvme0c0n1.in‚ 62470.90625 disk.nvme0n1.in‚ 62470.90625 disk.sda.out‚ 1556.34765625 disk.sda1.out‚0$ memory_percent‚82.05156326293945 disk.drbd2165.out‚0) disk.nvme0c0n1.out‚165873.20849609375 disk.sda2.out‚0 disk.sda3.out‚ 1556.34765625 disk.drbd2165.in‚0 disk.rbd1.in‚ 0.53564453125& disk.rbd0.in‚1.759218604441423e+13) proc.memory.percent‚17.48955249786377 cpu‚25.58493893726362 network.recv‚ 6.600404e+06* disk./.usagePercent‚38.355593051898175 disk.sda3.in‚ 108.92578125' disk.nvme0n1.out‚165873.20849609375# proc.memory.rssMB‚ 2865.48828125 disk.rbd1.out‚ 69.6005859375 disk.rbd0.out‚0' proc.memory.availableMB‚ 2940.671875 disk.sda2.in‚0 disk.dm-1.in‚ 108.92578125DRSKž‰?‚(:“ ̓êÒÔƒçö gpu.0.gpu‚90.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚291.631' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚58.32619999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ò¿Š?‚(:“ ̓êÒΉ¦ù$ disk./.usageGB‚571.5302085876465$ disk.dm-0.out‚165889.36669921875 disk.drbd2165.in‚0 network.recv‚ 6.603713e+06* disk./.usagePercent‚38.355871844102964 disk.sda.in‚ 108.92578125 disk.rbd0.out‚0$ disk.nvme0c0n1.in‚62477.52734375 disk.sda2.out‚0 disk.rbd1.in‚ 0.53564453125 disk.sda.out‚ 1556.7109375 disk.sda3.in‚ 108.92578125) disk.nvme0c0n1.out‚165889.36669921875' disk.nvme0n1.out‚165889.36669921875 network.sent‚ 3.1000867e+07 disk.dm-1.out‚ 1556.7109375 disk.rbd1.out‚ 69.6005859375# memory_percent‚82.0530891418457 cpu‚25.584901197835638 disk.dm-0.in‚62477.52734375) proc.memory.percent‚17.48955249786377& disk.rbd0.in‚1.759218604441423e+13 disk.sda1.in‚0' proc.memory.availableMB‚ 2940.421875 disk.sda3.out‚ 1556.7109375" disk.nvme0n1.in‚62477.52734375 disk.sda2.in‚0# proc.memory.rssMB‚ 2865.48828125 disk.dm-1.in‚ 108.92578125 disk.sda1.out‚0 disk.drbd2165.out‚0 proc.cpu.threads‚17KËÄU”‹?‚(:‰ ܃êÒ–›þö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚301.983' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚60.3966 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0L÷-vŒ?‚(:‘ ܃êÒÇƶù disk.dm-1.in‚ 108.92578125 disk.sda3.in‚ 108.92578125 disk.sda2.out‚0) proc.memory.percent‚17.48955249786377 disk.dm-0.in‚62477.74609375 disk.drbd2165.in‚0* disk./.usagePercent‚38.356225134500214 disk.rbd1.in‚ 0.53564453125 disk.sda.in‚ 108.92578125 disk.sda.out‚ 1557.3046875 disk.sda1.out‚0 disk.rbd0.out‚0# disk.dm-0.out‚165898.2568359375& disk.nvme0n1.out‚165898.2568359375 disk.sda2.in‚0 proc.cpu.threads‚17 network.sent‚ 3.1007272e+07& disk.rbd0.in‚1.759218604441423e+13 disk.drbd2165.out‚0" disk.nvme0n1.in‚62477.74609375( proc.memory.availableMB‚ 2940.3671875 network.recv‚ 6.607673e+06 disk.sda1.in‚0 disk.sda3.out‚ 1557.3046875( disk.nvme0c0n1.out‚165898.2568359375$ disk.nvme0c0n1.in‚62477.74609375# proc.memory.rssMB‚ 2865.48828125 cpu‚25.584918856717536 disk.dm-1.out‚ 1557.3046875$ memory_percent‚82.05342292785645# disk./.usageGB‚571.535472869873 disk.rbd1.out‚ 69.6630859375¿Æ€½”?‚(:‰ ëƒêÒ¨Øüö gpu.0.gpu‚94.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚257.997' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚51.5994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0“&ï"Ž?‚(:— ëƒêÒçÒåù network.recv‚ 6.611299e+06 disk.sda1.out‚0 network.sent‚ 3.1012335e+07 disk.sda2.out‚0 disk.rbd1.in‚ 0.53564453125& disk.rbd0.in‚1.759218604441423e+13 disk.drbd2165.out‚0 disk.rbd1.out‚ 69.6630859375$ memory_percent‚82.05347061157227* disk./.usagePercent‚38.356663163391296 disk.sda3.in‚ 108.92578125 disk.sda3.out‚ 1557.58203125" disk.nvme0n1.in‚62478.16796875' disk.nvme0n1.out‚165914.17236328125 disk.sda2.in‚0) proc.memory.percent‚17.48955249786377) disk.nvme0c0n1.out‚165914.17236328125 disk.sda.in‚ 108.92578125 disk.dm-1.in‚ 108.92578125# proc.memory.rssMB‚ 2865.48828125 disk.dm-1.out‚ 1557.58203125 disk.rbd0.out‚0$ disk.dm-0.out‚165914.17236328125 cpu‚25.584907821298412 disk.drbd2165.in‚0$ disk./.usageGB‚571.5419998168945 disk.sda1.in‚0$ disk.nvme0c0n1.in‚62478.16796875 disk.sda.out‚ 1557.58203125 disk.dm-0.in‚62478.16796875' proc.memory.availableMB‚ 2940.359375 proc.cpu.threads‚17çàru4?/ _wandbruntime‚26590  _runtime‚26590¥¼œÈ? trainer/global_step‚16049 % _timestamp‚1.7843164062757854e+09 _runtime‚26590.492955342 _step‚1320 % lr-AdamW/pg1‚6.70794101770217e-05 % lr-AdamW/pg2‚6.70794101770217e-05¤8.Í‘?Ç trainer/global_step‚16049 % _timestamp‚1.7843164062757854e+09 _runtime‚26590.492955342 _step‚1320 % lr-AdamW/pg1‚6.70794101770217e-05 % lr-AdamW/pg2‚6.70794101770217e-05¨ œSãÓ4’?/ _wandbruntime‚26590  _runtime‚26590¤FP“?“P TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚45467066 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚45467296 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚4630220236673536 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚4615118451761152 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚4628201328806078 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1265250378311168 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚32439296 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _step‚1321 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚4630218865110528 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚4615118477934592 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 276022896 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 139031985 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 400530432 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 _runtime‚26590.494243504 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚4613125410861056 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 $ _timestamp‚1.784316406276878e+09 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚4628199958777788 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚19476925724160 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚86643370 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_øFÃÉ pool.current‚ 47351595008 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 139022012 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15101758738944 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15075917945022 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚4630220236673536 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚4615118477934592 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 139022012 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚41176069 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚4630218865110528 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚235 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚41176074 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚230 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1265249977780736 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872:F>(”?˜P _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15075917945022 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚4630220236673536 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚4615118477934592 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 139022012 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚41176069 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚4630218865110528 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚235 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚41176074 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚230 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1265249977780736 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚45467066 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚45467296 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚4630220236673536 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚4615118451761152 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚4628201328806078 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1265250378311168 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚32439296 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _step‚1321 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚4630218865110528 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚4615118477934592 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 276022896 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 139031985 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 400530432 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 _runtime‚26590.494243504 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚4613125410861056 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 $ _timestamp‚1.784316406276878e+09 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚4628199958777788 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚19476925724160 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚86643370 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 139022012 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15101758738944 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979© ®0 4•?/ _wandbruntime‚26591  _runtime‚26591dýù$'–?žN F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚45470039 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚32432128 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚45469818 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1265328681660928 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 276039594 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚86648656 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 % _timestamp‚1.7843164076617172e+09 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚4628480134050612 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15076841411380 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 139030490 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1245850570903552 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 276039594 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚4613403292639232 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚4613403267211264 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚4630499162446336 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1245850938994688 _step‚1322 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚4615396452491264 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚4615396478664704 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚86648430 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 139030490 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚4615396478664704 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚41178612 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1265329082184192 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 139040477 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚41178617 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚4615396452491264 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 400523264 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 139040477 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚15101338384896 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15102683781632 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 trainer/global_step‚16049 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 136999117 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 136999117 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 276029601 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚4630499162446336 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚19478143189504N`HGž#—?£N JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚32432128 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚45469818 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1265328681660928 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 276039594 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚86648656 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 % _timestamp‚1.7843164076617172e+09 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚4628480134050612 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15076841411380 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 139030490 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1245850570903552 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 276039594 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚4613403292639232 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚4613403267211264 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚4630499162446336 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1245850938994688 _step‚1322 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚4615396452491264 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚4615396478664704 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚86648430 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 139030490 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚4615396478664704 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚41178612 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1265329082184192 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 139040477 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚41178617 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚4615396452491264 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 400523264 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 139040477 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚15101338384896 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15102683781632 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 trainer/global_step‚16049 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 136999117 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 136999117 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 276029601 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.curren™×L¶‹t‚6 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚4630499162446336 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚19478143189504 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚45470039 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175ª .‚p“4˜?/ _wandbruntime‚26591  _runtime‚26591ÀÍ‘j™?Œ 2loss/SRqqt1/assignment_loss‚0.0422129105217004 /loss/FRt2/detection_loss‚1.1859712295455829 /loss/FBt1/detection_loss‚0.4740483216510256 0loss/FBt2/detection_loss‚0.10251225937549183 %loss/total_loss‚4.121110290708033 trainer/global_step‚16049 % _timestamp‚1.7843164077487073e+09 _runtime‚26591.965654974 3loss/SRqqt2/assignment_loss‚0.10095517467145884 3loss/FBt1/assignment_loss‚4.108663809421209e-05 0loss/FRt1/detection_loss‚0.41485230563358927 1loss/SRqqt1/detection_loss‚0.4195367873552928 epoch‚40 /loss/FRt1/assignment_loss‚0.860513824469118 _step‚1323 1loss/FRt2/assignment_loss‚0.07299420984524473 3loss/FBt2/assignment_loss‚5.432818444866198e-06 1loss/SRqqt2/detection_loss‚0.4474667481829895RwÜÜš?‘ /loss/FRt1/assignment_loss‚0.860513824469118 _step‚1323 1loss/FRt2/assignment_loss‚0.07299420984524473 3loss/FBt2/assignment_loss‚5.432818444866198e-06 1loss/SRqqt2/detection_loss‚0.4474667481829895 2loss/SRqqt1/assignment_loss‚0.0422129105217004 /loss/FRt2/detection_loss‚1.1859712295455829 /loss/FBt1/detection_loss‚0.4740483216510256 0loss/FBt2/detection_loss‚0.10251225937549183 %loss/total_loss‚4.121110290708033 trainer/global_step‚16049 % _timestamp‚1.7843164077487073e+09 _runtime‚26591.965654974 3loss/SRqqt2/assignment_loss‚0.10095517467145884 3loss/FBt1/assignment_loss‚4.108663809421209e-05 0loss/FRt1/detection_loss‚0.41485230563358927 1loss/SRqqt1/detection_loss‚0.4195367873552928 epoch‚40« Aé'Þ”›?‚(:‰ úƒêÒ†Ùéö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚264.538' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.9076 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0á-zBœ?‚(:‡ úƒêÒˆ«†ù disk.sda3.out‚ 1557.796875 disk.rbd1.out‚ 69.7255859375$ memory_percent‚82.05571174621582# proc.memory.rssMB‚ 2865.48828125) proc.memory.percent‚17.48955249786377 disk.dm-0.in‚62478.38671875& disk.nvme0c0n1.out‚165921.75390625( proc.memory.availableMB‚ 2939.9921875 disk.drbd2165.in‚0 network.recv‚ 6.614891e+06 disk.dm-1.in‚ 108.92578125$ disk.nvme0c0n1.in‚62478.38671875$ disk./.usageGB‚571.5466194152832 disk.dm-1.out‚ 1557.796875 disk.rbd0.out‚0! disk.dm-0.out‚165921.75390625 disk.sda1.out‚0 disk.drbd2165.out‚0$ disk.nvme0n1.out‚165921.75390625 disk.sda2.out‚0 disk.sda1.in‚0 disk.sda.in‚ 108.92578125 cpu‚25.58488026365141 network.sent‚ 3.1016052e+07& disk.rbd0.in‚1.759218604441423e+13" disk.nvme0n1.in‚62478.38671875 disk.sda2.in‚0 disk.rbd1.in‚ 0.53955078125 disk.sda.out‚ 1557.796875 proc.cpu.threads‚17) disk./.usagePercent‚38.35697318851526 disk.sda3.in‚ 108.92578125Çžÿf?‚(:’ ‰„êÒ×¥ïö gpu.0.gpu‚83.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚46.0 gpu.0.powerWatts‚341.45' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚68.28999999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0vˆq\ž?‚(:ø ‰„êÒÙÃù network.sent‚ 3.1080736e+07$ disk./.usageGB‚571.5558815002441 disk.dm-0.in‚62650 disk.sda1.out‚0 disk.sda2.in‚0 disk.rbd1.in‚ 0.53955078125) proc.memory.availableMB‚ 2940.05859375 disk.rbd1.out‚ 69.7255859375* disk./.usagePercent‚38.357594774808405& disk.rbd0.in‚1.759218604441423e+13$ disk.dm-0.out‚165931.82763671875 disk.sda3.in‚ 108.92578125 disk.sda2.out‚0 disk.sda.in‚ 108.92578125 disk.dm-1.out‚ 1558.328125) disk.nvme0c0n1.out‚165931.82763671875' disk.nvme0n1.out‚165931.82763671875 disk.sda3.out‚ 1558.328125 disk.nvme0c0n1.in‚62650$ memory_percent‚82.05530643463135) proc.memory.percent‚17.48955249786377 cpu‚25.584840906419483 proc.cpu.threads‚17 disk.rbd0.out‚0 disk.drbd2165.in‚0# proc.memory.rssMB‚ 2865.48828125 network.recv‚ 6.619335e+06 disk.sda.out‚ 1558.328125 disk.dm-1.in‚ 108.92578125 disk.sda1.in‚0 disk.drbd2165.out‚0 disk.nvme0n1.in‚62650&Z—¿”Ÿ?‚(:‰ ˜„êÒ¨Èðø gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚265.067' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.0134 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0îàÿ ?‚(:Œ ˜„êÒáø‰ú disk.sda.out‚ 1558.41796875 disk.nvme0n1.in‚ 62650.21875 cpu‚25.584829558262285 network.recv‚ 6.622719e+06 disk.dm-1.out‚ 1558.41796875 disk.sda1.in‚0 disk.sda1.out‚0) proc.memory.percent‚17.48955249786377& disk.nvme0n1.out‚165944.5068359375 disk.sda2.out‚0 disk.drbd2165.in‚0! disk.nvme0c0n1.in‚ 62650.21875) disk./.usagePercent‚38.35789558366113# proc.memory.rssMB‚ 2865.48828125$ memory_percent‚82.05230236053467 disk.rbd1.in‚ 0.53955078125 disk.sda3.in‚ 108.92578125 disk.sda3.out‚ 1558.41796875( disk.nvme0c0n1.out‚165944.5068359375 disk.dm-1.in‚ 108.92578125 disk.rbd0.out‚0 disk.sda2.in‚0 disk.rbd1.out‚ 69.7880859375 disk.sda.in‚ 108.92578125 disk.dm-0.in‚ 62650.21875 disk.drbd2165.out‚0 proc.cpu.threads‚17# disk.dm-0.out‚165944.5068359375) proc.memory.availableMB‚ 2940.55078125& disk.rbd0.in‚1.759218604441423e+13 network.sent‚ 3.1084462e+07$ disk./.usageGB‚571.5603637695312*  ”¡?‚(:‰ §„êÒ¨Û³÷ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚333.089' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚66.6178 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0˜ÏJ’¢?‚(: §„êÒŽâ ù& disk.rbd0.in‚1.759218604441423e+13$ disk.dm-0.out‚165956.34716796875# memory_percent‚82.0554256439209 disk.sda3.in‚ 108.92578125) proc.memory.percent‚17.48955249786377) disk.nvme0c0n1.out‚165956.34716796875 disk.nvme0n1.in‚ 62650.421875 disk.dm-1.in‚ 108.92578125 disk.sda1.in‚0 network.sent‚ 3.1089607e+07$ disk./.usageGB‚571.5669441223145 disk.dm-1.out‚ 1558.76171875 disk.sda3.out‚ 1558.76171875( proc.memory.availableMB‚ 2940.0390625# proc.memory.rssMB‚ 2865.48828125 proc.cpu.threads‚17 network.recv‚ 6.626021e+06' disk.nvme0n1.out‚165956.34716796875 disk.dm-0.in‚ 62650.421875 disk.sda1.out‚0 disk.drbd2165.in‚0 cpu‚25.584802233678577( disk./.usagePercent‚38.3583371966577 disk.sda.out‚ 1558.76171875 disk.rbd1.in‚ 0.53955078125 disk.rbd1.out‚ 69.7880859375 disk.sda.in‚ 108.92578125 disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 62650.421875 disk.sda2.in‚0 disk.sda2.out‚0 disk.drbd2165.out‚0îs’”£?‚(:‰ ¶„êÒ¥ˆ¾÷ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚275.567' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.1134 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0í’Ó^¤?‚(: ¶„êÒœßÖú) disk./.usagePercent‚38.35880722649056) disk.nvme0c0n1.out‚165964.30908203125 disk.drbd2165.in‚0) proc.memory.percent‚17.48955249786377 disk.rbd1.out‚ 69.8505859375 disk.sda2.in‚0$ disk.dm-0.out‚165964.30908203125! disk.nvme0c0n1.in‚ 62650.84375 disk.nvme0n1.in‚ 62650.84375 disk.sda3.in‚ 108.92578125 disk.drbd2165.out‚0) proc.memory.availableMB‚ 2940.19921875 network.recv‚ 6.629678e+06' disk.nvme0n1.out‚165964.30908203125 disk.sda2.out‚0 disk.rbd1.in‚ 0.53955078125 disk.sda.in‚ 108.92578125 disk.sda.out‚ 1558.98828125& disk.rbd0.in‚1.759218604441423e+13 disk.dm-0.in‚ 62650.84375 disk.dm-1.out‚ 1558.98828125 disk.rbd0.out‚0 disk.sda1.out‚0$ memory_percent‚82.05444812774658# proc.memory.rssMB‚ 2865.48828125 network.sent‚ 3.1095995e+07$ disk./.usageGB‚571.5739479064941 disk.dm-1.in‚ 108.92578125 cpu‚25.584807962470673 disk.sda1.in‚0 disk.sda3.out‚ 1558.98828125 proc.cpu.threads‚17«bPº”¥?‚(:‰ Å„êÒòÝóö gpu.0.gpu‚99.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚303.048' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚60.6096 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚00pÄ ¦?‚(:• Å„êÒÞÔÐù disk.sda.out‚ 1559.30078125 disk.dm-0.in‚62830.44140625 disk.sda1.out‚0 network.sent‚ 3.1099711e+07$ disk./.usageGB‚571.5791549682617 disk.dm-1.in‚ 108.92578125 disk.sda3.in‚ 108.92578125 network.recv‚ 6.632954e+06* disk./.usagePercent‚38.359156676774795 disk.rbd0.out‚0 disk.sda3.out‚ 1559.30078125# proc.memory.rssMB‚ 2865.48828125 disk.drbd2165.in‚0& disk.nvme0n1.out‚165979.3974609375( proc.memory.availableMB‚ 2940.2109375# disk.dm-0.out‚165979.3974609375 disk.sda1.in‚0 disk.sda2.in‚0 disk.sda2.out‚0) proc.memory.percent‚17.48955249786377" disk.nvme0n1.in‚62830.44140625 proc.cpu.threads‚17 disk.sda.in‚ 108.92578125 disk.dm-1.out‚ 1559.30078125 disk.drbd2165.out‚0$ disk.nvme0c0n1.in‚62830.44140625( disk.nvme0c0n1.out‚165979.3974609375 disk.rbd1.out‚ 69.8505859375$ memory_percent‚82.05437660217285& disk.rbd0.in‚1.759218604441423e+13 disk.rbd1.in‚ 0.53955078125 cpu‚25.584753010641688¦¶J˜4§?/ _wandbruntime‚26670  _runtime‚26670‡,ÄÃɨ?à & lr-AdamW/pg2‚6.544772415180452e-05 trainer/global_step‚16099 % _timestamp‚1.7843164859758549e+09 _runtime‚26670.19614233 _step‚1324 & lr-AdamW/pg1‚6.544772415180452e-05H~ùÌΩ?È trainer/global_step‚16099 % _timestamp‚1.7843164859758549e+09 _runtime‚26670.19614233 _step‚1324 & lr-AdamW/pg1‚6.544772415180452e-05 & lr-AdamW/pg2‚6.544772415180452e-05¬ &/+4ª?/ _wandbruntime‚26670  _runtime‚26670…‘-(«?”P QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 137401205 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 _runtime‚26670.197232088 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 139457737 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15148074669568 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚4644165217546752 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1269184740979200 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚4627019474341888 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚1325 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1269184336254464 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 trainer/global_step‚16099 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1249647404879872 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15120810457808 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1249647772971008 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚15146729280000 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 276848969 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36633600 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚19536968008192 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404724736 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚19536931374592 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 137401205 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 139457737 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚4629018514440192 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 139447764 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚41303224 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 137401211 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 137401211 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚4629018488266752 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚265 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1249647772971008 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚15146729280000 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 276848969 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36633600 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚19536968008192 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404724736 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚19536931374592 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 137401205 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 139457737 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚4629018514440192 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 139447764 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚41303224 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 137401211 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 137401211 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚4629018488266752 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚265 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚4644165217546752 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1269184740979200 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚4627019474341888 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚1325 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1269184336254464 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973­ gÙØ4­?/ _wandbruntime‚26671  _runtime‚26671-\®?žN RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_traÂEí!&in_batch_end/active.all.current‚9993 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1269263015776768 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚86912104 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚45606342 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1269263418397184 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚4627297381548032 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚15147654315520 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚4629296488996864 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚4629296515170304 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1249725259378688 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚19538159018496 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 137409417 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 139466229 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15147654315520 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚257 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 276865653 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 139466229 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 139456242 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15148999712256 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 137409411 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 276865653 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚4644445514882560 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚45606594 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚4644444143312384 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚41305762 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 139456242 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 trainer/global_step‚16099 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 137409417 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚4629296515170304 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 276875646 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚4642420460072520*¿ƒƒ)'¯?£N ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚4629296515170304 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1249725259378688 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚19538159018496 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 137409417 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 139466229 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15147654315520 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚257 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 276865653 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 139466229 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 139456242 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15148999712256 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 137409411 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 276865653 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚4644445514882560 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚45606594 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚4644444143312384 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚41305762 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 139456242 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 trainer/global_step‚16099 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 137409417 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚4629296515170304 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 276875646 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚4642420460072520 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1269263015776768 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚86912104 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚45606342 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1269263418397184 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚4627297381548032 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚15147654315520 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚4629296488996864 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0® oÔH~4°?/ _wandbruntime‚26671  _runtime‚26671«þä8±? epoch‚40 trainer/global_step‚16099 0loss/FRt1/assignment_loss‚0.7804848477244377 1loss/FRt2/assignment_loss‚0.08058018382773342 2loss/SRqqt2/assignment_loss‚0.1168127451357332 3loss/FBt1/assignment_loss‚4.873518763972965e-07 1loss/SRqqt1/detection_loss‚0.3947835741608726 0loss/FBt1/detection_loss‚0.09744221665407984 _step‚1327 4loss/SRqqt1/assignment_loss‚0.059237027712175525 /loss/FRt2/detection_loss‚1.2036086871748768 %loss/total_loss‚3.680598168649733 $ _timestamp‚1.784316487453301e+09 _runtime‚26671.670237667 /loss/FRt1/detection_loss‚0.4421788286122311 2loss/SRqqt2/detection_loss‚0.41891155340125136 4loss/FBt2/assignment_loss‚0.00024725157456013177 0loss/FBt2/detection_loss‚0.08631076531990484{.–²?” /loss/FRt2/detection_loss‚1.2036086871748768 %loss/total_loss‚3.680598168649733 $ _timestamp‚1.784316487453301e+09 _runtime‚26671.670237667 /loss/FRt1/detection_loss‚0.4421788286122311 2loss/SRqqt2/detection_loss‚0.41891155340125136 4loss/FBt2/assignment_loss‚0.00024725157456013177 0loss/FBt2/detection_loss‚0.08631076531990484 epoch‚40 trainer/global_step‚16099 0loss/FRt1/assignment_loss‚0.7804848477244377 1loss/FRt2/assignment_loss‚0.08058018382773342 2loss/SRqqt2/assignment_loss‚0.1168127451357332 3loss/FBt1/assignment_loss‚4.873518763972965e-07 1loss/SRqqt1/detection_loss‚0.3947835741608726 0loss/FBt1/detection_loss‚0.09744221665407984 _step‚1327 4loss/SRqqt1/assignment_loss‚0.059237027712175525¯ Vû.y•³?‚(:Š Ô„êÒýñåö gpu.0.gpu‚100.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚339.701' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚67.9402 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0”%››´?‚(:’ Ô„êÒõ™†ù disk.dm-1.out‚ 1559.484375$ disk.dm-0.out‚165990.89501953125' disk.nvme0n1.out‚165990.89501953125$ memory_percent‚82.05134868621826 cpu‚25.584762741314428 proc.cpu.threads‚17" disk.nvme0n1.in‚62830.64453125 network.sent‚ 3.1104774e+07$ disk./.usageGB‚571.6942367553711 disk.rbd1.out‚ 69.9755859375 disk.drbd2165.out‚0$ disk.nvme0c0n1.in‚62830.64453125 disk.dm-1.in‚ 108.92578125& disk.rbd0.in‚1.759218604441423e+13 disk.sda1.in‚0 disk.sda3.in‚ 108.92578125 disk.sda3.out‚ 1559.484375# proc.memory.rssMB‚ 2865.48828125) proc.memory.percent‚17.48955249786377 network.recv‚ 6.636355e+06 disk.rbd1.in‚ 0.53955078125 disk.sda.out‚ 1559.484375 disk.sda2.in‚0) disk./.usagePercent‚38.36687991206776 disk.rbd0.out‚0 disk.drbd2165.in‚0 disk.dm-0.in‚62830.64453125) disk.nvme0c0n1.out‚165990.89501953125 disk.sda.in‚ 108.92578125 disk.sda1.out‚0 disk.sda2.out‚0) proc.memory.availableMB‚ 2940.70703125>ÆQ”µ?‚(:‰ ã„êÒíñæö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚286.551' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚57.3102 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ì€Æ¶?‚(:” ã„êÒã¾½ù) disk./.usagePercent‚38.37508904966071 disk.dm-1.out‚ 1559.85546875 disk.sda3.out‚ 1559.85546875( proc.memory.availableMB‚ 2940.4453125) proc.memory.percent‚17.48955249786377$ disk./.usageGB‚571.8165588378906 disk.rbd0.out‚0" disk.nvme0n1.in‚62831.06640625& disk.nvme0n1.out‚166008.4482421875 disk.sda2.out‚0 network.sent‚ 3.1169676e+07 network.recv‚ 6.640766e+06 disk.sda2.in‚0$ memory_percent‚82.05294609069824 proc.cpu.threads‚17 disk.sda.in‚ 108.92578125$ disk.nvme0c0n1.in‚62831.06640625( disk.nvme0c0n1.out‚166008.4482421875 disk.rbd1.out‚ 69.9755859375 disk.rbd1.in‚ 0.53955078125 disk.sda1.in‚0 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.sda.out‚ 1559.85546875 disk.dm-0.in‚62831.06640625 disk.sda1.out‚0 disk.sda3.in‚ 108.92578125# proc.memory.rssMB‚ 2865.48828125 disk.dm-1.in‚ 108.92578125& disk.rbd0.in‚1.759218604441423e+13# disk.dm-0.out‚166008.4482421875 cpu‚25.584733002346763½©-’·?‚(:‡ ò„êÒ¾¼êö gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚334.153' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚66.8306 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0<#"¸?‚(:’ ò„êÒð½Èø disk.rbd1.out‚ 70.0380859375 network.sent‚ 3.1174702e+07$ disk./.usageGB‚571.9320945739746 disk.dm-1.out‚ 1560.03515625' disk.nvme0c0n1.out‚166116.060546875$ memory_percent‚82.05139636993408" disk.nvme0n1.in‚62831.28515625 disk.sda.in‚ 108.92578125 disk.sda3.out‚ 1560.03515625 network.recv‚ 6.644213e+06 disk.sda.out‚ 1560.03515625 disk.sda3.in‚ 108.92578125# proc.memory.rssMB‚ 2865.48828125 proc.cpu.threads‚17 disk.sda2.out‚0 disk.dm-1.in‚ 108.92578125& disk.rbd0.in‚1.759218604441423e+13 disk.sda1.in‚0) proc.memory.availableMB‚ 2940.69921875 disk.rbd1.in‚ 0.53955078125 disk.dm-0.in‚62831.28515625 disk.drbd2165.in‚0% disk.nvme0n1.out‚166116.060546875" disk.dm-0.out‚166116.060546875 disk.sda1.out‚0 disk.drbd2165.out‚0$ disk.nvme0c0n1.in‚62831.28515625) proc.memory.percent‚17.48955249786377 cpu‚25.584743388951424) disk./.usagePercent‚38.38284274985024 disk.sda2.in‚0 disk.rbd0.out‚0Oô韹?‚(:” …êÒ’žûö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚290.369' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚58.073800000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Àp_<º?‚(:“ …êÒ«ðÕù disk.sda3.in‚ 108.92578125) disk.nvme0c0n1.out‚166223.46337890625 disk.sda2.in‚0 disk.sda.out‚ 1560.29296875 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd2165.out‚0$ memory_percent‚82.05289840698242 network.recv‚ 6.647288e+06) disk./.usagePercent‚38.39087140213155 disk.rbd1.in‚ 0.53955078125 disk.dm-1.in‚ 108.92578125& disk.rbd0.in‚1.759218604441423e+13# disk.nvme0c0n1.in‚ 62997.8359375! disk.nvme0n1.in‚ 62997.8359375) proc.memory.percent‚17.48955249786377 proc.cpu.threads‚17 disk.dm-1.out‚ 1560.29296875 disk.sda3.out‚ 1560.29296875 cpu‚25.584693656873505 disk.drbd2165.in‚0# proc.memory.rssMB‚ 2865.48828125$ disk./.usageGB‚572.0517272949219 disk.rbd1.out‚ 70.0380859375$ disk.dm-0.out‚166223.46337890625' disk.nvme0n1.out‚166223.46337890625 disk.rbd0.out‚0 disk.dm-0.in‚ 62997.8359375' proc.memory.availableMB‚ 2940.453125 network.sent‚ 3.1179769e+07 disk.sda2.out‚0 disk.sda.in‚ 108.92578125Å-ÖŸ»?‚(:” …êÒç°ë÷ gpu.0.gpu‚83.0 gpu.0.memory‚25, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚38.0 gpu.0.powerWatts‚110.415' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.083000000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 è›#¼?‚(:˜ …êÒȽÆù cpu‚25.584702477404665$ disk.dm-0.out‚166360.90673828125$ memory_percent‚82.05447196960449( proc.memory.availableMB‚ 2940.1953125' disk.nvme0n1.out‚166360.90673828125 disk.rbd1.in‚ 0.53955078125 disk.sda.out‚ 1561.02734375 disk.dm-1.in‚ 108.92578125 disk.dm-0.in‚62998.24609375 disk.sda3.in‚ 108.92578125 disk.sda2.out‚0& disk.rbd0.in‚1.759218604441423e+13 disk.sda3.out‚ 1561.02734375$ disk.nvme0c0n1.in‚62998.24609375 network.recv‚ 6.650824e+06 disk.rbd0.out‚0 disk.drbd2165.out‚0) proc.memory.percent‚17.48955249786377* disk./.usagePercent‚38.395010531945154 disk.sda1.out‚0# proc.memory.rssMB‚ 2865.48828125 proc.cpu.threads‚17$ disk./.usageGB‚572.1134033203125" disk.nvme0n1.in‚62998.24609375 disk.sda1.in‚0) disk.nvme0c0n1.out‚166360.90673828125 network.sent‚ 3.1183501e+07 disk.sda2.in‚0 disk.rbd1.out‚ 70.1005859375 disk.sda.in‚ 108.92578125 disk.dm-1.out‚ 1561.02734375 disk.drbd2165.in‚0Ë™àn4½?/ _wandbruntime‚26749  _runtime‚26749‚~5Åɾ?à & lr-AdamW/pg2‚6.383405260818417e-05 trainer/global_step‚16149 % _timestamp‚1.7843165654257872e+09 _runtime‚26749.64295467 _step‚1328 & lr-AdamW/pg1‚6.383405260818417e-05ëÂŒ´Î¿?È _step‚1328 & lr-AdamW/pg1‚6.383405260818417e-05 & lr-AdamW/pg2‚6.383405260818417e-05 trainer/global_step‚16149 % _timestamp‚1.7843165654257872e+09 _runtime‚26749.64295467° ª.I´4À?/ _wandbruntime‚26749  _runtime‚26749äôÍ!5 Á?”P YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15168392171238 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚19596685357568 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚4658112941545984 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚19596719894016 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚4658111569982976 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 277695000 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 139873516 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚4658112941545984 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚41430374 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _step‚1329 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 137811505 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15194390600192 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚87170363 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚45739989 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.pe\9håak‚ 1105813504 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _runtime‚26749.644206354 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚4640913588678656 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚4658111569982976 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚255 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 137811511 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 139883489 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚4642918524772352 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 137811511 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 137811505 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚4656081980849894 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 277685021 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 139883489 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚4642918550945792 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚250 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15167047570916 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15193045210624 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1253522093355008 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚4640913563250688 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚87170108 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚4656080610821604 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 trainer/global_step‚16149 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1273118410621440 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚41430369 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 139873516 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚19596685357568 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚4658112941545984 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚19596719894016 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚4658111569982976 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 277695000 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 139873516 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚4658112941545984 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚41430374 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _step‚1329 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 137811505 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15194390600192 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚87170363 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚45739989 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _runtime‚26749.644206354 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚4640913588678656 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚4658111569982976 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚255 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 137811511 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 139883489 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚4642918524772352 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 137811511 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 137811505 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚4656081980849894 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 277685021 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 139883489 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚4642918550945792 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚250 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15167047570916 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15193045210624 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1253522093355008 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚4640913563250688 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚87170108 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚4656080610821604 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 trainer/global_step‚16149 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1273118410621440 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚41430369 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 139873516 JDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 277711698 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1253599579762688 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚87175355 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚19597910357504 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 % _timestamp‚1.7843165668116047e+09 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚15193970246144 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚4658390495748608 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1253599211671552 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 277711698 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 137819717 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚4658391867318784 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚238 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 139881994 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1273197087499776 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 137819717 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 139891981 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15193970246144 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 277701705 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 137819711 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚4641191445028864 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚4643196525502464 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 277701705 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚41432912 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008@ÔOÅ?£N YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 _runtime‚26751.029033998 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚4658391867318784 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚4643196525502464 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 139891981 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚243 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚4641191470456832 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15167971037162 trainer/global_step‚16149 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15195315642880 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚4656359416066026 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 139881994 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚45742681 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚45742443 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1273197490120192 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 277711698 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1253599579762688 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚87175355 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚19597910357504 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 % _timestamp‚1.7843165668116047e+09 EDeviceStatsMonitor.on_train_batch_end/reser¥Áèved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚15193970246144 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚4658390495748608 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1253599211671552 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 277711698 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 137819717 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚4658391867318784 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚238 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 139881994 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1273197087499776 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 137819717 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 139891981 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15193970246144 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 277701705 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 137819711 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚4641191445028864 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚4643196525502464 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 277701705 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚41432912 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995² ÆÙF@4Æ?/ _wandbruntime‚26751  _runtime‚26751-nÞ!Ç?’ 4loss/FBt2/assignment_loss‚2.8190664619071664e-06 1loss/SRqqt2/detection_loss‚0.4504083765155462 /loss/FBt1/detection_loss‚0.2462509419652334 0loss/FBt2/detection_loss‚0.06690691035171102 &loss/total_loss‚3.8837032770130215 trainer/global_step‚16149 epoch‚40 _step‚1331 0loss/FRt1/assignment_loss‚0.8716390058468735 3loss/SRqqt1/assignment_loss‚0.06190263830349581 /loss/FRt2/detection_loss‚1.1732621143803517 % _timestamp‚1.7843165668986197e+09 1loss/FRt2/assignment_loss‚0.09390171925635449 3loss/SRqqt2/assignment_loss‚0.09043473855046147 4loss/FBt1/assignment_loss‚0.00010859726462034891 0loss/FRt1/detection_loss‚0.42109179277156844 2loss/SRqqt1/detection_loss‚0.40779362274034336 _runtime‚26751.115540727•,WeÈ?— 0loss/FRt1/assignment_loss‚0.8716390058468735 3loss/SRqqt1/assignment_loss‚0.06190263830349581 /loss/FRt2/detection_loss‚1.1732621143803517 % _timestamp‚1.7843165668986197e+09 1loss/FRt2/assignment_loss‚0.09390171925635449 3loss/SRqqt2/assignment_loss‚0.09043473855046147 4loss/FBt1/assignment_loss‚0.00010859726462034891 0loss/FRt1/detection_loss‚0.42109179277156844 2loss/SRqqt1/detection_loss‚0.40779362274034336 _runtime‚26751.115540727 4loss/FBt2/assignment_loss‚2.8190664619071664e-06 1loss/SRqqt2/detection_loss‚0.4504083765155462 /loss/FBt1/detection_loss‚0.2462509419652334 0loss/FBt2/detection_loss‚0.06690691035171102 &loss/total_loss‚3.8837032770130215 trainer/global_step‚16149 epoch‚40 _step‚1331³ Y}Z”É?‚(:‰ Ÿ…êÒ ÷ gpu.0.gpu‚91.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚293.613' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚58.7226 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0™S^Ê?‚(:Š Ÿ…êÒž¶ù disk.dm-1.in‚ 108.92578125 disk.rbd0.out‚0 disk.dm-0.in‚62998.46484375$ disk.nvme0c0n1.in‚62998.46484375" disk.nvme0n1.in‚62998.46484375 disk.sda.in‚ 108.92578125! disk.dm-0.out‚166504.41015625 disk.sda1.in‚0 disk.sda3.out‚ 1561.2109375) proc.memory.availableMB‚ 2940.21484375 network.sent‚ 3.1188541e+07$ disk./.usageGB‚572.1182250976562 disk.rbd1.out‚ 70.1005859375$ memory_percent‚82.05435276031494 network.recv‚ 6.654072e+06 disk.sda1.out‚0 proc.cpu.threads‚17) disk./.usagePercent‚38.39533412546843 disk.sda3.in‚ 108.92578125$ disk.nvme0n1.out‚166504.41015625 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda.out‚ 1561.2109375 disk.drbd2165.in‚0) proc.memory.percent‚17.48955249786377 disk.dm-1.out‚ 1561.2109375& disk.rbd0.in‚1.759218604441423e+13 disk.drbd2165.out‚0& disk.nvme0c0n1.out‚166504.41015625 disk.rbd1.in‚ 0.53955078125# proc.memory.rssMB‚ 2865.48828125 cpu‚25.5847113211708L¾9ŸË?‚(:” ®…êÒìÊãö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚274.958' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚54.991600000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0GëlZ#Ì?‚(:˜ ®…êÒàŸ¤ù network.recv‚ 6.658423e+06$ disk./.usageGB‚572.1011390686035 disk.dm-1.out‚ 1561.68359375 disk.dm-0.in‚62998.88671875) proc.memory.availableMB‚ 2939.98046875 disk.dm-1.in‚ 108.92578125 disk.rbd0.out‚0 disk.sda1.in‚0 disk.sda1.out‚0$ disk.nvme0c0n1.in‚62998.88671875# proc.memory.rssMB‚ 2865.48828125) disk./.usagePercent‚38.39418746772257$ disk.dm-0.out‚166551.66748046875 disk.sda3.in‚ 108.92578125" disk.nvme0n1.in‚62998.88671875 disk.rbd1.out‚ 70.1630859375 disk.sda2.in‚0 cpu‚25.584673497201262$ memory_percent‚82.05578327178955 network.sent‚ 3.1253139e+07 disk.sda.in‚ 108.92578125 disk.sda.out‚ 1561.68359375) proc.memory.percent‚17.48955249786377 disk.drbd2165.in‚0) disk.nvme0c0n1.out‚166551.66748046875 disk.sda2.out‚0 disk.rbd1.in‚ 0.53955078125& disk.rbd0.in‚1.759218604441423e+13 disk.sda3.out‚ 1561.68359375 disk.drbd2165.out‚0' disk.nvme0n1.out‚166551.66748046875 proc.cpu.threads‚17=êà•Í?‚(:Š ½…êÒÛöö gpu.0.gpu‚100.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚337.146' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚67.4292 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‰v³[Î?‚(:“ ½…êÒ˼Îù) disk.nvme0c0n1.out‚166567.71630859375 disk.sda.out‚ 1562.046875 disk.dm-1.in‚ 108.92578125 disk.sda1.out‚0 disk.rbd1.in‚ 0.53955078125 disk.dm-0.in‚63097.74609375 disk.sda3.in‚ 108.92578125 disk.sda3.out‚ 1562.046875$ memory_percent‚82.05430507659912 cpu‚25.584653439619455" disk.nvme0n1.in‚63097.74609375 disk.sda2.out‚0 disk.rbd1.out‚ 70.1630859375$ disk.dm-0.out‚166567.71630859375 network.sent‚ 3.1258226e+07$ disk./.usageGB‚572.1052131652832 disk.sda.in‚ 108.92578125 disk.rbd0.out‚0 disk.drbd2165.out‚0$ disk.nvme0c0n1.in‚63097.74609375 disk.sda1.in‚0 disk.drbd2165.in‚0 proc.cpu.threads‚17* disk./.usagePercent‚38.394460883769135 disk.sda2.in‚0 disk.dm-1.out‚ 1562.046875 network.recv‚ 6.662016e+06' disk.nvme0n1.out‚166567.71630859375& disk.rbd0.in‚1.759218604441423e+13) proc.memory.availableMB‚ 2940.22265625# proc.memory.rssMB‚ 2865.48828125) proc.memory.percent‚17.48955249786377û«ß”Ï?‚(:‰ Ì…êÒ”üþö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚277.698' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.5396 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ñ;’#Ð?‚(:˜ Ì…êÒ„ñ¾ù disk.rbd1.out‚ 70.2255859375 disk.sda.in‚ 108.92578125 disk.dm-1.in‚ 108.92578125 disk.rbd1.in‚ 0.53955078125 cpu‚25.584624986076125 disk.dm-0.in‚63097.96484375 disk.sda1.out‚0) disk./.usagePercent‚38.39492297736844$ disk./.usageGB‚572.1120986938477 disk.drbd2165.in‚0 disk.dm-1.out‚ 1562.19921875$ memory_percent‚82.05487728118896# proc.memory.rssMB‚ 2865.48828125 network.sent‚ 3.1262044e+07 disk.sda3.in‚ 108.92578125 disk.sda.out‚ 1562.19921875$ disk.dm-0.out‚166576.54833984375 disk.sda2.in‚0 disk.sda2.out‚0) proc.memory.availableMB‚ 2940.12890625) proc.memory.percent‚17.48955249786377 proc.cpu.threads‚17 network.recv‚ 6.665562e+06 disk.rbd0.out‚0 disk.sda1.in‚0' disk.nvme0n1.out‚166576.54833984375& disk.rbd0.in‚1.759218604441423e+13 disk.sda3.out‚ 1562.19921875$ disk.nvme0c0n1.in‚63097.96484375 disk.drbd2165.out‚0) disk.nvme0c0n1.out‚166576.54833984375" disk.nvme0n1.in‚63097.96484375b@jM•Ñ?‚(:Š Û…êÒáæö gpu.0.gpu‚100.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚333.913' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚66.7826 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0„“’¾Ò?‚(:“ Û…êÒ¥£Òù disk.rbd0.out‚0 disk.sda3.in‚ 108.92578125 disk.drbd2165.in‚0' disk.nvme0n1.out‚166591.70849609375 disk.sda2.out‚0 disk.sda.out‚ 1562.4296875# proc.memory.rssMB‚ 2865.48828125 proc.cpu.threads‚17 network.sent‚ 3.1268446e+07 network.recv‚ 6.668634e+06$ disk./.usageGB‚572.1172790527344) disk.nvme0c0n1.out‚166591.70849609375 disk.dm-1.in‚ 108.92578125( proc.memory.availableMB‚ 2939.8828125 disk.rbd1.out‚ 70.2255859375 cpu‚25.584670528163006) proc.memory.percent‚17.48955249786377# memory_percent‚82.0563793182373 disk.sda1.in‚0 disk.dm-1.out‚ 1562.4296875& disk.rbd0.in‚1.759218604441423e+13 disk.dm-0.in‚63098.38671875 disk.sda3.out‚ 1562.4296875) disk./.usagePercent‚38.39527063559994$ disk.dm-0.out‚166591.70849609375 disk.sda1.out‚0$ disk.nvme0c0n1.in‚63098.38671875" disk.nvme0n1.in‚63098.38671875 disk.sda.in‚ 108.92578125 disk.drbd2165.out‚0 disk.sda2.in‚0 disk.rbd1.in‚ 0.53955078125\x}4Ó?/ _wandbruntime‚26829  _runtime‚26829rQÎÊÔ?Ä _step‚1332 & lr-AdamW/pg1‚6.223849918926871e-05 & lr-AdamW/pg2‚6.223849918926871e-05 trainer/global_step‚16199 % _timestamp‚1.7843166450458033e+09 _runtime‚26829.262958822§YÇ3ÏÕ?É _runtime‚26829.262958822 _step‚1332 & lr-AdamW/pg1‚6.223849918926871e-05 & lr-AdamW/pg2‚6.223849918926871e-05 trainer/global_step‚16199 % _timestamp‚1.7843166450458033e+09´ Km_)4Ö?/ _wandbruntime‚26829  _runtime‚26829H4(×?”P G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚87432909 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15214629284346 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 138221805 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 138221811 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 140309241 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15240706530816 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 _step‚1333 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚4672057922419200 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1277052455586304 % _timestamp‚1.7843166450465925e+09 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚45875390 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15240706530816 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚4654807677587456 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 trainer/global_step‚16199 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚247 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚19656444474880 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 _runtime‚26829.263875343 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚4672057922419200 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚4656818561277952 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 138221811 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15213284684024 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 138221805 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚41557519 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚4656818561277952 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1277052858213888 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1257396045647872 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚15239361141248 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 140299268 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15239361141248 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1257396413739008 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚87433156 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚4670022306871802 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 140299268 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚4654807677587456 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 trainer/global_step‚16199 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚247 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚19656444474880 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 _runtime‚26829.263875343 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚4672057922419200 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚4656818561277952 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 138221811 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15213284684024 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 138221805 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚41557519 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚4656818561277952 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1277052858213888 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1257396045647872 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚15239361141248 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 140299268 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15239361141248 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1257396413739008 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚87433156 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚4670022306871802 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 140299268 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚87432909 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15214629284346 G?DevÝØÔt_iceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 138221805 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 138221811 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 140309241 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15240706530816 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 _step‚1333 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚4672057922419200 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1277052455586304 % _timestamp‚1.7843166450465925e+09 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚45875390 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995µ aYñ4Ù?/ _wandbruntime‚26830  _runtime‚26830+!˜$'Ú?žN WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 138230011 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 _runtime‚26830.648681556 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 278547750 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 % _timestamp‚1.7843166464313238e+09 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15241631573504 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚15240286176768 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚231 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1257473900146688 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚4657096562008064 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚4670299742087934 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚87438159 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15215552750704 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 140307746 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚4670301112116336 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 138230017 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚4655085559365632 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚45878328 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 278547750 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 400523264 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1277131146629632 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15214208150270 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚4672338219755008 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15241631573504 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚45878097 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 trainer/global_step‚16199 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15240286176768 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚19657614574080 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 138230017 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚41560062 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 138230011 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 140317733 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚4657096588181504 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 278537757 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 140307746 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚4655085533937664 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚4672336848184832 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1277131547152896 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _step‚1334 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119D×_/)'Û?£N H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚236 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚87438159 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15215552750704 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 140307746 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚4670301112116336 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 138230017 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚4655085559365632 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚45878328 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 278547750 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 400523264 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1277131146629632 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15214208150270 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚4672338219755008 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15241631573504 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚45878097 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 trainer/global_step‚16199 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15240286176768 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚19657614574080 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 138230017 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚41560062 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 138230011 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 140317733 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚4657096588181504 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 278537757 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 140307746 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚4655085533937664 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚4672336848184832 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1277131547152896 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _step‚1334 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 138230011 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 _runtime‚26830.648681556 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 278547750 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 % _timestamp‚1.7843166464313238e+09 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15241631573504 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚15240286176768 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚231 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1257473900146688 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚4657096562008064 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚4670299742087934 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 EDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚4686004274855424 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚87695759 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚46011333 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚4668701741068288 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚4683962632893710 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15259521797132 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15260866397454 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15287022461440 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚4668701766496256 % _timestamp‚1.7843167247158475e+09 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚4670718597783552 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1261270734123008 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 138632105 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 140734993 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15285677071872 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚87696007 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 279367104 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚4686005646418432 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 trainer/global_step‚16249 _step‚1337 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15287022461440 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 138632105 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚243 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚248 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚41684674 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚4686004274855424 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚4686004274855424 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚87695759 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚46011333 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚4668701741068288 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚4683962632893710 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15259521797132 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15260866397454 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15287022461440 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚4668701766496256 % _timestamp‚1.7843167247158475e+09 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚4670718597783552 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1261270734123008 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 138632105 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 140734993 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15285677071872 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚87696007 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 279367104 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚4686005646418432 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 trainer/global_step‚16249 _step‚1337 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15287022461440 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 138632105 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚243 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚248 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚41684674 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚4686004274855424 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚87700953 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 140733498 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15287947504128 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1261347852439552 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚46013741 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚4668979648274432 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 QDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚41687217 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 140743485 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 140743485 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1281065296455168 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 138640311 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚241 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚4670996598513664 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15287947504128 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 279383802 _step‚1338 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 138640317 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 140733498 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚87701194 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 279373809 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15286602107392 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 $ _timestamp‚1.784316726101371e+09 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚19717444015616 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚4670996598513664 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚236 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 trainer/global_step‚16249 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚4686283200621056 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 138640317 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15260445263378 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚4670996624687104 _runtime‚26910.318843807 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚41687212 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15261789863812 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚4684240068109842 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0ºœžñCó?¢N TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚41687212 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15261789863812 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚4684240068109842 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚87700953 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 140733498 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15287947504128 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1261347852439552 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚46013741 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚4668979648274432 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 QDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚41687217 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 140743485 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 140743485 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1281065296455168 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 138640311 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚241 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚4670996598513664 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15287947504128 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 279383802 _step‚1338 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 138640317 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 140733498 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚87701194 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 279373809 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15286602107392 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 $ _timestamp‚1.784316726101371e+09 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚19717444015616 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚4670996598513664 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚236 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 trainer/global_step‚16249 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚4686283200621056 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 138640317 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15260445263378 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚4670996624687104 _runtime‚26910.318843807º öë4ô?/ _wandbruntime‚26910  _runtime‚26910Ñ{.Bõ?Ž _step‚1339 2loss/FRt2/assignment_loss‚0.060519862633485064 4loss/FBt1/assignment_loss‚0.00016137088568681323 1loss/SRqqt1/detection_loss‚0.4535535606415601 % _timestamp‚1.7843167261883104e+09 %loss/total_loss‚4.101016041403748 0loss/FRt1/assignment_loss‚0.8362691799031005 3loss/SRqqt1/assignment_loss‚0.08066085100663199 0loss/FRt1/detection_loss‚0.44261270892019555 .loss/FRt2/detection_loss‚1.222035651825298 /loss/FBt1/detection_loss‚0.3288888972535795 3loss/SRqqt2/assignment_loss‚0.08728444298176975 3loss/FBt2/assignment_loss‚4.887688691435003e-06 1loss/SRqqt2/detection_loss‚0.4775516627012288 0loss/FBt2/detection_loss‚0.11147296496252138 epoch‚40 trainer/global_step‚16249 _runtime‚26910.40524434cOHbö?“ 0loss/FRt1/detection_loss‚0.44261270892019555 .loss/FRt2/detection_loss‚1.222035651825298 /loss/FBt1/detection_loss‚0.3288888972535795 3loss/SRqqt2/assignment_loss‚0.08728444298176975 3loss/FBt2/assignment_loss‚4.887688691435003e-06 1loss/SRqqt2/detection_loss‚0.4775516627012288 0loss/FBt2/detection_loss‚0.11147296496252138 epoch‚40 trainer/global_step‚16249 _runtime‚26910.40524434 _step‚1339 2loss/FRt2/assignment_loss‚0.060519862633485064 4loss/FBt1/assignment_loss‚0.00016137088568681323 1loss/SRqqt1/detection_loss‚0.4535535606415601 % _timestamp‚1.7843167261883104e+09 %loss/total_loss‚4.101016041403748 0loss/FRt1/assignment_loss‚0.8362691799031005 3loss/SRqqt1/assignment_loss‚0.08066085100663199» °¢—”÷?‚(:‰ ĆêÒßôüö gpu.0.gpu‚99.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚38.0 gpu.0.powerWatts‚110.543' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.1086 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ªUAäø?‚(:‘ ĆêÒéõàù) proc.memory.availableMB‚ 2939.83203125 network.recv‚ 6.693646e+06# disk./.usageGB‚573.622875213623 disk.sda.out‚ 1564.41796875 disk.sda2.in‚0$ memory_percent‚82.05668926239014 proc.cpu.threads‚17 disk.dm-0.in‚ 63439.4140625# disk.dm-0.out‚167568.9580078125( disk.nvme0c0n1.out‚167568.9580078125 disk.rbd1.out‚ 77.0927734375 cpu‚25.58460942876318* disk./.usagePercent‚38.496312457236506 disk.drbd2165.in‚0 disk.sda.in‚ 108.92578125 disk.dm-1.out‚ 1564.41796875 disk.sda3.in‚ 108.92578125 disk.sda2.out‚0 disk.rbd1.in‚ 1.32861328125 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd2165.out‚0 disk.dm-1.in‚ 108.92578125 disk.rbd0.out‚0 disk.sda3.out‚ 1564.41796875& disk.nvme0n1.out‚167568.9580078125 network.sent‚ 3.1408825e+07# disk.nvme0c0n1.in‚ 63439.4140625# proc.memory.rssMB‚ 2865.48828125) proc.memory.percent‚17.48955249786377& disk.rbd0.in‚1.759218604441423e+13! disk.nvme0n1.in‚ 63439.4140625–[¹ ù?‚(:• Ó†êÒµÓÞö gpu.0.gpu‚100.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚255.879' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚51.175799999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0û×´ú?‚(: Ó†êÒòüðø( proc.memory.availableMB‚ 2939.8359375 disk.sda.in‚ 108.92578125$ disk.dm-0.out‚167770.59228515625 disk.sda1.out‚0 disk.sda2.out‚0# disk.nvme0c0n1.in‚ 63699.1953125' disk.nvme0n1.out‚167770.59228515625 disk.rbd1.out‚ 191.314453125) proc.memory.percent‚17.48955249786377 network.sent‚ 3.1460143e+07 disk.rbd1.in‚ 2.517578125) disk./.usagePercent‚38.49623283889335$ disk./.usageGB‚573.6216888427734 disk.rbd0.out‚0! disk.nvme0n1.in‚ 63699.1953125 disk.sda1.in‚0 disk.sda3.in‚ 108.92578125 disk.sda3.out‚ 1564.87109375 network.recv‚ 6.69747e+06 disk.dm-0.in‚ 63699.1953125 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.sda2.in‚0# proc.memory.rssMB‚ 2865.48828125 cpu‚25.58461047365884 proc.cpu.threads‚17 disk.sda.out‚ 1564.87109375 disk.dm-1.in‚ 108.92578125 disk.dm-1.out‚ 1564.87109375& disk.rbd0.in‚1.759218604441423e+13) disk.nvme0c0n1.out‚167770.59228515625$ memory_percent‚82.05666542053223D15¹’û?‚(:‡ â†êÒ¶Èòö gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚287.91' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚57.582 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0kZÏü?‚(:‰ â†êÒ¡Ðòù& disk.rbd0.in‚1.759218604441423e+13 disk.dm-0.in‚ 63699.4140625$ memory_percent‚82.05666542053223! disk.dm-0.out‚168200.76953125# proc.memory.rssMB‚ 2865.48828125 disk.sda1.in‚0 disk.rbd0.out‚0 disk.sda2.out‚0 disk.sda.in‚ 108.92578125 disk.dm-1.out‚ 1567.25390625 disk.sda3.out‚ 1567.25390625 disk.rbd1.out‚ 409.412109375) proc.memory.percent‚17.48955249786377 disk.sda1.out‚0 network.sent‚ 3.1466493e+07) disk./.usagePercent‚38.49687208970635 disk.sda3.in‚ 108.92578125 disk.rbd1.in‚ 2.517578125 proc.cpu.threads‚17 disk.drbd2165.out‚0 disk.sda2.in‚0( proc.memory.availableMB‚ 2939.8359375$ disk./.usageGB‚573.6312141418457 disk.sda.out‚ 1567.25390625 disk.dm-1.in‚ 108.92578125 disk.drbd2165.in‚0& disk.nvme0c0n1.out‚168200.76953125$ disk.nvme0n1.out‚168200.76953125 cpu‚25.584589221545325 network.recv‚ 6.700806e+06# disk.nvme0c0n1.in‚ 63699.4140625! disk.nvme0n1.in‚ 63699.4140625-àÁPý?‚(:‘ ñ†êÒ›ÍÙø network.recv‚ 6.703955e+06& disk.rbd0.in‚1.759218604441423e+13 proc.cpu.threads‚17' disk.nvme0n1.out‚168290.23876953125 disk.rbd0.out‚0 disk.dm-0.in‚ 63699.8515625) disk.nvme0c0n1.out‚168290.23876953125) proc.memory.percent‚17.48955249786377$ disk./.usageGB‚574.1609039306641 disk.rbd1.out‚ 592.306640625 disk.sda.in‚ 108.92578125 disk.dm-1.out‚ 1569.14453125 disk.sda1.out‚0! disk.nvme0n1.in‚ 63699.8515625 disk.sda3.in‚ 108.92578125 disk.sda2.in‚0# proc.memory.rssMB‚ 2865.48828125 disk.drbd2165.in‚0$ memory_percent‚82.05423355102539* disk./.usagePercent‚38.532420015873285 disk.dm-1.in‚ 108.92578125$ disk.dm-0.out‚168290.23876953125 disk.sda3.out‚ 1569.14453125# disk.nvme0c0n1.in‚ 63699.8515625 cpu‚25.58458162156479 network.sent‚ 3.1470217e+07 disk.sda.out‚ 1569.14453125 disk.sda1.in‚0 disk.drbd2165.out‚0 disk.sda2.out‚0 disk.rbd1.in‚ 2.517578125' proc.memory.availableMB‚ 2940.234375•òuÈ”þ?‚(:‰ ñ†êÒŸ€ gpu.0.gpu‚83.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚333.089' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚66.6178 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0wb7Ùžÿ?‚(:“ €‡êÒ‹ß÷ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚294.623' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚58.92459999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0±'nÞ€@‚(:Ž €‡êÒ²Æù& disk.rbd0.in‚1.759218604441423e+13 disk.rbd1.out‚792.0146484375) proc.memory.availableMB‚ 2940.21484375 network.sent‚ 3.1476015e+07 disk.sda.out‚ 1571.16796875 disk.dm-1.out‚ 1571.16796875 disk.sda2.out‚0) proc.memory.percent‚17.48955249786377 disk.sda3.out‚ 1571.16796875 disk.drbd2165.in‚0 cpu‚25.584523700367672% disk.nvme0n1.out‚168767.970703125# proc.memory.rssMB‚ 2865.48828125 proc.cpu.threads‚17$ disk./.usageGB‚574.1603050231934 disk.sda.in‚ 108.92578125) disk./.usagePercent‚38.53237982269041 disk.rbd0.out‚0 disk.dm-0.in‚ 63849.0546875" disk.dm-0.out‚168767.970703125 disk.sda2.in‚0$ memory_percent‚82.05435276031494# disk.nvme0c0n1.in‚ 63849.0546875' disk.nvme0c0n1.out‚168767.970703125 disk.sda1.in‚0 disk.sda1.out‚0 network.recv‚ 6.707362e+06 disk.sda3.in‚ 108.92578125 disk.drbd2165.out‚0! disk.nvme0n1.in‚ 63849.0546875 disk.rbd1.in‚ 2.517578125 disk.dm-1.in‚ 108.92578125O Üu4@/ _wandbruntime‚26988  _runtime‚26988Á˜cÉ‚@à _step‚1340 & lr-AdamW/pg1‚5.910215547293491e-05 & lr-AdamW/pg2‚5.910215547293491e-05 trainer/global_step‚16299 $ _timestamp‚1.784316804368764e+09 _runtime‚26988.5890604448³à#΃@È $ _timestamp‚1.784316804368764e+09 _runtime‚26988.589060444 _step‚1340 & lr-AdamW/pg1‚5.910215547293491e-05 & lr-AdamW/pg2‚5.910215547293491e-05 trainer/global_step‚16299¼ :û!4„@/ _wandbruntime‚26988  _runtime‚26988(ˆ&(…@“P ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚4684618660462592 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚19775831706112 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚4697901588887328 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _runtime‚26988.589981153 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚4684618634289152 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15333338392064 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚262 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚4699950627291648 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 141150772 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚4697902958915618 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15331993002496 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚41811824 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚41811819 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚19775797169664 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15305758910240 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚4699950627291648 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1265144686415872 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚16299 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 139042411 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚257 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚4682595829977088 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008Cbb5ž†@˜P G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 139042405 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15333338392064 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1284920483585536 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 141150772 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚4684618660462592 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 139042411 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚87958246 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚4684618634289152 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 $ _timestamp‚1.784316804372719e+09 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚4699951998854656 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚46146427 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1284920886213120 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 _step‚1341 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚4697902958915618 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15331993002496 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚41811824 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚41811819 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚19775797169664 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15305758910240 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚4699950627291648 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1265144686415872 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚16299 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 H?DeviceStatsMonit|¹‹€or.on_train_batch_start/segment.large_pool.peak‚1175 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 139042411 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚257 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚4682595829977088 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚4684618660462592 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚19775831706112 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚4697901588887328 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _runtime‚26988.589981153 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚4684618634289152 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15333338392064 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚262 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚4699950627291648 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 141150772 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 J8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 280219854 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚4700230924627456 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15306682376486 _step‚1342 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚4684896661192704 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15332918038016 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 280209861 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚4682873737183232 _runtime‚26989.978203957 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15334263434752 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1284999562069504 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚255 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚87963511 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚4698181764160152 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚250 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚4698180394131750 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚4684896635019264 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚41814367 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 280209861 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚4700230924627456 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚46149399 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 139050611 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 139050617 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚19777021154816 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7843168057607577e+09 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 141169237 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚4684896661192704 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚46149149 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚16299 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 280219854 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1265222172823552 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚41814362 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1265222540914688 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0Òæ®)'‰@£N RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚16299 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 280219854 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1265222172823552 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚41814362 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1265222540914688 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚15332918038016 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36626432 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 141169237 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚4700229553057280 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 139050617 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 141159250 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 280219854 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚4700230924627456 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15306682376486 _step‚1342 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚4684896661192704 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15332918038016 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 280209861 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚4682873737183232 _runtime‚26989.978203957 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15334263434752 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1284999562069504 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚255 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚87963511 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚4698181764160152 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚250 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚4698180394131750 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚4684896635019264 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚41814367 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 280209861 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚4700230924627456 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚46149399 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 139050611 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 139050617 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚19777021154816 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7843168057607577e+09 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 141169237 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚4684896661192704 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚46149149 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640¾ £N¶24Š@/ _wandbruntime‚26990  _runtime‚26990ûá¨{‹@‹ 1loss/FRt2/assignment_loss‚0.08502360627621035 3loss/SRqqt1/assignment_loss‚0.06241052547396401 .loss/FBt2/detection_loss‚0.072759522557914 trainer/global_step‚16299 0loss/FRt1/assignment_loss‚0.8396422140067443 3loss/FBt1/assignment_loss‚0.0009043559224121762 4loss/FBt2/assignment_loss‚0.00037040439487444425 .loss/FRt2/detection_loss‚1.185085454107735 1loss/SRqqt1/detection_loss‚0.4167267373327307 1loss/SRqqt2/detection_loss‚0.4390853924718929 _step‚1343 /loss/FBt1/detection_loss‚0.2384311838274142 %loss/total_loss‚3.918859926364363 epoch‚40 % _timestamp‚1.7843168058477852e+09 3loss/SRqqt2/assignment_loss‚0.11178138619597958 /loss/FRt1/detection_loss‚0.4666391437964914 _runtime‚26990.064725396-L‚Œ@ % _timestamp‚1.7843168058477852e+09 3loss/SRqqt2/assignment_loss‚0.11178138619597958 /loss/FRt1/detection_loss‚0.4666391437964914 _runtime‚26990.064725396 1loss/FRt2/assignment_loss‚0.08502360627621035 3loss/SRqqt1/assignment_loss‚0.06241052547396401 .loss/FBt2/detection_loss‚0.072759522557914 trainer/global_step‚16299 0loss/FRt1/assignment_loss‚0.8396422140067443 3loss/FBt1/assignment_loss‚0.0009043559224121762 4loss/FBt2/assignment_loss‚0.00037040439487444425 .loss/FRt2/detection_loss‚1.185085454107735 1loss/SRqqt1/detection_loss‚0.4167267373327307 1loss/SRqqt2/detection_loss‚0.4390853924718929 _step‚1343 /loss/FBt1/detection_loss‚0.2384311838274142 %loss/total_loss‚3.918859926364363 epoch‚40¿ –¼iSž@‚(:“ ‡êÒÿœ·ø gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚340.097' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚68.01939999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‹ØkäŽ@‚(: ‡êÒÛå©ù disk.sda.in‚ 108.92578125 disk.drbd2165.out‚0 disk.sda2.in‚0# memory_percent‚82.0561170578003) proc.memory.percent‚17.48955249786377 proc.cpu.threads‚17$ disk./.usageGB‚574.1674041748047& disk.rbd0.in‚1.759218604441423e+13 disk.sda1.in‚0 disk.sda3.in‚ 108.92578125# disk.nvme0c0n1.in‚ 63849.2734375) disk./.usagePercent‚38.53285625271163 disk.rbd1.in‚ 2.591796875 disk.rbd0.out‚0 disk.sda3.out‚ 1572.74609375' disk.nvme0c0n1.out‚168780.021484375 disk.dm-1.out‚ 1572.74609375% disk.nvme0n1.out‚168780.021484375 cpu‚25.58450499787645! disk.rbd1.out‚975.58056640625 disk.dm-1.in‚ 108.92578125" disk.dm-0.out‚168780.021484375 disk.sda2.out‚0 disk.sda1.out‚0! disk.nvme0n1.in‚ 63849.2734375 network.sent‚ 3.1480325e+07 network.recv‚ 6.710598e+06 disk.sda.out‚ 1572.74609375 disk.dm-0.in‚ 63849.2734375 disk.drbd2165.in‚0) proc.memory.availableMB‚ 2939.92578125# proc.memory.rssMB‚ 2865.48828125V˜öŸ@‚(:” ž‡êÒë‚÷ gpu.0.gpu‚90.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚275.038' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚55.007600000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0§ÇNÑ@‚(: ž‡êÒÎß›ú disk.sda1.out‚0) proc.memory.percent‚17.48955249786377 network.recv‚ 6.715449e+06& disk.rbd0.in‚1.759218604441423e+13 network.sent‚ 3.1545022e+07 disk.rbd1.in‚ 2.591796875 disk.rbd1.out‚ 1198.0703125 disk.dm-0.in‚ 63849.7890625) proc.memory.availableMB‚ 2940.16015625 disk.sda3.in‚ 108.92578125 disk.sda3.out‚ 1573.1953125 disk.drbd2165.in‚0) disk.nvme0c0n1.out‚168792.38818359375! disk.nvme0n1.in‚ 63849.7890625 disk.rbd0.out‚0# proc.memory.rssMB‚ 2865.48828125 cpu‚25.58451184425747 disk.sda1.in‚0 disk.drbd2165.out‚0# disk.nvme0c0n1.in‚ 63849.7890625 disk.sda2.in‚0 disk.sda.in‚ 108.92578125$ disk./.usageGB‚574.1742210388184 disk.sda2.out‚0 disk.sda.out‚ 1573.1953125 disk.dm-1.in‚ 108.92578125$ memory_percent‚82.05468654632568 proc.cpu.threads‚17* disk./.usagePercent‚38.533313738175316' disk.nvme0n1.out‚168792.38818359375 disk.dm-1.out‚ 1573.1953125$ disk.dm-0.out‚168792.38818359375?C·ÑŸ‘@‚(:” ­‡êÒ÷°öö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚290.567' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚58.113400000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÀäW’@‚(:Ž ­‡êÒÙÄÎù# disk.nvme0c0n1.in‚ 63849.9921875( disk.nvme0c0n1.out‚168804.3056640625$ memory_percent‚82.05547332763672 disk.dm-1.in‚ 108.92578125 disk.sda3.out‚ 1573.64453125 disk.drbd2165.in‚0 disk.sda.out‚ 1573.64453125 disk.sda1.in‚0 disk.dm-1.out‚ 1573.64453125# disk.dm-0.out‚168804.3056640625 network.sent‚ 3.1549033e+07& proc.memory.availableMB‚ 2940.03125! disk.nvme0n1.in‚ 63849.9921875 network.recv‚ 6.718635e+06$ disk./.usageGB‚574.1788101196289 disk.sda.in‚ 108.92578125 disk.dm-0.in‚ 63849.9921875 disk.sda1.out‚0# proc.memory.rssMB‚ 2865.48828125 cpu‚25.58454931484041) disk./.usagePercent‚38.53362171523901! disk.rbd1.out‚1430.1494140625) proc.memory.percent‚17.48955249786377 proc.cpu.threads‚17 disk.rbd0.out‚0 disk.sda2.in‚0 disk.sda2.out‚0& disk.rbd0.in‚1.759218604441423e+13 disk.drbd2165.out‚0 disk.rbd1.in‚ 2.591796875 disk.sda3.in‚ 108.92578125& disk.nvme0n1.out‚168804.3056640625“}g»ž“@‚(:“ ¼‡êÒ¹¦÷ gpu.0.gpu‚99.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚296.216' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚59.24320000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0èÙL'v”@‚(:ƒ ¼‡êÒ‚Ø”ú cpu‚25.58450139520922 network.sent‚ 3.1554114e+07 network.recv‚ 6.722226e+06' disk.nvme0c0n1.out‚168813.572265625 disk.sda2.out‚0 disk.sda3.in‚ 108.92578125 disk.drbd2165.out‚0 disk.nvme0n1.in‚ 63975.484375% disk.nvme0n1.out‚168813.572265625$ memory_percent‚82.05361366271973) proc.memory.percent‚17.48955249786377 proc.cpu.threads‚17 disk.drbd2165.in‚0 disk.sda2.in‚0 disk.sda.in‚ 108.92578125 disk.sda1.in‚0 disk.rbd1.out‚1644.919921875# proc.memory.rssMB‚ 2865.48828125* disk./.usagePercent‚38.534147042699274# disk.A™t˜/.usageGB‚574.186637878418 disk.dm-1.in‚ 108.92578125 disk.rbd0.out‚0 disk.dm-0.in‚ 63975.484375" disk.dm-0.out‚168813.572265625" disk.nvme0c0n1.in‚ 63975.484375( proc.memory.availableMB‚ 2940.3359375 disk.sda.out‚ 1574.296875& disk.rbd0.in‚1.759218604441423e+13 disk.sda1.out‚0 disk.dm-1.out‚ 1574.296875 disk.sda3.out‚ 1574.296875 disk.rbd1.in‚ 2.591796875ª‰ûM”•@‚(:‰ ˇêÒØÉ–ù gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚282.783' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚56.5566 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Æ•7æ–@‚(:Œ ˇêÒ›¶çù disk.dm-0.in‚ 63975.703125# disk.dm-0.out‚168829.2744140625 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.sda2.in‚0 disk.rbd1.in‚ 2.62353515625& disk.rbd0.in‚1.759218604441423e+13 disk.rbd0.out‚0 network.sent‚ 3.1559142e+07 disk.sda3.in‚ 108.92578125 disk.sda3.out‚ 1574.671875( disk.nvme0c0n1.out‚168829.2744140625" disk.nvme0c0n1.in‚ 63975.703125" disk.rbd1.out‚1725.26611328125 disk.sda2.out‚0# proc.memory.rssMB‚ 2865.48828125 cpu‚25.584455306573783 proc.cpu.threads‚17 disk.sda.in‚ 108.92578125) proc.memory.availableMB‚ 2940.30078125) disk./.usagePercent‚38.53444605949927$ disk./.usageGB‚574.1910934448242 disk.dm-1.in‚ 108.92578125 disk.drbd2165.out‚0 disk.nvme0n1.in‚ 63975.703125) proc.memory.percent‚17.48955249786377 network.recv‚ 6.725432e+06 disk.sda.out‚ 1574.671875 disk.dm-1.out‚ 1574.671875& disk.nvme0n1.out‚168829.2744140625$ memory_percent‚82.05382823944092,èŽI4—@/ _wandbruntime‚27068  _runtime‚27068‚%ʘ@Ä & lr-AdamW/pg1‚5.756156661701289e-05 & lr-AdamW/pg2‚5.756156661701289e-05 trainer/global_step‚16349 % _timestamp‚1.7843168840368533e+09 _runtime‚27068.254082363 _step‚1344µ¬¾:Ï™@É & lr-AdamW/pg1‚5.756156661701289e-05 & lr-AdamW/pg2‚5.756156661701289e-05 trainer/global_step‚16349 % _timestamp‚1.7843168840368533e+09 _runtime‚27068.254082363 _step‚1344À ¬Zëú4š@/ _wandbruntime‚27068  _runtime‚27068 žô(›@”P QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 141576524 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚4713896979727872 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚4713896979727872 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 141586497 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚4698518696968192 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚4711841914909236 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 141586497 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1288854926983680 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 400530432 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚41938969 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚46281471 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 139452711 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15379654322688 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 281029229 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚4698518670794752 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 281039208 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15378308933120 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚88220683 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚19835552092672 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15379654322688 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1288854526453248 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚32439296 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820h~L°(œ@™P UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 139452711 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15379654322688 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 281029229 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚4698518670794752 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 281039208 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15378308933120 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚88220683 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚19835552092672 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15379654322688 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1288854526453248 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚32439296 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 141576524 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚4713896979727872 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚4713896979727872 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 141586497 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚4698518696968192 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚4711841914909236 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 141586497 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1288854926983680 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 400530432 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚41938969 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚46281471 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0Á ž9 4@/ _wandbruntime‚27069  _runtime‚270698‹â&ž@žN E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚4714175905493504 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 281055906 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 139460911 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 139460917 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 141594989 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 139460917 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1269096861298688 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 141585002 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 281055906 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚236 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚231 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚46284167 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15380579365376 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15380579365376 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 139460911 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚88225679 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚4696767800664064 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 281045913 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚32432128 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 trainer/global_step‚16349 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚4698796671524864 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 400523264 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 141594989 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 _step‚1346 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚15379233968640 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚4696767826092032 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15352919489594 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚4712122090182060 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚4698796697698304 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚4698796671524864 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 281045913 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_endåÇaÃ"/allocated_bytes.small_pool.peak‚ 1680116736 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚88225915 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0%‘Ìx)'Ÿ@£N K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 141585002 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 281055906 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚236 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚231 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚46284167 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15380579365376 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15380579365376 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 139460911 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚88225679 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚4696767800664064 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 281045913 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚32432128 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 trainer/global_step‚16349 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚4698796671524864 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 400523264 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 141594989 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 _step‚1346 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚15379233968640 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚4696767826092032 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15352919489594 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚4712122090182060 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚4698796697698304 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚4698796671524864 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 281045913 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚88225915 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚4714175905493504 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 281055906 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 139460911 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 139460917 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 141594989 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 139460917 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1269096861298688 ‰d„4 @/ _wandbruntime‚27069  _runtime‚27069oœ¡@ 3loss/FBt2/assignment_loss‚0.0010315188138699347 0loss/FBt1/detection_loss‚0.10758749035958023 epoch‚40 _step‚1347 0loss/FRt1/assignment_loss‚0.8349373002958654 4loss/SRqqt1/assignment_loss‚0.034783331978879485 3loss/FBt1/assignment_loss‚1.115181913838393e-07 0loss/FRt1/detection_loss‚0.41659103888127347 1loss/SRqqt1/detection_loss‚0.3330581830365077 1loss/SRqqt2/detection_loss‚0.4238185473680885 %loss/total_loss‚3.566486252595409 0loss/FRt2/assignment_loss‚0.0685824781893419 3loss/SRqqt2/assignment_loss‚0.10416098965840319 /loss/FRt2/detection_loss‚1.1340482374647622 0loss/FBt2/detection_loss‚0.10788702503064532 trainer/global_step‚16349 % _timestamp‚1.7843168855088909e+09 _runtime‚27069.725831209žãôý¢@” %loss/total_loss‚3.566486252595409 0loss/FRt2/assignment_loss‚0.0685824781893419 3loss/SRqqt2/assignment_loss‚0.10416098965840319 /loss/FRt2/detection_loss‚1.1340482374647622 0loss/FBt2/detection_loss‚0.10788702503064532 trainer/global_step‚16349 % _timestamp‚1.7843168855088909e+09 _runtime‚27069.725831209 3loss/FBt2/assignment_loss‚0.0010315188138699347 0loss/FBt1/detection_loss‚0.10758749035958023 epoch‚40 _step‚1347 0loss/FRt1/assignment_loss‚0.8349373002958654 4loss/SRqqt1/assignment_loss‚0.034783331978879485 3loss/FBt1/assignment_loss‚1.115181913838393e-07 0loss/FRt1/detection_loss‚0.41659103888127347 1loss/SRqqt1/detection_loss‚0.3330581830365077 1loss/SRqqt2/detection_loss‚0.4238185473680885à Àˆr‚‘£@‚(:† Ú‡êÒÜÄîø gpu.0.gpu‚1.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚330.145' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚66.029 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 %.ó¤@‚(:† Ú‡êÒ¢êîù disk.dm-1.out‚ 1574.91015625& disk.rbd0.in‚1.759218604441423e+13 disk.rbd0.out‚0 network.recv‚ 6.728682e+06 disk.rbd1.in‚ 2.62353515625 disk.dm-1.in‚ 108.92578125 disk.drbd2165.out‚0 disk.nvme0n1.in‚ 63975.90625 disk.sda2.out‚0# proc.memory.rssMB‚ 2865.48828125 cpu‚25.584473009336797$ disk./.usageGB‚574.1977386474609 disk.dm-0.in‚ 63975.90625 disk.sda3.out‚ 1574.91015625 disk.drbd2165.in‚0! disk.nvme0c0n1.in‚ 63975.90625$ disk.nvme0n1.out‚168837.26953125% proc.memory.availableMB‚ 2940.0625 network.sent‚ 3.1564154e+07* disk./.usagePercent‚38.534892024623915" disk.rbd1.out‚1725.26611328125$ memory_percent‚82.05528259277344 disk.sda.in‚ 108.92578125 disk.sda.out‚ 1574.91015625& disk.nvme0c0n1.out‚168837.26953125! disk.dm-0.out‚168837.26953125 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda3.in‚ 108.92578125 disk.sda2.in‚0) proc.memory.percent‚17.48955249786377 proc.cpu.threads‚17|ÿ²ŒŸ¥@‚(:” é‡êÒ°ªûö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚263.954' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚52.790800000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0?º7¦@‚(:Š é‡êÒ”›ù# proc.memory.rssMB‚ 2865.48828125 disk.dm-1.out‚ 1575.18359375 disk.sda3.in‚ 108.92578125 disk.sda1.out‚0 disk.drbd2165.in‚0& disk.nvme0c0n1.out‚168849.56640625 disk.nvme0n1.in‚ 63976.328125" disk.nvme0c0n1.in‚ 63976.328125# memory_percent‚82.0540189743042) proc.memory.availableMB‚ 2940.26953125 network.recv‚ 6.733154e+06 disk.sda.in‚ 108.92578125 disk.rbd0.out‚0 disk.dm-0.in‚ 63976.328125 disk.sda3.out‚ 1575.18359375 cpu‚25.584427550794267 network.sent‚ 3.1627679e+07$ disk./.usageGB‚574.2045555114746 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda.out‚ 1575.18359375 disk.dm-1.in‚ 108.92578125 disk.sda1.in‚0" disk.rbd1.out‚1725.79052734375) proc.memory.percent‚17.48955249786377 proc.cpu.threads‚17 disk.rbd1.in‚ 2.62353515625& disk.rbd0.in‚1.759218604441423e+13$ disk.nvme0n1.out‚168849.56640625( disk./.usagePercent‚38.5353495100876! disk.dm-0.out‚168849.56640625 disk.drbd2165.out‚0¾5Ý,’§@‚(:‡ ø‡êÒåüøø gpu.0.gpu‚94.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚40.0 gpu.0.powerWatts‚137.96' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚27.592 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0S|ͨ@‚(:Ž ø‡ê҆ذú disk.sda2.in‚0 disk.sda.out‚ 1576.59375! disk.nvme0n1.in‚ 64171.3984375 cpu‚25.584452313281734 network.sent‚ 3.1634004e+07 disk.sda1.in‚0 disk.sda3.in‚ 108.92578125) disk.nvme0c0n1.out‚168868.89111328125 disk.sda2.out‚0' proc.memory.availableMB‚ 2940.515625) proc.memory.percent‚17.48955249786377 disk.dm-1.in‚ 108.92578125& disk.rbd0.in‚1.759218604441423e+13 disk.dm-0.in‚ 64171.3984375 disk.sda.in‚ 108.92578125 disk.dm-1.out‚ 1576.59375 disk.rbd0.out‚0# disk.nvme0c0n1.in‚ 64171.3984375# proc.memory.rssMB‚ 2865.48828125$ disk.dm-0.out‚168868.89111328125 disk.drbd2165.in‚0 network.recv‚ 6.736632e+06" disk.rbd1.out‚1725.79052734375 disk.sda3.out‚ 1576.59375 disk.drbd2165.out‚0$ memory_percent‚82.05251693725586* disk./.usagePercent‚38.535970072350594$ disk./.usageGB‚574.2138023376465 disk.rbd1.in‚ 2.62353515625 disk.sda1.out‚0' disk.nvme0n1.out‚168868.89111328125 proc.cpu.threads‚17`˜û8Ÿ©@‚(:” ‡ˆêÒ©–¨÷ gpu.0.gpu‚88.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚290.369' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚58.073800000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ËPó¤ ª@‚(:€ ‡ˆêÒ°Ãœú! disk.nvme0n1.in‚ 64171.8203125 disk.drbd2165.in‚0 proc.cpu.threads‚17# disk.dm-0.out‚168889.7158203125 disk.sda1.in‚0 disk.sda2.in‚0 disk.dm-1.in‚ 108.92578125& disk.rbd0.in‚1.759218604441423e+13 disk.dm-0.in‚ 64171.8203125# disk.nvme0c0n1.in‚ 64171.8203125 disk.sda3.in‚ 108.92578125 disk.sda3.out‚1579.75( proc.memory.availableMB‚ 2940.0234375# proc.memory.rssMB‚ 2865.48828125 network.sent‚ 3.1637719e+07) disk./.usagePercent‚38.53667614113003" disk.rbd1.out‚1725.86865234375 disk.rbd0.out‚0 disk.sda1.out‚0$ disk./.usageGB‚574.2243232727051& disk.nvme0n1.out‚168889.7158203125 disk.sda2.out‚0 disk.sda.out‚1579.75 disk.drbd2165.out‚0 network.recv‚ 6.73973e+06 disk.rbd1.in‚ 2.62353515625 disk.sda.in‚ 108.92578125 disk.dm-1.out‚1579.75( disk.nvme0c0n1.out‚168889.7158203125$ memory_percent‚82.05552101135254) proc.memory.percent‚17.48955249786377 cpu‚25.58445201215577ȸ²”«@‚(:‰ –ˆêÒè•ÿö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚296.693' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚59.3386 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Œ"ˆ ¬@‚(:‚ –ˆêÒ†Ôú disk.sda.in‚ 108.92578125& disk.rbd0.in‚1.759218604441423e+13 disk.drbd2165.in‚0 disk.sda2.out‚0 disk.dm-1.in‚ 108.92578125 network.recv‚ 6.743137e+06 disk.sda1.out‚0) proc.memory.percent‚17.48955249786377 network.sent‚ 3.1642851e+07 disk.dm-0.in‚ 64172.03125 disk.nvme0n1.in‚ 64172.03125 disk.rbd0.out‚0$ memory_percent‚82.05704689025879! disk.nvme0c0n1.in‚ 64172.03125) disk./.usagePercent‚38.53675038331496 disk.sda3.out‚ 1583.1796875 disk.drbd2165.out‚0# disk.nvme0n1.out‚168914.2109375 cpu‚25.584497606245947 disk.sda3.in‚ 108.92578125% disk.nvme0c0n1.out‚168914.2109375 disk.sda2.in‚0 disk.sda.out‚ 1583.1796875# proc.memory.rssMB‚ 2865.48828125 disk.dm-0.out‚168914.2109375 disk.rbd1.in‚ 2.62353515625" disk.rbd1.out‚1725.86865234375 disk.dm-1.out‚ 1583.1796875 proc.cpu.threads‚17( proc.memory.availableMB‚ 2939.7734375$ disk./.usageGB‚574.2254295349121 disk.sda1.in‚0Å1¡…4­@/ _wandbruntime‚27147  _runtime‚27147ô¬«ÅÊ®@Ä & lr-AdamW/pg1‚5.603949875584979e-05 & lr-AdamW/pg2‚5.603949875584979e-05 trainer/global_step‚16399 % _timestamp‚1.7843169634836898e+09 _runtime‚27147.700918397 _step‚1348¨Yçϯ@É _step‚1348 & lr-AdamW/pg1‚5.603949875584979e-05 & lr-AdamW/pg2‚5.603949875584979e-05 trainer/global_step‚16399 % _timestamp‚1.7843169634836898e+09 _runtime‚27147.700918397Ä ¾‚0#4°@/ _wandbruntime‚27147  _runtime‚27147ǃ )(±@“P H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 _step‚1349 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 trainer/global_step‚16399 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 139863005 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15398233136456 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚4712418707300352 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚4725783610959434 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 _runtime‚27147.70215379 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1272893695275008 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚4727844703727104 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚4712418733473792 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 142012249 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1292788696314368 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 281875260 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚46416300 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15424624863744 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1272893327183872 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚4710384033222656 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚4727844703727104 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚19895403666944 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 142002276 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚252 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚42066124 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚88482671 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 139863011 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚19895369130496 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 % _timestamp‚1.7843169634848058e+09 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15425970253312 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚42066119 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 139863005 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 142012249 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚4712418733473792 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚247 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚15424624863744 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚46416547 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 142002276 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 281865281 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1292789098941952 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15399577736778 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚88482419 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚4727843332164096 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15425970253312 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚4710384007794688 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 139863011 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0CÜo®Y ²@˜P =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚4727843332164096 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15425970253312 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚4710384007794688 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 139863011 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 _step‚1349 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 trainer/global_step‚16399 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 139863005 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15398233136456 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚4712418707300352 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚4725783610959434 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 _runtime‚27147.70215379 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1272893695275008 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚4727844703727104 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚4712418733473792 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 142012249 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1292788696314368 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 281875260 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚46416300 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15424624863744 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1272893327183872 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚4710384033222656 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚4727844703727104 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚19895403666944 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 142002276 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚252 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚42066124 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚88482671 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 139863011 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚19895369130496 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 % _timestamp‚1.7843169634848058e+09 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15425970253312 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚42066119 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 139863005 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 142012249 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚4712418733473792 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚247 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚15424624863744 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚46416547 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 142002276 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 281865281 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1292789098941952 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15399577736778 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚88482419Å ý±u’4³@/ _wandbruntime‚27149  _runtime‚27149ÝÁ,S#'´@N XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚4712696708030464 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚42068662 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚16399 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 142020741 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚46419186 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚46418948 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚88487610 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1292867776592384 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 _step‚1350 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1272971181682688 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚4712696734203904 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 142020741 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 281891958 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 281881965 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15426895296000 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 281891958 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1272970813591552 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 142010754 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 139871217 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚88487853 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 281881965 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚238 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 139871217 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚19896594909696 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚4726061046175566 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 $ _timestamp‚1.784316964871088e+09 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚4728123629499904 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚19896560380416 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15426895296000 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚4710661889572864 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 142010754 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15399156602702 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1292867373971968 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440ÃÌð('µ@¢N TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚4728123629499904 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚88487610 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1292867776592384 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 _step‚1350 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1272971181682688 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚4712696734203904 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 142020741 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 281891958 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 281881965 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15426895296000 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 281891958 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1272970813591552 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 142010754 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 139871217 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚88487853 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 281881965 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚238 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 139871217 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚19896594909696 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚4726061046175566 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 $ _timestamp‚1.784316964871088e+09 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚4728123629499904 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚19896560380416 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15426895296000 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚4710661889572864 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 142010754 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15399156602702 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1292867373971968 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚4712696708030464 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚42068662 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚16399 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 142020741 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚46419186 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚46418948 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 E&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 *"Purity/2FRt*SRqqt2FBt/event_purity‚0.5 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7142857313156128 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9130856394767761 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7177000641822815 7"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.66620272397995 +jet/accuracy_1_of_1‚0.28231921792030334 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9905714988708496 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5144564509391785 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7607119083404541 *jet/accuracy_1_of_2‚0.7300025224685669 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.6875 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.8564814329147339 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9936507940292358 particle/accuracy_1_of_2‚1 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.13504526019096375 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 particle/accuracy_2_of_5‚1 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.23857121169567108 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6873555183410645 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.024530449882149696 *jet/accuracy_1_of_3‚0.9067254066467285 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9876719117164612 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6313541531562805 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7824074029922485 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5073119401931763 6 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.984964907169342 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6414054036140442 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.13393375277519226 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 particle/accuracy_1_of_5‚1 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9768246412277222 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.09503225237131119 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.42500001192092896 'particle/f_score‚0.5960361361503601 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6568373441696167 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.3333333432674408 /particle/accuracy_3_of_4‚0.9974937438964844 (particle/accuracy‚0.8294812440872192 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6217145919799805 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.8500000238418579 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.12468050420284271 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.02976994775235653 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.6410430669784546 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.8500000238418579 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.05000000074505806 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 . Purity/2FRt0SRqqt1FBt/FRt_purity‚ 0.5390625 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9722036719322205 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4542064666748047 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.03283771499991417 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.1469258815050125 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 epoch‚40 6 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.982394278049469 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9841269254684448 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.12770673632621765 8"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.826911211013794 particle/accuracy_3_of_3‚1 +particle/sensitivity‚0.5858088731765747 *particle/specificity‚0.896125316619873 _runtime‚27176.483249014 - Purity/0FRt2SRqqt*FBt/FBt_purity‚0.734375 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.5185185670852661 - Purity/0FRt2SRqqt1FBt/FBt_purity‚0.734375 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7424664497375488 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.18019475042819977 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 +jet/accuracy_2_of_2‚0.11707030236721039 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.5657446980476379 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8333333134651184 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 particle/accuracy_2_of_2‚1 particle/accuracy_3_of_5‚1 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6829236149787903 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.14112237095832825 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6617327928543091 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9930846095085144 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.5577718019485474 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.02266469970345497 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9959579706192017 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.21083688735961914 8"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.756297767162323 particle/accuracy_4_of_5‚1 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6344172358512878 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.14650291204452515 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9943969249725342 8"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.742336630821228 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9913582801818848 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6068782806396484 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.14648228883743286 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 particle/accuracy_1_of_3‚1 *loss/val_total_loss‚12.143552635193004 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9852820038795471 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.11239825189113617 ,jet/accuracy_4_of_4‚0.013899499550461769 8"Purity/*FRt1SRqqt*FBt/event_purity‚0.345355361700058 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6927808523178101 +jet/accuracy_2_of_3‚0.46902501583099365 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.26923078298568726 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.21485713124275208 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.10612355172634125 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.13812248408794403 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9906229376792908 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6687664985656738 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6569045186042786 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6868550777435303 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9810190796852112 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4495686888694763 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.5862537622451782 ."Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.53125 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9870997667312622 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.02762816660106182 particle/accuracy_2_of_4‚1 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.4685831665992737 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9705640077590942 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.48254531621932983 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.6995465159416199 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6719503998756409 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7824074029922485 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7303290963172913 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.13423828780651093 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 jet/accuracy_3_of_5‚1 jet/accuracy_4_of_5‚0 $ _timestamp‚1.784316992265454e+09 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 /"Purity/2FRt0SRqqt1FBt/event_purity‚0.203125 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 particle/accuracy_1_of_1‚1 /particle/accuracy_4_of_4‚0.9405004382133484 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.7054988741874695 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.09595829993486404 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.8653846383094788 particle/accuracy_5_of_5‚1 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.2870370149612427 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7393242716789246 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.018152717500925064 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.8965981602668762 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.7070673704147339 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.00958407111465931 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 8"Purity/0FRt*SRqqt0FBt/event_purity‚0.593932569026947 6 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.966495156288147 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.13423828780651093 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.005986332893371582 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.7770084142684937 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9774846434593201 trainer/global_step‚16399 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.1336446851491928 8"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.636402428150177 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.09666667133569717 7validation_average_jet_accuracy‚0.32501763105392456 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.22797852754592896 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2584720253944397 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 particle/accuracy_2_of_3‚1 jet/accuracy_1_of_5‚1 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6571688055992126 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6930657625198364 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0032324804924428463 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.02266469970345497 *jet/accuracy_1_of_4‚0.9622095823287964 *jet/accuracy_2_of_4‚0.7000592947006226 _step‚1352 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6742002964019775 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6087103486061096 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7203326225280762 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.12867894768714905 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.1978885382413864 )jet/accuracy_3_of_4‚0.043703593313694 jet/accuracy_2_of_5‚1 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.1765308976173401 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.7884081602096558 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.7592592835426331 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9895829558372498 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.14449027180671692 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.5894553661346436 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.18088912963867188 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.7592592835426331 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.05000000074505806 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.3333333432674408 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.10787402093410492 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.03013569489121437 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.032385796308517456 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.1276121735572815 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 jet/accuracy_5_of_5‚0 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8377842903137207 8"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.622153103351593 6 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.976256787776947 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6068782806396484 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7208617329597473 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.49958333373069763 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.35452064871788025 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 particle/accuracy_1_of_4‚1 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.5714285969734192 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.2870370149612427 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.44907405972480774 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9824985265731812 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.12117373198270798 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9851381182670593 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.10283532738685608 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129ZÎß_.>¿@¨| 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.18019475042819977 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 +jet/accuracy_2_of_2‚0.11707030236721039 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.5657446980476379 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8333333134651184 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 particle/accuracy_2_of_2‚1 particle/accuracy_3_of_5‚1 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6829236149787903 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.14112237095832825 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6617327928543091 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9930846095085144 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.5577718019485474 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.02266469970345497 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9959579706192017 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.21083688735961914 8"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.756297767162323 particle/accuracy_4_of_5‚1 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6344172358512878 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.14650291204452515 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9943969249725342 8"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.742336630821228 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9913582801818848 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6068782806396484 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.14648228883743286 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 particle/accuracy_1_of_3‚1 *loss/val_total_loss‚12.143552635193004 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9852820038795471 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.11239825189113617 ,jet/accuracy_4_of_4‚0.013899499550461769 8"Purity/*FRt1SRqqt*FBt/event_purity‚0.345355361700058 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6927808523178101 +jet/accuracy_2_of_3‚0.46902501583099365 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.26923078298568726 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.21485713124275208 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.10612355172634125 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.13812248408794403 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9906229376792908 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6687664985656738 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6569045186042786 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6868550777435303 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9810190796852112 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4495686888694763 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.5862537622451782 ."Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.53125 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9870997667312622 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.02762816660106182 particle/accuracy_2_of_4‚1 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.4685831665992737 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9705640077590942 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.48254531621932983 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.6995465159416199 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6719503998756409 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7824074029922485 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7303290963172913 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.13423828780651093 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 jet/accuracy_3_of_5‚1 jet/accuracy_4_of_5‚0 $ _timestamp‚1.784316992265454e+09 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 /"Purity/2FRt0SRqqt1FBt/event_purity‚0.203125 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 particle/accuracy_1_of_1‚1 /particle/accuracy_4_of_4‚0.9405004382133484 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.7054988741874695 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.09595829993486404 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.8653846383094788 particle/accuracy_5_of_5‚1 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.2870370149612427 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7393242716789246 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.018152717500925064 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.8965981602668762 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.7070673704147339 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.00958407111465931 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 8"Purity/0FRt*SRqqt0FBt/event_purity‚0.593932569026947 6 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.966495156288147 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.13423828780651093 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.005986332893371582 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.7770084142684937 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9774846434593201 trainer/global_step‚16399 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.1336446851491928 8"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.636402428150177 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.09666667133569717 7validation_average_jet_accuracy‚0.32501763105392456 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.22797852754592896 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2584720253944397 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 particle/accuracy_2_of_3‚1 jet/accuracy_1_of_5‚1 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6571688055992126 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6930657625198364 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0032324804924428463 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.02266469970345497 *jet/accuracy_1_of_4‚0.9622095823287964 *jet/accuracy_2_of_4‚0.7000592947006226 _step‚1352 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6742002964019775 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6087103486061096 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7203326225280762 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.12867894768714905 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.1978885382413864 )jet/accuracy_3_of_4‚0.043703593313694 jet/accuracy_2_of_5‚1 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.1765308976173401 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.7884081602096558 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.7592592835426331 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9895829558372498 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.14449027180671692 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.5894553661346436 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.18088912963867188 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.7592592835426331 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.05000000074505806 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.3333333432674408 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.10787402093410492 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.03013569489121437 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.032385796308517456 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.1276121735572815 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 jet/accuracy_5_of_5‚0 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8377842903137207 8"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.622153103351593 6 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.976256787776947 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6068782806396484 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7208617329597473 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.49958333373069763 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.35452064871788025 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 particle/accuracy_1_of_4‚1 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.5714285969734192 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.2870370149612427 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.44907405972480774 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9824985265731812 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.12117373198270798 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9851381182670593 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.10283532738685608 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.19789691269397736 +jet/accuracy_3_of_3‚0.04006023705005646 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.12181765586137772 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9943969249725342 8"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.826911211013794 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.26923078298568726 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7344954609870911 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 *"Purity/2FRt*SRqqt2FBt/event_purity‚0.5 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7142857313156128 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9130856394767761 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7177000641822815 7"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.66620272397995 +jet/accuracy_1_of_1‚0.28231921792030334 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9905714988708496 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5144564509391785 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7607119083404541 *jet/accuracy_1_of_2‚0.7300025224685669 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.6875 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.8564814329147339 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9936507940292358 particle/accuracy_1_of_2‚1 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.13504526019096375 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 particle/accuracy_2_of_5‚1 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.23857121169567108 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6873555183410645 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.024530449882149696 *jet/accuracy_1_of_3‚0.9067254066467285 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9876719117164612 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6313541531562805 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7824074029922485 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5073119401931763 6 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.984964907169342 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6414054036140442 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.13393375277519226 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 particle/accuracy_1_of_5‚1 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9768246412277222 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.09503225237131119 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.42500001192092896 'particle/f_score‚0.5960361361503601 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6568373441696167 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.3333333432674408 /particle/accuracy_3_of_4‚0.9974937438964844 (particle/accuracy‚0.8294812440872192 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6217145919799805 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.8500000238418579 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.12468050420284271 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.02976994775235653 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.6410430669784546 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.8500000238418579 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.05000000074505806 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 . Purity/2FRt0SRqqt1FBt/FRt_purity‚ 0.5390625 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9722036719322205 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4542064666748047 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.03283771499991417 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.1469258815050125 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 epoch‚40 6 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.982394278049469 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9841269254684448 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.12770673632621765 8"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.826911211013794 particle/accuracy_3_of_3‚1 +particle/sensitivity‚0.5858088731765747 *particle/specificity‚0.896125316619873 _runtime‚27176.483249014 - Purity/0FRt2SRqqt*FBt/FBt_purity‚0.734375 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.5185185670852661 - Purity/0FRt2SRqqt1FBt/FBt_purity‚0.734375 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7424664497375488È !眔À@‚(:‰ ÈêÒÛŒýö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚333.089' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚66.6178 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ƙ¾rjÁ@‚(:— ÈêÒÙë¸ù$ disk.dm-0.out‚168948.83154296875 disk.sda3.in‚ 108.92578125$ disk./.usageGB‚574.2433700561523 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd1.in‚ 2.62353515625" disk.rbd1.out‚1726.11865234375 disk.dm-1.in‚ 108.92578125) disk.nvme0c0n1.out‚168948.83154296875 disk.sda1.in‚0 disk.sda3.out‚ 1585.6796875) disk./.usagePercent‚38.53795438674849& disk.rbd0.in‚1.759218604441423e+13$ memory_percent‚81.77540302276611* proc.memory.percent‚17.087507247924805 cpu‚25.58721726627564 proc.cpu.threads‚17$ disk.nvme0c0n1.in‚64320.71484375 disk.dm-1.out‚ 1585.6796875 disk.rbd0.out‚0 disk.dm-0.in‚64320.71484375 disk.sda1.out‚0 disk.drbd2165.in‚0" disk.nvme0n1.in‚64320.71484375 disk.sda.in‚ 108.92578125 network.sent‚ 3.1716188e+07 network.recv‚ 6.75ïá^¸4066e+06' disk.nvme0n1.out‚168948.83154296875 disk.sda.out‚ 1585.6796875" proc.memory.rssMB‚ 2799.6171875 disk.drbd2165.out‚0) proc.memory.availableMB‚ 2985.91796875hå!“Â@‚(:ˆ ÒˆêÒÖÿïø gpu.0.gpu‚90.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚282.295' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚56.459 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0TX»#&Ã@‚(:› ÒˆêÒ§¦‘ú network.sent‚ 3.1775227e+07) proc.memory.availableMB‚ 2985.58203125 cpu‚25.587268881340982 disk.dm-0.in‚64321.13671875$ disk.nvme0c0n1.in‚64321.13671875 disk.sda.in‚ 108.92578125 proc.cpu.threads‚17 network.recv‚ 6.757772e+06) disk.nvme0c0n1.out‚168959.23876953125' disk.nvme0n1.out‚168959.23876953125 disk.dm-1.out‚ 1585.91796875" proc.memory.rssMB‚ 2799.6171875$ disk./.usageGB‚574.2499694824219$ disk.dm-0.out‚168959.23876953125 disk.sda1.in‚0 disk.rbd0.out‚0$ memory_percent‚81.77745342254639 disk.drbd2165.out‚0" disk.nvme0n1.in‚64321.13671875 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd1.in‚ 2.62353515625" disk.rbd1.out‚1726.11865234375 disk.sda.out‚ 1585.91796875 disk.sda3.out‚ 1585.91796875& disk.rbd0.in‚1.759218604441423e+13) disk./.usagePercent‚38.53839727978273 disk.sda3.in‚ 108.92578125 disk.drbd2165.in‚0 disk.dm-1.in‚ 108.92578125 disk.sda1.out‚0* proc.memory.percent‚17.087507247924805ô¼AŸÄ@‚(:” áˆêÒúÒ÷ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚176.885' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚35.376999999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Z ìÅ@‚(: áˆêÒ‹°Äù network.recv‚ 6.761085e+06 disk.sda1.out‚0$ disk.nvme0c0n1.in‚64321.35546875" proc.memory.rssMB‚ 2799.6171875 network.sent‚ 3.1778297e+07) disk./.usagePercent‚38.53871498513272 disk.dm-1.in‚ 108.92578125$ disk.dm-0.out‚168972.24658203125 disk.sda3.out‚ 1586.328125 disk.sda.out‚ 1586.328125 disk.drbd2165.in‚0 cpu‚25.5873596836819& disk.rbd0.in‚1.759218604441423e+13' disk.nvme0n1.out‚168972.24658203125 disk.sda2.out‚0# memory_percent‚81.7774772644043* proc.memory.percent‚17.087507247924805 disk.dm-0.in‚64321.35546875 disk.drbd2165.out‚0 disk.dm-1.out‚ 1586.328125" disk.nvme0n1.in‚64321.35546875 disk.sda2.in‚0 proc.cpu.threads‚17 disk.rbd0.out‚0) disk.nvme0c0n1.out‚168972.24658203125" disk.rbd1.out‚1726.18115234375 disk.sda1.in‚0$ disk./.usageGB‚574.2547035217285 disk.sda.in‚ 108.92578125 disk.sda3.in‚ 108.92578125 disk.rbd1.in‚ 2.62353515625' proc.memory.availableMB‚ 2985.578125úæ‰ÞŸÆ@‚(:” ðˆêÒðñæö gpu.0.gpu‚99.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚280.079' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚56.015800000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¢O)PÇ@‚(:” ðˆêÒ¼…Éù& disk.rbd0.in‚1.759218604441423e+13 disk.sda1.in‚0 disk.sda3.in‚ 108.92578125 disk.drbd2165.in‚0 cpu‚25.587412441384487 disk.sda.out‚ 1586.5234375$ memory_percent‚81.77638053894043$ disk./.usageGB‚574.2621231079102$ disk.dm-0.out‚168981.99755859375) disk.nvme0c0n1.out‚168981.99755859375' disk.nvme0n1.out‚168981.99755859375 disk.sda2.out‚0 disk.dm-1.out‚ 1586.5234375 disk.sda3.out‚ 1586.5234375 network.sent‚ 3.1783344e+07 disk.sda.in‚ 108.92578125 disk.rbd0.out‚0 disk.sda1.out‚0 disk.rbd1.in‚ 2.62353515625* proc.memory.percent‚17.087507247924805 network.recv‚ 6.764541e+06 disk.dm-1.in‚ 108.92578125( proc.memory.availableMB‚ 2985.7578125" proc.memory.rssMB‚ 2799.6171875) disk./.usagePercent‚38.53921291978682 disk.dm-0.in‚ 64426.0703125" disk.rbd1.out‚1726.18115234375 disk.drbd2165.out‚0# disk.nvme0c0n1.in‚ 64426.0703125! disk.nvme0n1.in‚ 64426.0703125 disk.sda2.in‚0 proc.cpu.threads‚17á×ñžÈ@‚(:“ ÿˆêÒ­¬åö gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚286.799' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚57.35979999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÉÚv%É@‚(:š ÿˆêÒúÏù disk.sda1.in‚0 network.sent‚ 3.1788419e+07' disk.nvme0n1.out‚168994.52197265625 disk.dm-1.in‚ 108.92578125 cpu‚25.587457143559025 disk.sda3.out‚ 1586.77734375 disk.rbd1.in‚ 2.62353515625$ disk.dm-0.out‚168994.52197265625 disk.sda1.out‚0" proc.memory.rssMB‚ 2799.6171875 proc.cpu.threads‚17 disk.sda2.out‚0" disk.rbd1.out‚1726.24365234375# disk./.usageGB‚574.268497467041 disk.drbd2165.out‚0 disk.sda.out‚ 1586.77734375 disk.dm-1.out‚ 1586.77734375 disk.rbd0.out‚0 disk.drbd2165.in‚0$ disk.nvme0c0n1.in‚64426.48046875) disk.nvme0c0n1.out‚168994.52197265625& disk.rbd0.in‚1.759218604441423e+13 disk.dm-0.in‚64426.48046875* proc.memory.percent‚17.087507247924805 network.recv‚ 6.767991e+06 disk.sda2.in‚0$ memory_percent‚81.77492618560791) disk./.usagePercent‚38.53964070837653 disk.sda3.in‚ 108.92578125" disk.nvme0n1.in‚64426.48046875 disk.sda.in‚ 108.92578125) proc.memory.availableMB‚ 2985.99609375Yä5z4Ê@/ _wandbruntime‚27254  _runtime‚27254¿ÑjúÄË@¾ # lr-AdamW/pg1‚5.453604964902e-05 # lr-AdamW/pg2‚5.453604964902e-05 trainer/global_step‚16449 % _timestamp‚1.7843170702627723e+09 _runtime‚27254.479954736 _step‚13539¬ 6ÉÌ@à # lr-AdamW/pg2‚5.453604964902e-05 trainer/global_step‚16449 % _timestamp‚1.7843170702627723e+09 _runtime‚27254.479954736 _step‚1353 # lr-AdamW/pg1‚5.453604964902e-05É c4a4Í@/ _wandbruntime‚27254  _runtime‚27254%3-(Î@”P dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚19962778634752 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚4745623508646912 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚4728104031064064 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 142487010 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 140405311 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15478128795648 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚19962746195456 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚15476783406080 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚46589537 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚16449 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 282892321 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 140405311 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚244 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚32439296 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 142496983 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 _step‚1354 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1296778906540544 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 400530432 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 142496983 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 % _timestamp‚1.7843170702638373e+09 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 140405317 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚4745623508646912 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 282902300 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚4730146725240832 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚4730146725240832 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1296778506010112 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1276815759814656 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚4745623508646912 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚4728104031064064 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 142487010 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 140405311 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15478128795648 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚19962746195456 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚15476783406080 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚46589537 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚16449 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 282892321 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 140405311 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚244 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚32439296 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 142496983 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 _step‚1354 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1296778906540544 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 400530432 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 142496983 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 % _timestamp‚1.7843170702638373e+09 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 140405317 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚4745623508646912 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 282902300 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136Ê ‚éþŸÐ@‚(:” Ž‰êÒ¨ëö gpu.0.gpu‚99.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚204.689' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚40.937799999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÒœÒú Ñ@‚(: Ž‰êÒ´†íú network.recv‚ 6.771704e+06! disk./.usageGB‚575.3798828125 disk.drbd2165.in‚0 disk.sda1.out‚0" proc.memory.rssMB‚ 2799.6171875' proc.memory.availableMB‚ 2985.765625 disk.dm-1.in‚ 108.92578125$ memory_percent‚81.77633285522461 proc.cpu.threads‚17 disk.sda1.in‚0 disk.drbd2165.out‚0# disk.nvme0c0n1.out‚ 169048.71875! disk.nvme0n1.in‚ 64426.7109375 disk.rbd1.in‚ 2.62353515625" disk.rbd1.out‚1726.24365234375 network.sent‚ 3.179494e+07 disk.dm-1.out‚ 1586.98046875 disk.sda2.in‚0 disk.rbd0.out‚0 disk.sda3.out‚ 1586.98046875# disk.nvme0c0n1.in‚ 64426.7109375 cpu‚25.587542927215104 disk.dm-0.in‚ 64426.7109375 disk.dm-0.out‚ 169048.71875* disk./.usagePercent‚38.614226711424706 disk.sda.in‚ 108.92578125 disk.sda.out‚ 1586.98046875! disk.nvme0n1.out‚ 169048.71875& disk.rbd0.in‚1.759218604441423e+13 disk.sda3.in‚ 108.92578125 disk.sda2.out‚0* proc.memory.percent‚17.087507247924805wÕ^`4Ò@/ _wandbruntime‚27255  _runtime‚27255(Ä[”ÂÓ@žN PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚4745903805982720 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 282918998 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚4728381887414272 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15452567447681 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚4730424752144384 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚236 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚4730424725970944 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 142505475 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1276893246222336 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1296857206530560 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚4730424725970944 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚19963960308224 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 _runtime‚27255.865386489 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 282909005 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚4730424752144384 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚32432128 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚88797044 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚4743833110261519 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 282909005 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚46592314 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 140413517 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 142505475 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚88797280 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 140413523 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15479053838336 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚4743834480289921 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚15477708441600 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820ôTðŠ)'Ô@£N @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚16449 % _timestamp‚1.7843170716480076e+09 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15479053838336 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1296857206530560 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚4730424725970944 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚19963960308224 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 _runtime‚27255.865386489 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 282909005 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚4730424752144384 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚32432128 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚88797044 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚4743833110261519 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 282909005 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚46592314 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 140413517 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 142505475 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚88797280 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 140413523 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15479053838336 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚4743834480289921 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚15477708441600 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚4745903805982720 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 282918998 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚4728381887414272 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15452567447681 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚4730424752144384 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚236 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚4730424725970944 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 142505475 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1276893246222336 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 E­§XÙ@‚(:û ‰êÒèø¦ù network.recv‚ 6.776021e+06 disk.rbd0.out‚0" disk.rbd1.out‚1726.30615234375 disk.sda.in‚ 108.92578125 disk.dm-1.in‚ 108.92578125 disk.sda3.in‚ 108.92578125 disk.sda3.out‚ 1587.21875 disk.sda2.out‚0* proc.memory.percent‚17.087507247924805! disk.dm-0.out‚169064.85546875" proc.memory.rssMB‚ 2799.6171875 cpu‚25.587580280177473 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda.out‚ 1587.21875 disk.dm-1.out‚ 1587.21875& disk.rbd0.in‚1.759218604441423e+13 disk.sda1.in‚0 disk.drbd2165.out‚0& disk.nvme0c0n1.out‚169064.85546875 proc.memory.availableMB‚2986! disk.nvme0n1.in‚ 64427.1328125$ disk.nvme0n1.out‚169064.85546875 disk.rbd1.in‚ 2.62353515625 proc.cpu.threads‚17$ disk./.usageGB‚575.3483238220215 disk.dm-0.in‚ 64427.1328125! memory_percent‚81.77490234375 network.sent‚ 3.1846146e+07* disk./.usagePercent‚38.612108761093936 disk.drbd2165.in‚0# disk.nvme0c0n1.in‚ 64427.1328125©G„>’Ú@‚(:‡ ¬‰êÒ»‘÷ gpu.0.gpu‚99.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚347.51' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚69.502 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0nÄ\*Û@‚(: ¬‰êÒí¼Ëù disk.sda3.out‚ 1589.4453125 network.recv‚ 6.780298e+06 disk.sda1.in‚0 disk.sda.in‚ 108.92578125" disk.dm-0.out‚170206.376953125" disk.nvme0n1.in‚64537.60546875 cpu‚25.58765977971461* disk./.usagePercent‚38.541286580814166 disk.drbd2165.in‚0 disk.drbd2165.out‚0% disk.nvme0n1.out‚170206.376953125" proc.memory.rssMB‚ 2799.6171875* proc.memory.percent‚17.087507247924805 disk.rbd0.out‚0 disk.dm-0.in‚64537.60546875 disk.rbd1.in‚ 2.62353515625" disk.rbd1.out‚1726.30615234375 disk.sda.out‚ 1589.4453125$ memory_percent‚81.78014755249023' proc.memory.availableMB‚ 2985.140625 network.sent‚ 3.1900092e+07 disk.sda1.out‚0 disk.sda3.in‚ 108.92578125$ disk.nvme0c0n1.in‚64537.60546875 disk.dm-1.in‚ 108.92578125 disk.dm-1.out‚ 1589.4453125' disk.nvme0c0n1.out‚170206.376953125 disk.sda2.out‚0$ disk./.usageGB‚574.2930221557617& disk.rbd0.in‚1.759218604441423e+13 disk.sda2.in‚0 proc.cpu.threads‚17Æ)%>ŸÜ@‚(:” »‰êÒ¬Žéö gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚264.778' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚52.955600000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0²rPÝ@‚(:’ »‰êÒ–ž€ú' disk.nvme0c0n1.out‚170228.658203125 network.recv‚ 6.783339e+06* disk./.usagePercent‚38.541946824244974 disk.sda3.in‚ 108.92578125 disk.sda.in‚ 108.92578125* proc.memory.percent‚17.087507247924805 disk.sda1.in‚0$ disk.nvme0c0n1.in‚64537.82421875$ disk./.usageGB‚574.3028602600098 disk.sda2.out‚0" disk.rbd1.out‚1726.36865234375 disk.dm-1.in‚ 108.92578125& proc.memory.availableMB‚ 2984.78125 disk.drbd2165.in‚0 disk.drbd2165.out‚0 network.sent‚ 3.1903807e+07" disk.dm-0.out‚170228.658203125" disk.nvme0n1.in‚64537.82421875 disk.sda.out‚ 1591.69921875 disk.dm-1.out‚ 1591.69921875& disk.rbd0.in‚1.759218604441423e+13% disk.nvme0n1.out‚170228.658203125 disk.rbd1.in‚ 2.62353515625 disk.rbd0.out‚0 cpu‚25.58772066480773 proc.cpu.threads‚17 disk.dm-0.in‚64537.82421875 disk.sda3.out‚ 1591.69921875 disk.sda2.in‚0$ memory_percent‚81.78234100341797" proc.memory.rssMB‚ 2799.6171875 disk.sda1.out‚0¾µS‰•Þ@‚(:Š ʉêÒÜå÷ö gpu.0.gpu‚100.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚46.0 gpu.0.powerWatts‚338.986' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚67.7972 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚03§Î¹ß@‚(:‘ ʉêÒ©²ù network.recv‚ 6.786881e+06 disk.drbd2165.out‚0 proc.cpu.threads‚17$ disk./.usageGB‚574.2706336975098 disk.drbd2165.in‚0& proc.memory.availableMB‚ 2985.03125 cpu‚25.587820019741496) disk./.usagePercent‚38.53978407259571$ disk.nvme0c0n1.in‚64538.24609375 disk.sda2.in‚0 disk.dm-1.in‚ 108.92578125 disk.dm-1.out‚ 1592.265625 disk.sda1.out‚0 disk.sda3.in‚ 108.92578125 disk.sda3.out‚ 1592.265625 network.sent‚ 3.190888e+07 disk.sda2.out‚0 disk.sda1.in‚0" proc.memory.rssMB‚ 2799.6171875) disk.nvme0c0n1.out‚170245.95361328125 disk.sda.out‚ 1592.265625 disk.rbd0.out‚0$ memory_percent‚81.78081512451172* proc.memory.percent‚17.087507247924805' disk.nvme0n1.out‚170245.95361328125 disk.sda.in‚ 108.92578125& disk.rbd0.in‚1.759218604441423e+13" disk.nvme0n1.in‚64538.24609375 disk.rbd1.in‚ 2.62353515625" disk.rbd1.out‚1726.36865234375 disk.dm-0.in‚64538.24609375$ disk.dm-0.out‚170245.95361328125ñ—ážà@‚(:“ Ù‰êÒäÄ÷ö gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚273.684' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚54.73680000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0üÝfá@‚(:‰ Ù‰êÒü¦¦ù$ disk.nvme0c0n1.in‚64538.44921875 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd1.in‚ 2.62353515625 disk.dm-1.in‚ 108.92578125 disk.rbd0.out‚0 disk.sda1.out‚0 proc.cpu.threads‚17& proc.memory.availableMB‚ 2984.78125 cpu‚25.58788172280243 disk.sda3.out‚ 1594.828125 disk.drbd2165.out‚0 disk.sda1.in‚0 disk.sda.in‚ 108.92578125 disk.sda.out‚ 1594.828125& disk.rbd0.in‚1.759218604441423e+13$ memory_percent‚81.78234100341797 disk.sda3.in‚ 108.92578125 disk.drbd2165.in‚0" proc.memory.rssMB‚ 2799.6171875 network.sent‚ 3.1915281e+07* disk./.usagePercent‚38.539866763029266" disk.nvme0n1.in‚64538.44921875 disk.dm-1.out‚ 1594.828125! disk.dm-0.out‚170258.14453125* proc.memory.percent‚17.087507247924805 network.recv‚ 6.790021e+06$ disk./.usageGB‚574.2718658447266& disk.nvme0c0n1.out‚170258.14453125$ disk.nvme0n1.out‚170258.14453125" disk.rbd1.out‚1726.49365234375 disk.dm-0.in‚64538.44921875l ÿ4â@/ _wandbruntime‚27333  _runtime‚27333~Ûl Éã@à & lr-AdamW/pg1‚5.305131586025034e-05 & lr-AdamW/pg2‚5.305131586025034e-05 trainer/global_step‚16499 $ _timestamp‚1.784317149339724e+09 _runtime‚27333.556929818 _step‚1357zcHÎä@È & lr-AdamW/pg2‚5.305131586025034e-05 trainer/global_step‚16499 $ _timestamp‚1.784317149339724e+09 _runtime‚27333.556929818 _step‚1357 & lr-AdamW/pg1‚5.305131586025034e-05Í VÚ-ä4å@/ _wandbruntime‚27333  _runtime‚27333*„7æ@”P @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚4757494631039005 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚4744046787919872 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚42329337 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚15523099336704 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 140815617 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚20022939557888 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚4759569861083136 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚4744046761746432 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚4757496001067295 _runtime‚27333.558241405 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚261 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15497881094431 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 142912762 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚42329342 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1280690080198656 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚89058115 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚266 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1300713387847680 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 _step‚1358 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15524444726272 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚89058381 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1300712985220096 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 140815617 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15523099336704 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 142922735 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚46728778 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚4744046761746432 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 283738352 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15524444726272 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 142922735 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚4759571232646144 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 % _timestamp‚1.7843171493408525e+09 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ^IDeviceStatsMonitor.on_t¤2 rain_batch_start/active_bytes.large_pool.allocated‚4744046787919872 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚20022905021440 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚4759569861083136 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 140815611 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1280690448289792 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚4759571232646144 trainer/global_step‚16499 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15496536494109 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚46729039 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚4741998094544896 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 142912762 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚4741998119972864 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 283728373 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 140815611 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448/Ñrt(ç@™P OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 % _timestamp‚1.7843171493408525e+09 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚4744046787919872 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚20022905021440 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚4759569861083136 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 140815611 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1280690448289792 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚4759571232646144 trainer/global_step‚16499 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15496536494109 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚46729039 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚4741998094544896 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 142912762 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚4741998119972864 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 283728373 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 140815611 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚4757494631039005 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚4744046787919872 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚42329337 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚15523099336704 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 140815617 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚20022939557888 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚4759569861083136 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚4744046761746432 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚4757496001067295 _runtime‚27333.558241405 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚261 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15497881094431 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 142912762 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚42329342 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1280690080198656 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚89058115 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚266 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1300713387847680 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 _step‚1358 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15524444726272 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚89058381 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1300712985220096 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 140815617 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15523099336704 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 142922735 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚46728778 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚4744046761746432 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 283738352 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15524444726272 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 142922735 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚4759571232646144 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995Î -{Ë 4è@/ _wandbruntime‚27334  _runtime‚27334é/?…$'é@žN PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 140823823 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 142921240 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1300791662090240 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚89063661 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚89063407 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚16499 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1280767934697472 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 142921240 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚42331880 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 142931227 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 283755050 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 140823817 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚4759850158418944 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 140823817 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚4759850158418944 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚4759848786848768 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚20024130013184 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1280767566606336 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚20024095483904 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚4757773436283427 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15497459960355 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚42331885 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚46731776 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 283745057 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚4744324788649984 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 142931227 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1300792064710656 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 140823823 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚4744324762476544 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 283755050 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚46731527 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15524024372224 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 283745057 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 QDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚4759850158418944 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 140823817 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚4759850158418944 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚4759848786848768 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚20024130013184 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1280767566606336 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚20024095483904 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚4757773436283427 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15497459960355 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚42331885 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚46731776 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 283745057 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚4744324788649984 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 142931227 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1300792064710656 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 140823823 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚4744324762476544 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 283755050 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚46731527 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15524024372224 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 283745057 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 Q8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚89063661 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚89063407 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚16499 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1280767934697472 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 142921240 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚42331880 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 142931227 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?u®ËÆDeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 283755050 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 140823817 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995Ï ÑÚ$Í4ë@/ _wandbruntime‚27335  _runtime‚27335©£ì@Œ 1loss/FRt2/assignment_loss‚0.06873868634461094 4loss/FBt1/assignment_loss‚0.00036722867541924716 0loss/FRt1/detection_loss‚0.43969170923969064 1loss/FBt2/detection_loss‚0.021800441154448014 epoch‚41 3loss/FBt2/assignment_loss‚2.373294459569361e-05 .loss/FRt2/detection_loss‚1.162752403798102 1loss/SRqqt1/detection_loss‚0.3306464620448843 /loss/FBt1/detection_loss‚0.2769230090342556 $ _timestamp‚1.784317150812438e+09 3loss/SRqqt1/assignment_loss‚0.03163029244382802 1loss/SRqqt2/detection_loss‚0.4246864111113011 _runtime‚27335.029334157 _step‚1360 .loss/FRt1/assignment_loss‚0.82455889257016 3loss/SRqqt2/assignment_loss‚0.08805101108353718 %loss/total_loss‚3.669870280444833 trainer/global_step‚16499ÌY)åí@‘ /loss/FBt1/detection_loss‚0.2769230090342556 $ _timestamp‚1.784317150812438e+09 3loss/SRqqt1/assignment_loss‚0.03163029244382802 1loss/SRqqt2/detection_loss‚0.4246864111113011 _runtime‚27335.029334157 _step‚1360 .loss/FRt1/assignment_loss‚0.82455889257016 3loss/SRqqt2/assignment_loss‚0.08805101108353718 %loss/total_loss‚3.669870280444833 trainer/global_step‚16499 1loss/FRt2/assignment_loss‚0.06873868634461094 4loss/FBt1/assignment_loss‚0.00036722867541924716 0loss/FRt1/detection_loss‚0.43969170923969064 1loss/FBt2/detection_loss‚0.021800441154448014 epoch‚41 3loss/FBt2/assignment_loss‚2.373294459569361e-05 .loss/FRt2/detection_loss‚1.162752403798102 1loss/SRqqt1/detection_loss‚0.3306464620448843Ð ´“î@‚(:ˆ è‰êÒ×Äó÷ gpu.0.gpu‚93.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚335.505' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚67.101 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0œû "ï@‚(:— è‰êÒÍϸù) disk./.usagePercent‚38.54039081045186& disk.nvme0n1.out‚170271.6201171875 disk.rbd1.in‚ 2.62353515625& disk.rbd0.in‚1.759218604441423e+13 disk.dm-0.in‚64553.91015625* proc.memory.percent‚17.087507247924805 disk.sda2.out‚0" proc.memory.rssMB‚ 2799.6171875 network.recv‚ 6.793554e+06" disk.rbd1.out‚1726.49365234375 disk.dm-1.in‚ 108.92578125 disk.sda1.out‚0 disk.sda3.in‚ 108.92578125) proc.memory.availableMB‚ 2985.08984375 disk.sda2.in‚0 disk.rbd0.out‚0 disk.sda1.in‚0( disk.nvme0c0n1.out‚170271.6201171875$ memory_percent‚81.78045749664307 proc.cpu.threads‚17 disk.sda.in‚ 108.92578125 disk.sda.out‚ 1595.74609375 disk.drbd2165.out‚0 network.sent‚ 3.192031e+07 disk.dm-1.out‚ 1595.74609375# disk.dm-0.out‚170271.6201171875 disk.drbd2165.in‚0 cpu‚25.587906361053992$ disk.nvme0c0n1.in‚64553.91015625$ disk./.usageGB‚574.2796745300293" disk.nvme0n1.in‚64553.91015625 disk.sda3.out‚ 1595.74609375lµ¨žð@‚(:“ ÷‰êÒ­øö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚265.603' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚53.12060000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0åf‘"ñ@‚(:— ÷‰êÒ¦ˆù) disk./.usagePercent‚38.54070544371144" disk.rbd1.out‚1726.55615234375# memory_percent‚81.7805528640747* proc.memory.percent‚17.087507247924805 disk.sda.in‚ 108.92578125 disk.dm-1.in‚ 108.92578125 disk.dm-1.out‚ 1596.2109375& disk.rbd0.in‚1.759218604441423e+13 disk.dm-0.in‚64623.93359375$ disk.nvme0c0n1.in‚64623.93359375' disk.nvme0n1.out‚170284.13720703125 disk.sda2.out‚0 disk.drbd2165.out‚0 disk.sda2.in‚0" proc.memory.rssMB‚ 2799.6171875 cpu‚25.587785896786876$ disk./.usageGB‚574.2843627929688 disk.sda3.out‚ 1596.2109375 disk.sda.out‚ 1596.2109375 disk.rbd0.out‚0$ disk.dm-0.out‚170284.13720703125 disk.sda3.in‚ 108.92578125 network.sent‚ 3.1983741e+07) disk.nvme0c0n1.out‚170284.13720703125" disk.nvme0n1.in‚64623.93359375 proc.cpu.threads‚17 disk.sda1.in‚0 disk.sda1.out‚0) proc.memory.availableMB‚ 2985.07421875 network.recv‚ 6.797885e+06 disk.drbd2165.in‚0 disk.rbd1.in‚ 2.62353515625VKw¨Ÿò@‚(:” †ŠêÒáÜ‘÷ gpu.0.gpu‚99.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚271.794' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚54.358799999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0$ÿù£ó@‚(:’ †ŠêÒºÏþù proc.cpu.threads‚17" disk.rbd1.out‚1726.55615234375* disk./.usagePercent‚38.541443257425115 disk.dm-1.out‚ 1597.359375 disk.dm-0.in‚64624.18359375# disk.dm-0.out‚170304.5576171875 disk.sda1.in‚0$ memory_percent‚81.78210258483887* proc.memory.percent‚17.087507247924805 disk.drbd2165.in‚0( proc.memory.availableMB‚ 2984.8203125$ disk.nvme0c0n1.in‚64624.18359375 disk.rbd1.in‚ 2.62353515625 disk.sda1.out‚0" disk.nvme0n1.in‚64624.18359375 disk.sda3.in‚ 108.92578125 disk.sda.in‚ 108.92578125( disk.nvme0c0n1.out‚170304.5576171875 disk.sda2.in‚0 disk.dm-1.in‚ 108.92578125& disk.rbd0.in‚1.759218604441423e+13 cpu‚25.587681307024422& disk.nvme0n1.out‚170304.5576171875 disk.sda3.out‚ 1597.359375 disk.sda.out‚ 1597.359375 network.recv‚ 6.801523e+06$ disk./.usageGB‚574.2953567504883 disk.sda2.out‚0 disk.rbd0.out‚0" proc.memory.rssMB‚ 2799.6171875 network.sent‚ 3.1988885e+07 disk.drbd2165.out‚0­ä–Ÿô@‚(:” •ŠêÒ“úÿö gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚281.431' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚56.286199999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0\ ±õ@‚(:Œ •ŠêÒ¯øÁù* disk./.usagePercent‚38.541937095958666 disk.dm-1.out‚ 1600.234375 disk.sda3.out‚ 1600.234375$ memory_percent‚81.78362846374512$ disk.nvme0c0n1.in‚64624.60546875 network.recv‚ 6.804707e+06& disk.nvme0c0n1.out‚170313.98828125$ disk.nvme0n1.out‚170313.98828125 disk.rbd0.out‚0 disk.dm-0.in‚64624.60546875 disk.drbd2165.in‚0" disk.nvme0n1.in‚64624.60546875 disk.rbd1.in‚ 2.62353515625 cpu‚25.587570389899913! disk.dm-0.out‚170313.98828125$ disk./.usageGB‚574.3027153015137 disk.sda.in‚ 108.92578125 disk.sda3.in‚ 108.92578125 disk.sda2.in‚0( proc.memory.availableMB‚ 2984.5703125* proc.memory.percent‚17.087507247924805 disk.sda1.in‚0" disk.rbd1.out‚1726.61865234375 network.sent‚ 3.1992613e+07 disk.sda.out‚ 1600.234375& disk.rbd0.in‚1.759218604441423e+13 disk.drbd2165.out‚0 proc.cpu.threads‚17 disk.dm-1.in‚ 108.92578125 disk.sda1.out‚0 disk.sda2.out‚0" proc.memory.rssMB‚ 2799.6171875'eù”ö@‚(:‰ ¤ŠêÒ­Ì÷ gpu.0.gpu‚95.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚343.578' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚68.7156 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0$¢‘c÷@‚(:” ¤ŠêÒ±ìÁú disk.sda.in‚ 108.92578125 disk.dm-1.out‚ 1602.0703125 disk.sda1.out‚0" disk.nvme0n1.in‚64624.83984375" proc.memory.rssMB‚ 2799.6171875 network.recv‚ 6.808253e+06 disk.dm-1.in‚ 108.92578125 disk.dm-0.in‚64624.83984375 disk.sda3.in‚ 108.92578125 network.sent‚ 3.1999061e+07$ disk./.usageGB‚574.3034782409668& disk.rbd0.in‚1.759218604441423e+13 disk.drbd2165.out‚0$ disk.nvme0c0n1.in‚64624.83984375( disk.nvme0c0n1.out‚170337.5771484375* disk./.usagePercent‚38.541988297465515 disk.rbd0.out‚0 disk.sda3.out‚ 1602.0703125 disk.drbd2165.in‚0 disk.sda2.out‚0* proc.memory.percent‚17.087507247924805 disk.sda.out‚ 1602.0703125 disk.sda1.in‚0 disk.rbd1.in‚ 2.62353515625# disk.dm-0.out‚170337.5771484375& disk.nvme0n1.out‚170337.5771484375 disk.sda2.in‚0# memory_percent‚81.7746639251709 cpu‚25.587465900248887" disk.rbd1.out‚1726.61865234375( proc.memory.availableMB‚ 2986.0390625 proc.cpu.threads‚17 ÿñÅ4ø@/ _wandbruntime‚27412  _runtime‚27412C EÌù@Æ ' lr-AdamW/pg1‚5.1585392751217726e-05 ' lr-AdamW/pg2‚5.1585392751217726e-05 trainer/global_step‚16549 % _timestamp‚1.7843172281365023e+09 _runtime‚27412.353620796 _step‚1361í%äÑú@Ë ' lr-AdamW/pg1‚5.1585392751217726e-05 ' lr-AdamW/pg2‚5.1585392751217726e-05 trainer/global_step‚16549 % _timestamp‚1.7843172281365023e+09 _runtime‚27412.353620796 _step‚1361Ñ øíÃ@4û@/ _wandbruntime‚27412  _runtime‚27412O•d‹(ü@”P S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚4773516213519360 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚4757946798252032 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚42456492 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚20082521109504 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15570760656896 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1304647322222592 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 % _timestamp‚1.7843172281376119e+09 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 284574404 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚20082553548800 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 141225917 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15569415267328 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 141225917 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 143348487 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚251 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚4757946798252032 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚46866949 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚4757946824425472 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚4773516213519360 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15542773607217 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚89323436 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 400530432 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚4757946824425472 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 141225911 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 143338514 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚4755892208881664 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚4755892183453696 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 141225911 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚256 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1304646921692160 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 143338514 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚4771434957060913 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚15569415267328 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 trainer/global_step‚16549 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 284564425 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15570760656896 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 143348487 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15544118207539 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _runtime‚27412.354949755 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚42456487 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 141225911 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚256 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1304646921692160 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 143338514 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚4771434957060913 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚15569415267328 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 trainer/global_step‚16549 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 284564425 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15570760656896 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 143348487 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15544118207539 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _runtime‚27412.354949755 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚42456487 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚4773516213519360 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚4757946798252032 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚42456492 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚20082521109504 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15570760656896 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1304647322222592 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 % _timestamp‚1.7843172281376119e+09 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 284574404 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚20082553548800 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 141225917 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15569415267328 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 141225917 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 143348487 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚251 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚4757946798252032 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚46866949 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚4757946824425472 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚4773516213519360 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15542773607217 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚89323436 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 400530432 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0Ò ­o9ü4þ@/ _wandbruntime‚27413  _runtime‚27413«Êø7ÿ@žN F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚89328728 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚20083711163392 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚4756170065231872 % _timestamp‚1.7843172295227056e+09 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 284581109 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 143356979 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚16549 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 284591102 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚4773796510855168 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚4758224798982144 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.ÓoÑísmall_pool.allocated‚15545041673897 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 141234123 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 141234117 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 141234123 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚46869942 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚4758224798982144 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚244 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1304725598153728 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚4771715132333737 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1284641886990336 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15570340302848 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 284591102 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1304726000774144 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15571685699584 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 143346992 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚42459035 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15543697073463 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚1363 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚249 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 141234117 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 _runtime‚27413.740117595 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0—ži)'€A£N [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚4758224798982144 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚244 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1304725598153728 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚4771715132333737 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1284641886990336 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15570340302848 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 284591102 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1304726000774144 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15571685699584 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 143346992 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚42459035 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15543697073463 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚1363 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚249 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 141234117 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 _runtime‚27413.740117595 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚89328728 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚20083711163392 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚4756170065231872 % _timestamp‚1.7843172295227056e+09 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 284581109 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 143356979 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚16549 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 284591102 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚4773796510855168 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚4758224798982144 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15545041673897 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 141234123 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 141234117 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 141234123 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚46869942 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926Ó Z ­í4A/ _wandbruntime‚27413  _runtime‚27413Ù d‚A 2loss/SRqqt1/detection_loss‚0.37638758486259644 0loss/FBt1/detection_loss‚0.14546664948366664 0loss/FRt1/assignment_loss‚0.8355035995853455 0loss/FRt2/assignment_loss‚0.0695775891381669 3loss/SRqqt2/assignment_loss‚0.11496902146315699 _step‚1364 1loss/FBt2/assignment_loss‚0.00032999530871899 .loss/FRt2/detection_loss‚1.131297826236119 &loss/total_loss‚3.6606581812939436 epoch‚41 % _timestamp‚1.7843172296091335e+09 _runtime‚27413.826061659 3loss/SRqqt1/assignment_loss‚0.03508688826976923 3loss/FBt1/assignment_loss‚8.140192546436862e-07 /loss/FRt1/detection_loss‚0.4160469585093365 2loss/SRqqt2/detection_loss‚0.46773143382702564 0loss/FBt2/detection_loss‚0.06825982059078718 trainer/global_step‚16549ÇødáƒA’ 3loss/SRqqt1/assignment_loss‚0.03508688826976923 3loss/FBt1/assignment_loss‚8.140192546436862e-07 /loss/FRt1/detection_loss‚0.4160469585093365 2loss/SRqqt2/detection_loss‚0.46773143382702564 0loss/FBt2/detection_loss‚0.06825982059078718 trainer/global_step‚16549 2loss/SRqqt1/detection_loss‚0.37638758486259644 0loss/FBt1/detection_loss‚0.14546664948366664 0loss/FRt1/assignment_loss‚0.8355035995853455 0loss/FRt2/assignment_loss‚0.0695775891381669 3loss/SRqqt2/assignment_loss‚0.11496902146315699 _step‚1364 1loss/FBt2/assignment_loss‚0.00032999530871899 .loss/FRt2/detection_loss‚1.131297826236119 &loss/total_loss‚3.6606581812939436 epoch‚41 % _timestamp‚1.7843172296091335e+09 _runtime‚27413.826061659Ô Å¦æ甄A‚(:‰ ³ŠêÒ€ãöö gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚275.279' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.0558 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ãd…A‚(:’ ³ŠêÒ£‚Òù disk.sda.out‚ 1604.890625 disk.drbd2165.in‚0& disk.nvme0n1.out‚170350.3701171875 disk.rbd1.in‚ 2.62353515625" disk.rbd1.out‚1726.68115234375 disk.dm-1.in‚ 108.92578125# disk.dm-0.out‚170350.3701171875 disk.sda1.out‚0* disk./.usagePercent‚38.542217424208665$ disk./.usageGB‚574.3068923950195( disk.nvme0c0n1.out‚170350.3701171875& disk.rbd0.in‚1.759218604441423e+13 disk.dm-0.in‚64742.55859375 proc.cpu.threads‚17 network.recv‚ 6.811624e+06 disk.sda3.in‚ 108.92578125 disk.sda3.out‚ 1604.890625" proc.memory.rssMB‚ 2799.6171875 disk.dm-1.out‚ 1604.890625* proc.memory.percent‚17.087507247924805$ memory_percent‚81.77609443664551 disk.drbd2165.out‚0 network.sent‚ 3.2002779e+07" disk.nvme0n1.in‚64742.55859375 disk.sda.in‚ 108.92578125 disk.rbd0.out‚0 disk.sda1.in‚0( proc.memory.availableMB‚ 2985.8046875 disk.sda2.out‚0 cpu‚25.587337769331047$ disk.nvme0c0n1.in‚64742.55859375 disk.sda2.in‚0®>¾”†A‚(:‰ ŠêÒ›©…÷ gpu.0.gpu‚91.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚226.267' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚45.2534 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0åÌ)ø‡A‚(: ŠêÒŸ¸Øù$ memory_percent‚81.77413940429688) disk./.usagePercent‚38.54267439765729 disk.sda2.out‚0 disk.sda.out‚ 1605.265625 disk.sda.in‚ 108.92578125$ disk.dm-0.out‚170368.90576171875* proc.memory.percent‚17.087507247924805 network.recv‚ 6.816347e+06" disk.rbd1.out‚1726.68115234375 disk.dm-0.in‚64742.98046875" proc.memory.rssMB‚ 2799.6171875 proc.cpu.threads‚17 network.sent‚ 3.2066276e+07$ disk./.usageGB‚574.3137016296387$ disk.nvme0c0n1.in‚64742.98046875' disk.nvme0n1.out‚170368.90576171875& disk.rbd0.in‚1.759218604441423e+13 disk.sda3.in‚ 108.92578125 disk.drbd2165.in‚0$ proc.memory.availableMB‚2986.125) disk.nvme0c0n1.out‚170368.90576171875 disk.dm-1.out‚ 1605.265625 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd2165.out‚0" disk.nvme0n1.in‚64742.98046875 disk.dm-1.in‚ 108.92578125 disk.rbd0.out‚0 disk.sda3.out‚ 1605.265625 disk.sda2.in‚0 disk.rbd1.in‚ 2.62353515625 cpu‚25.587244464663613ΣžˆA‚(:“ ÑŠêÒÖçâö gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚293.324' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚58.66480000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0;-N‰A‚(:Š ÑŠêÒò±Öù network.recv‚ 6.819703e+06 disk.dm-1.in‚ 108.92578125" disk.nvme0n1.in‚64743.19921875 disk.rbd1.in‚ 2.62353515625 disk.sda.out‚ 1605.90625# memory_percent‚81.7718505859375$ disk./.usageGB‚574.3183860778809 disk.sda2.in‚0" proc.memory.rssMB‚ 2799.6171875* proc.memory.percent‚17.087507247924805 disk.rbd0.out‚0 disk.sda1.in‚0 proc.cpu.threads‚17 network.sent‚ 3.2071445e+07 disk.dm-0.in‚64743.19921875 disk.sda3.out‚ 1605.90625) disk.nvme0c0n1.out‚170376.94287109375' disk.nvme0n1.out‚170376.94287109375* disk./.usagePercent‚38.542988774909325$ disk.dm-0.out‚170377.04833984375 disk.sda1.out‚0 disk.sda3.in‚ 108.92578125$ disk.nvme0c0n1.in‚64743.19921875 disk.sda2.out‚0 disk.sda.in‚ 108.92578125 cpu‚25.58711496464245 disk.dm-1.out‚ 1605.90625& disk.rbd0.in‚1.759218604441423e+13 disk.drbd2165.in‚0 disk.drbd2165.out‚0" disk.rbd1.out‚1726.74365234375" proc.memory.availableMB‚2986.5ü`ê"”ŠA‚(:‰ àŠêÒ÷óèø gpu.0.gpu‚90.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚270.943' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.1886 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0œ#‹A‚(:‰ àŠêÒ”©õù disk.sda3.out‚ 1606.24609375 disk.drbd2165.in‚0$ disk.nvme0c0n1.out‚ 170387.140625" disk.nvme0n1.out‚ 170387.140625 disk.sda.in‚ 108.92578125& disk.rbd0.in‚1.759218604441423e+13 disk.dm-0.out‚ 170387.140625) proc.memory.availableMB‚ 2986.00390625$ disk./.usageGB‚574.3226623535156 disk.sda3.in‚ 108.92578125# disk.nvme0c0n1.in‚ 64743.4140625! disk.nvme0n1.in‚ 64743.4140625 disk.rbd0.out‚0 disk.sda1.in‚0 network.sent‚ 3.2076476e+07 disk.dm-1.out‚ 1606.24609375$ memory_percent‚81.77487850189209 proc.cpu.threads‚17 disk.sda.out‚ 1606.24609375* proc.memory.percent‚17.087507247924805 cpu‚25.587012374036533 disk.drbd2165.out‚0 disk.sda1.out‚0" proc.memory.rssMB‚ 2799.6171875 network.recv‚ 6.823221e+06 disk.sda2.out‚0 disk.dm-1.in‚ 108.92578125) disk./.usagePercent‚38.54327575935521 disk.sda2.in‚0 disk.rbd1.in‚ 2.62353515625" disk.rbd1.out‚1726.74365234375 disk.dm-0.in‚ 64743.4140625š.›V’ŒA‚(:‡ ïŠêÒßê‰÷ gpu.0.gpu‚93.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚338.59' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚67.718 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0»¼¦ A‚(:• ïŠêÒç¥õù) disk./.usagePercent‚38.54369945182438 disk.sda.in‚ 108.92578125 disk.dm-0.in‚ 64859.6796875$ disk.dm-0.out‚170400.35791015625" disk.rbd1.out‚1726.86865234375 disk.sda3.out‚ 1607.35546875 disk.drbd2165.in‚0" proc.memory.rssMB‚ 2799.6171875 network.sent‚ 3.2081533e+07 disk.dm-1.in‚ 108.92578125 disk.dm-1.out‚ 1607.35546875$ memory_percent‚81.77359104156494$ disk./.usageGB‚574.3289756774902) disk.nvme0c0n1.out‚170400.35791015625) proc.memory.availableMB‚ 2986.21484375 cpu‚25.5869194684126' disk.nvme0n1.out‚170400.35791015625& disk.rbd0.in‚1.759218604441423e+13 disk.rbd0.out‚0! disk.nvme0n1.in‚ 64859.6796875 disk.sda1.in‚0 disk.sda3.in‚ 108.92578125# disk.nvme0c0n1.in‚ 64859.6796875 disk.sda2.in‚0 disk.rbd1.in‚ 2.62353515625 disk.sda1.out‚0 disk.drbd2165.out‚0 disk.sda.out‚ 1607.35546875* proc.memory.percent‚17.087507247924805 proc.cpu.threads‚17 network.recv‚ 6.82623e+06 disk.sda2.out‚0ì I‚4ŽA/ _wandbruntime‚27490  _runtime‚27490ߤOÊAÄ ' lr-AdamW/pg1‚5.0138374475423916e-05 ' lr-AdamW/pg2‚5.0138374475423916e-05 trainer/global_step‚16599 $ _timestamp‚1.784317306706467e+09 _runtime‚27490.92362391 _step‚1365ÍþÁÏAÉ $ _timestamp‚1.784317306706467e+09 _runtime‚27490.92362391 _step‚1365 ' lr-AdamW/pg1‚5.0138374475423916e-05 ' lr-AdamW/pg2‚5.0138374475423916e-05 trainer/global_step‚16599Õ ð^Ôk4‘A/ _wandbruntime‚27490  _runtime‚27490-a…”‘’A”P RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 trainer/global_step‚16599 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 141636217 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚47005302 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚4787462565955584 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15615731197952 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚4785376653111111 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚47005558 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15589010720325 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚42583642 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15617076587520 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 143764266 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 141636211 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚4785375283082821 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 285410456 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15590355320647 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚4771846834757632 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚4787462565955584 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 143774239 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚20142298863616 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚4771846860931072 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1308581019830272 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚4769786297790464 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 143774239 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0Äû'?(“A™P WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 285410456 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15590355320647 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚4771846834757632 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚4787462565955584 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 143774239 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚20142298863616 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚4771846860931072 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1308581019830272 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚4769786297790464 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 143774239 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 trainer/global_step‚16599 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 141636217 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚47005302 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚4787462565955584 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15615731197952 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚4785376653111111 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚47005558 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15589010720325 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚42583642 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15617076587520 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 143764266 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 141636211 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚4785375283082821 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568Ö ü954”A/ _wandbruntime‚27492  _runtime‚27492•UF»$'•AžN B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15616656233472 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚4785654088327243 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚89594220 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚253 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 143782731 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1308659701051392 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚4787742863291392 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1308660103671808 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚4772124835487744 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚4787742863291392 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚15616656233472 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 143772744 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚4772124861661184 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1288516207374336 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 141644417 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 285427154 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚47008040 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚20143493677056 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15591278787005 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚16599e¸^¡Ý–A£N KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15616656233472 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚4785654088327243 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚89594220 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚253 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 143782731 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1308659701051392 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚4787742863291392 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1308660103671808 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚4772124835487744 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚r!8 L 4787742863291392 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚15616656233472 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 143772744 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚4772124861661184 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1288516207374336 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 141644417 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 285427154 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚47008040 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚20143493677056 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15591278787005 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚16599 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995×  °4—A/ _wandbruntime‚27492  _runtime‚27492WJ-“˜A 0loss/FRt1/assignment_loss‚0.8602437874236364 3loss/FBt2/assignment_loss‚6.779818231272827e-07 epoch‚41 % _timestamp‚1.7843173081796494e+09 trainer/global_step‚16599 _step‚1368 1loss/FRt2/assignment_loss‚0.07353831312292063 0loss/FBt1/detection_loss‚0.19963586330044336 0loss/FBt2/detection_loss‚0.06980317388753765 _runtime‚27492.396576561 3loss/SRqqt1/assignment_loss‚0.04263259081771402 3loss/SRqqt2/assignment_loss‚0.10866313655308735 4loss/FBt1/assignment_loss‚1.9868210188178637e-08 0loss/FRt1/detection_loss‚0.39011354877853693 /loss/FRt2/detection_loss‚1.1527318618468732 0loss/SRqqt1/detection_loss‚0.437459847469065 0loss/SRqqt2/detection_loss‚0.407154222296295 &loss/total_loss‚3.7419770433461426ÃvÜ•™A” 3loss/SRqqt2/assignment_loss‚0.10866313655308735 4loss/FBt1/assignment_loss‚1.9868210188178637e-08 0loss/FRt1/detection_loss‚0.39011354877853693 /loss/FRt2/detection_loss‚1.1527318618468732 0loss/SRqqt1/detection_loss‚0.437459847469065 0loss/SRqqt2/detection_loss‚0.407154222296295 &loss/total_loss‚3.7419770433461426 0loss/FRt1/assignment_loss‚0.8602437874236364 3loss/FBt2/assignment_loss‚6.779818231272827e-07 epoch‚41 % _timestamp‚1.7843173081796494e+09 trainer/global_step‚16599 _step‚1368 1loss/FRt2/assignment_loss‚0.07353831312292063 0loss/FBt1/detection_loss‚0.19963586330044336 0loss/FBt2/detection_loss‚0.06980317388753765 _runtime‚27492.396576561 3loss/SRqqt1/assignment_loss‚0.04263259081771402Ø X"ðo“šA‚(:ˆ þŠêÒá®ëö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚265.005' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¯$µÿ ›A‚(:• þŠêÒÖÔÊù disk.sda.in‚ 108.92578125" proc.memory.rssMB‚ 2799.6171875 network.recv‚ 6.829888e+06$ disk./.usageGB‚574.3365707397461! disk.nvme0n1.in‚ 64859.8984375 disk.dm-1.in‚ 108.92578125$ memory_percent‚81.77492618560791# disk.nvme0c0n1.in‚ 64859.8984375 disk.sda2.out‚0 disk.rbd1.in‚ 2.62353515625& disk.rbd0.in‚1.759218604441423e+13 disk.drbd2165.in‚0 network.sent‚ 3.2086548e+07) disk.nvme0c0n1.out‚170411.86669921875' disk.nvme0n1.out‚170411.86669921875 disk.sda2.in‚0 disk.dm-1.out‚ 1607.8046875 disk.sda1.out‚0) proc.memory.availableMB‚ 2985.99609375 proc.cpu.threads‚17 disk.rbd0.out‚0 disk.dm-0.in‚ 64859.8984375 disk.sda1.in‚0 disk.sda3.in‚ 108.92578125 disk.drbd2165.out‚0 cpu‚25.586790960243338 disk.sda.out‚ 1607.8046875$ disk.dm-0.out‚170411.86669921875) disk./.usagePercent‚38.54420916282505" disk.rbd1.out‚1726.86865234375 disk.sda3.out‚ 1607.8046875* proc.memory.percent‚17.0875072479248054˜LŸœA‚(:” ‹êÒ¿è˜÷ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚38.0 gpu.0.powerWatts‚113.907' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.781399999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0fs˜_A‚(:Ž ‹êÒ£Ó¦ú network.sent‚ 3.214989e+07 network.recv‚ 6.834307e+06 disk.drbd2165.out‚0 disk.dm-1.in‚ 108.92578125& disk.rbd0.in‚1.759218604441423e+13 proc.cpu.threads‚17$ disk./.usageGB‚574.3436317443848! disk.nvme0n1.in‚ 64860.3203125' disk.nvme0n1.out‚170419.54052734375 disk.dm-0.in‚ 64860.3203125$ memory_percent‚81.77640438079834" proc.memory.rssMB‚ 2799.6171875 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.in‚ 108.92578125 disk.rbd1.in‚ 2.62353515625) disk./.usagePercent‚38.54468303277093" disk.rbd1.out‚1726.99365234375 disk.sda1.out‚0) disk.nvme0c0n1.out‚170419.54052734375 disk.sda.in‚ 108.92578125 disk.sda.out‚ 1608.28125 disk.dm-1.out‚ 1608.28125) proc.memory.availableMB‚ 2985.75390625 disk.sda1.in‚0 disk.sda3.out‚ 1608.28125 disk.drbd2165.in‚0 disk.rbd0.out‚0# disk.nvme0c0n1.in‚ 64860.3203125$ disk.dm-0.out‚170419.54052734375* proc.memory.percent‚17.087507247924805 cpu‚25.586669673990187Í럞A‚(:” œ‹êÒ‰÷øö gpu.0.gpu‚94.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚273.894' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚54.778800000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0]L¬8ŸA‚(: œ‹êÒЗù disk.dm-0.in‚ 64860.5234375 disk.drbd2165.in‚0 disk.rbd1.in‚ 2.62353515625$ memory_percent‚81.77967071533203# disk.dm-0.out‚170434.0732421875 disk.sda3.in‚ 108.92578125 proc.cpu.threads‚17 disk.sda3.out‚ 1608.5078125 disk.sda2.out‚0) disk./.usagePercent‚38.54496105695311$ disk./.usageGB‚574.3477745056152& disk.rbd0.in‚1.759218604441423e+13 disk.sda1.in‚0" proc.memory.rssMB‚ 2799.6171875* proc.memory.percent‚17.087507247924805 network.sent‚ 3.2154932e+07 disk.sda.out‚ 1608.5078125& disk.nvme0n1.out‚170434.0732421875 disk.sda.in‚ 108.92578125# disk.nvme0c0n1.in‚ 64860.5234375( disk.nvme0c0n1.out‚170434.0732421875! disk.nvme0n1.in‚ 64860.5234375" disk.rbd1.out‚1726.99365234375 disk.dm-1.out‚ 1608.5078125 disk.rbd0.out‚0 disk.drbd2165.out‚0 cpu‚25.586570175172422 disk.sda2.in‚0& proc.memory.availableMB‚ 2985.21875 disk.sda1.out‚0 network.recv‚ 6.838076e+06 disk.dm-1.in‚ 108.92578125æöÀ¡Ÿ A‚(:” «‹êÒ儆÷ gpu.0.gpu‚55.0 gpu.0.memory‚17, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚40.0 gpu.0.powerWatts‚109.997' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.999399999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0–½G ¡A‚(:• «‹êҌ̵ù disk.sda.in‚ 108.92578125$ disk.dm-0.out‚170442.11181640625* proc.memory.percent‚17.087507247924805) disk./.usagePercent‚38.54542007846201 disk.sda2.in‚0 proc.cpu.threads‚17) disk.nvme0c0n1.out‚170442.11181640625" disk.rbd1.out‚1727.05615234375& disk.rbd0.in‚1.759218604441423e+13 disk.sda1.in‚0 disk.sda3.in‚ 108.92578125" proc.memory.rssMB‚ 2799.6171875 cpu‚25.586468185076786' disk.nvme0n1.out‚170442.11181640625 disk.rbd0.out‚0) proc.memory.availableMB‚ 2985.46484375" disk.nvme0n1.in‚64924.42578125 disk.sda2.out‚0$ memory_percent‚81.77816867828369 network.recv‚ 6.840516e+06$ disk.nvme0c0n1.in‚64924.42578125 disk.rbd1.in‚ 2.62353515625 disk.dm-1.out‚ 1608.796875 disk.sda1.out‚0 network.sent‚ 3.2160559e+07 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.sda.out‚ 1608.796875 disk.dm-1.in‚ 108.92578125 disk.dm-0.in‚64924.42578125 disk.sda3.out‚ 1608.796875$ disk./.usageGB‚574.3546142578125ëRCអA‚(:“ º‹êÒºá‚÷ gpu.0.gpu‚90.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚291.384' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚58.27680000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0{˜Ó¯£A‚(: º‹êÒëÖ¶ù) disk.nvme0c0n1.out‚170456.36376953125& disk.rbd0.in‚1.759218604441423e+13 disk.rbd0.out‚0 disk.dm-0.in‚64924.84765625 disk.sda3.in‚ 108.92578125$ disk./.usageGB‚574.3613510131836 disk.dm-1.in‚ 108.92578125 disk.dm-1.out‚ 1608.9375 cpu‚25.586348619040074" disk.nvme0n1.in‚64924.84765625 disk.sda2.out‚0 disk.sda.out‚ 1608.9375$ disk.dm-0.out‚170456.36376953125 disk.sda3.out‚ 1608.9375) disk./.usagePercent‚38.54587218776748$ disk.nvme0c0n1.in‚64924.84765625 disk.sda2.in‚0 disk.sda1.out‚0$ memory_percent‚81.77816867828369" proc.memory.rssMB‚ 2799.6171875 network.sent‚ 3.2165596e+07 network.recv‚ 6.843858e+06 disk.sda1.in‚0* proc.memory.percent‚17.087507247924805 disk.rbd1.in‚ 2.62353515625" disk.rbd1.out‚1727.05615234375 disk.drbd2165.in‚0 disk.drbd2165.out‚0 proc.cpu.threads‚17' disk.nvme0n1.out‚170456.36376953125 disk.sda.in‚ 108.92578125) proc.memory.availableMB‚ 2985.46484375€Û¨4¤A/ _wandbruntime‚27569  _runtime‚27569n›ÔûË¥AÅ _step‚1369 ' lr-AdamW/pg1‚4.8710353972148975e-05 ' lr-AdamW/pg2‚4.8710353972148975e-05 trainer/global_step‚16649 $ _timestamp‚1.784317385436943e+09 _runtime‚27569.654981578¬‘ÇЦAÊ ' lr-AdamW/pg1‚4.8710353972148975e-05 ' lr-AdamW/pg2‚4.8710353972148975e-05 trainer/global_step‚16649 $ _timestamp‚1.784317385436943e+09 _runtime‚27569.654981578 _step‚1369Ù tÉš†4§A/ _wandbruntime‚27569  _runtime‚27569HåO‰(¨A”P MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 142046517 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚4801410289954816 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚4801408918391808 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚20202060189696 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1292313041350656 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚15662047128576 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚4785746897436672 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚256 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 144190018 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 144199991 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 144190018 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 286236529 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚4801408918391808 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚4799315609104729 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚47143364 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1292313409441792 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚47143113 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚20202027750400 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚4799316979133019 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 142046517 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15663392518144 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚89853900 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚4801410289954816 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 400530432 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 _step‚1370 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚42710787 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1312515069101056 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 144199991 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚4785746871263232 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15662047128576 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚4785746871263232 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 286246508 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 142046511 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1312515469631488 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚4783680361271296 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15636592433755 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15635247833433 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 142046511 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 _runtime‚27569.655092838 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚89854156 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175øeh '©A™P P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 142046511 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1312515469631488 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚4783680361271296 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15636592433755 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15635247833433 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 142046511 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 _runtime‚27569.655092838 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚89854156 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 142046517 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚4801410289954816 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚4801408918391808 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚20202060189696 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1292313041350656 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚15662047128576 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚4785746897436672 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚256 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 144190018 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 144199991 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 144190018 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 286236529 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚4801408918391808 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚4799315609104729 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚47143364 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1292313409441792 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚47143113 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚20202027750400 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚4799316979133019 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 142046517 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15663392518144 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚89853900 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚4801410289954816 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 400530432 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 _step‚1370 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚42710787 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1312515069101056 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 144199991 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚4785746871263232 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15662047128576 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 ZEDeviceStatsMonitor.on_train_batch_start/act4jø"ive_bytes.large_pool.freed‚4785746871263232 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 286246508 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008Ú N~@”ªA‚(:‰ É‹êÒ£†¸÷ gpu.0.gpu‚92.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚335.402' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚67.0804 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ñ|”>«A‚(:ˆ É‹êÒ¥”àù disk.rbd1.in‚ 2.62353515625 disk.dm-0.in‚64925.05078125 disk.sda1.out‚0" disk.nvme0n1.in‚64925.05078125" disk.rbd1.out‚1727.11865234375 disk.dm-0.out‚170464.1015625 disk.rbd0.out‚0 disk.drbd2165.in‚0 disk.drbd2165.out‚0 network.sent‚ 3.217077e+07 disk.dm-1.out‚ 1609.2578125 disk.sda3.out‚ 1609.2578125' proc.memory.availableMB‚ 2985.546875 cpu‚25.58623763031338 proc.cpu.threads‚17 disk.sda2.in‚0 disk.sda.in‚ 108.92578125$ disk.nvme0c0n1.in‚64925.05078125# disk.nvme0n1.out‚170464.1015625$ memory_percent‚81.77766799926758* proc.memory.percent‚17.087507247924805 network.recv‚ 6.847335e+06$ disk./.usageGB‚574.3659400939941 disk.sda2.out‚0 disk.sda.out‚ 1609.2578125 disk.dm-1.in‚ 108.92578125 disk.sda1.in‚0 disk.sda3.in‚ 108.92578125" proc.memory.rssMB‚ 2799.6171875) disk./.usagePercent‚38.54618016483117& disk.rbd0.in‚1.759218604441423e+13% disk.nvme0c0n1.out‚170464.1015625˜ŒðÁ4¬A/ _wandbruntime‚27571  _runtime‚27571Ư#_$'­AžN ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15664317560832 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 286253213 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15636171299679 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15662972164096 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚47145845 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1312594155768832 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 144208483 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚89859415 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1292390527758336 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚20203227487232 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚4783958268477440 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 142054717 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 144198496 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚4801689215727616 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚4799595784377553 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚16649 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚89859175 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 286263206 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 144208483 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚235 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 142054723 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15637515900113 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1312593755245568 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚4801689215727616 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15664317560832 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 142054723 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚42713330 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _step‚1371 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 286263206 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚20203259919360 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚240 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 144198496 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 400523264 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚4801687844157440 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 % _timestamp‚1.7843173868217587e+09 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚4786024898166784 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚42713335 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚47146080 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚4799594414349151 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 286253213 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 142054717 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 _runtime‚27571.039168152a5¥q)'®A£N UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚47145845 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1312594155768832 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 144208483 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚89859415 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1292390527758336 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚20203227487232 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚4783958268477440 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 142054717 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 144198496 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚4801689215727616 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚4799595784377553 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚16649 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚89859175 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 286263206 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 144208483 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚235 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 142054723 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15637515900113 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1312593755245568 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚4801689215727616 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15664317560832 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 142054723 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚42713330 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _step‚1371 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 286263206 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚20203259919360 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚240 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 144198496 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 400523264 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚4801687844157440 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 % _timestamp‚1.7843173868217587e+09 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚4786024898166784 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚42713335 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚47146080 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚4799594414349151 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 286253213 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 142054717 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 _runtime‚27571.039168152 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15664317560832 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 286253213 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15636171299679 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15662972164096 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440Û lžÂD4¯A/ _wandbruntime‚27571  _runtime‚27571z¶i¢°A 4loss/SRqqt1/assignment_loss‚0.022352477824132872 3loss/SRqqt2/assignment_loss‚0.08474448003725556 epoch‚41 trainer/global_step‚16649 1loss/SRqqt2/detection_loss‚0.4351334512342469 _step‚1372 0loss/FRt1/assignment_loss‚0.8609550540991324 1loss/FRt2/assignment_loss‚0.11962236109936591 3loss/FBt1/assignment_loss‚6.082663221881819e-07 1loss/SRqqt1/detection_loss‚0.4251398228973964 0loss/FBt2/detection_loss‚0.08635946340723166 % _timestamp‚1.7843173869086068e+09 _runtime‚27571.125542463 4loss/FBt2/assignment_loss‚1.5130333617335987e-07 /loss/FRt1/detection_loss‚0.4316301245743049 /loss/FRt2/detection_loss‚1.1035188104174312 /loss/FBt1/detection_loss‚0.2866297933955232 %loss/total_loss‚3.856086598555679ö&ˆ±A” 1loss/SRqqt2/detection_loss‚0.4351334512342469 _step‚1372 0loss/FRt1/assignment_loss‚0.8609550540991324 1loss/FRt2/assignment_loss‚0.11962236109936591 3loss/FBt1/assignment_loss‚6.082663221881819e-07 1loss/SRqqt1/detection_loss‚0.4251398228973964 0loss/FBt2/detection_loss‚0.08635946340723166 % _timestamp‚1.7843173869086068e+09 _runtime‚27571.125542463 4loss/FBt2/assignment_loss‚1.5130333617335987e-07 /loss/FRt1/detection_loss‚0.4316301245743049 /loss/FRt2/detection_loss‚1.1035188104174312 /loss/FBt1/detection_loss‚0.2866297933955232 %loss/total_loss‚3.856086598555679 4loss/SRqqt1/assignment_loss‚0.022352477824132872 3loss/SRqqt2/assignment_loss‚0.08474448003725556 epoch‚41 trainer/global_step‚16649Ü ÎC˜ã”²A‚(:‰ Ø‹êÒÁµþö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚277.374' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.4748 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ne§¥$³A‚(:™ Ø‹êÒñߤú$ disk./.usageGB‚574.3728141784668$ disk.nvme0c0n1.in‚64925.26953125 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-0.in‚64925.26953125" proc.memory.rssMB‚ 2799.6171875 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-1.out‚ 1609.53515625# disk.dm-0.out‚170473.3544921875) proc.memory.availableMB‚ 2985.52734375$ memory_percent‚81.77778720855713 network.sent‚ 3.2222026e+07 disk.sda3.out‚ 1609.53515625 disk.drbd2165.in‚0 disk.drbd2165.out‚0( disk.nvme0c0n1.out‚170473.3544921875 disk.rbd1.in‚ 2.67138671875 disk.sda.in‚ 108.92578125 network.recv‚ 6.851559e+06* disk./.usagePercent‚38.546641490407865" disk.nvme0n1.in‚64925.26953125 disk.dm-1.in‚ 108.92578125& disk.rbd0.in‚1.759218604441423e+13 cpu‚25.586126370532867 proc.cpu.threads‚17& disk.nvme0n1.out‚170473.3544921875 disk.sda.out‚ 1609.53515625 disk.sda3.in‚ 108.92578125 disk.rbd0.out‚0* proc.memory.percent‚17.087507247924805" disk.rbd1.out‚1727.134277343750û©ùž´A‚(:“ ç‹êÒôÀ€÷ gpu.0.gpu‚93.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚346.118' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚69.22359999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ø(Ó µA‚(:• ç‹êÒˆ¿¶ù$ memory_percent‚81.77828788757324 disk.sda1.out‚0 disk.sda3.out‚ 1609.77734375! disk.nvme0n1.in‚ 65024.2109375" disk.rbd1.out‚1728.09912109375 disk.drbd2165.in‚0) disk.nvme0c0n1.out‚170485.72216796875 disk.sda.in‚ 108.92578125 disk.dm-1.in‚ 108.92578125 disk.rbd1.in‚ 2.90673828125" proc.memory.rssMB‚ 2799.6171875$ disk./.usageGB‚574.3796463012695 disk.dm-1.out‚ 1609.77734375& disk.rbd0.in‚1.759218604441423e+13 disk.rbd0.out‚0 disk.dm-0.in‚ 65024.2109375# disk.nvme0c0n1.in‚ 65024.2109375* proc.memory.percent‚17.087507247924805 disk.drbd2165.out‚0' disk.nvme0n1.out‚170485.72216796875 disk.sda2.in‚0 disk.sda2.out‚0 proc.cpu.threads‚17 network.sent‚ 3.227467e+07 network.recv‚ 6.855677e+06) disk./.usagePercent‚38.54709999990169$ disk.dm-0.out‚170485.72216796875 cpu‚25.58601789614257 disk.sda.out‚ 1609.77734375( proc.memory.availableMB‚ 2985.4453125 disk.sda1.in‚0 disk.sda3.in‚ 108.92578125Ÿ’”¶A‚(:‰ ö‹êÒäïãö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚280.902' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚56.1804 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚00pŸ§!·A‚(:– ö‹êÒ‹Õ¹ù network.recv‚ 6.859141e+06 disk.sda3.out‚ 1610.05859375# disk.nvme0c0n1.in‚ 65024.4140625" proc.memory.rssMB‚ 2799.6171875 proc.cpu.threads‚17 disk.sda2.in‚0 disk.sda.in‚ 108.92578125& disk.rbd0.in‚1.759218604441423e+13 disk.sda2.out‚0 disk.dm-1.in‚ 108.92578125 disk.dm-1.out‚ 1610.05859375! disk.nvme0n1.in‚ 65024.4140625' disk.nvme0n1.out‚170495.91162109375" disk.rbd1.out‚1730.78662109375 disk.rbd0.out‚0 disk.dm-0.in‚ 65024.4140625 disk.sda1.in‚0 disk.drbd2165.in‚0) disk./.usagePercent‚38.54739287252087 disk.sda3.in‚ 108.92578125) proc.memory.availableMB‚ 2985.44921875$ disk.dm-0.out‚170495.91162109375 disk.drbd2165.out‚0$ memory_percent‚81.77826404571533 cpu‚25.58589880174421 disk.rbd1.in‚ 2.9619140625 disk.sda.out‚ 1610.05859375) disk.nvme0c0n1.out‚170495.91162109375* proc.memory.percent‚17.087507247924805$ disk./.usageGB‚574.3840103149414 disk.sda1.out‚0 network.sent‚ 3.2281067e+07źéž¸A‚(:“ …ŒêÒªµÔö gpu.0.gpu‚48.0 gpu.0.memory‚16, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚333.524' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚66.70479999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0>ÍÁ/ ¹A‚(:• …ŒêÒë›ù network.sent‚ 3.2286522e+07$ disk./.usageGB‚574.3908500671387 disk.sda3.out‚ 1610.26171875' disk.nvme0n1.out‚170504.83251953125 disk.sda.in‚ 108.92578125* proc.memory.percent‚17.087507247924805 disk.dm-0.in‚ 65024.6328125 disk.drbd2165.in‚0 disk.sda2.out‚0 cpu‚25.585798446992985$ disk.dm-0.out‚170504.83251953125 disk.sda3.in‚ 108.92578125 disk.dm-1.in‚ 108.92578125 disk.dm-1.out‚ 1610.26171875* disk./.usagePercent‚38.547851894029755! disk.nvme0n1.in‚ 65024.6328125 disk.sda2.in‚0 disk.rbd1.out‚ 1734.07421875& disk.rbd0.in‚1.759218604441423e+13$ memory_percent‚81.77821636199951" proc.memory.rssMB‚ 2799.6171875 network.recv‚ 6.862638e+06 disk.sda1.in‚0# disk.nvme0c0n1.in‚ 65024.6328125 disk.sda1.out‚0 disk.drbd2165.out‚0) disk.nvme0c0n1.out‚170504.83251953125 disk.sda.out‚ 1610.26171875 disk.rbd0.out‚0) proc.memory.availableMB‚ 2985.45703125 disk.rbd1.in‚ 2.9619140625 proc.cpu.threads‚17&ã{šŸºA‚(:” ”ŒêÒ¶ñö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚284.714' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚56.942800000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0xwÞÒ»A‚(:‡ ”ŒêÒž£òø disk.sda.in‚ 108.92578125 disk.sda1.out‚0# disk.nvme0c0n1.in‚ 65025.0546875$ disk./.usageGB‚574.3976364135742 disk.dm-1.out‚ 1610.40625 disk.sda2.in‚0( proc.memory.availableMB‚ 2985.1796875$ memory_percent‚81.77990913391113* proc.memory.percent‚17.087507247924805( disk.nvme0c0n1.out‚170520.8759765625# disk.dm-0.out‚170520.8759765625 network.recv‚ 6.873406e+06* disk./.usagePercent‚38.548307331433165 disk.sda.out‚ 1610.40625& disk.rbd0.in‚1.759218604441423e+13 disk.dm-0.in‚ 65025.0546875 disk.sda1.in‚0& disk.nvme0n1.out‚170520.8759765625" proc.memory.rssMB‚ 2799.6171875 proc.cpu.threads‚17 disk.rbd0.out‚0 disk.sda3.in‚ 108.92578125 disk.sda3.out‚ 1610.40625 cpu‚25.58568024457586 disk.drbd2165.out‚0! disk.nvme0n1.in‚ 65025.0546875 disk.sda2.out‚0 disk.rbd1.in‚ 2.9619140625 disk.rbd1.out‚ 1735.12890625 disk.dm-1.in‚ 108.92578125 disk.drbd2165.in‚0 network.sent‚ 3.2295009e+07ÈÆq4¼A/ _wandbruntime‚27648  _runtime‚27648¸ú_°È½A _step‚1373 % lr-AdamW/pg1‚4.73014229604813e-05 % lr-AdamW/pg2‚4.73014229604813e-05 trainer/global_step‚16699 % _timestamp‚1.7843174642026093e+09 _runtime‚27648.419925445zÅ');AÇ % lr-AdamW/pg1‚4.73014229604813e-05 % lr-AdamW/pg2‚4.73014229604813e-05 trainer/global_step‚16699 % _timestamp‚1.7843174642026093e+09 _runtime‚27648.419925445 _step‚1373Ý <Ô4ô4¿A/ _wandbruntime‚27648  _runtime‚27648£Åì $ÀA”P TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 142456811 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚20261796200448 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚4813255935126637 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚4799646933942272 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytôO‹·ö"es.all.allocated‚1316449526026240 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 144615770 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 _runtime‚27648.421240227 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 144615770 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚4815355270828032 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚47281014 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15681484946541 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚4813257305154927 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚90118951 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 144625743 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1296187361734656 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚90119202 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚4815356642391040 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚4799646933942272 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚42837937 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚4799646907768832 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 % _timestamp‚1.7843174642038794e+09 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 142456817 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚16699 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚4815355270828032 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚4799646907768832 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚4815355270828032 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚47281014 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15681484946541 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚4813257305154927 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚90118951 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 144625743 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1296187361734656 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚90119202 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚4815356642391040 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚4799646933942272 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚42837937 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚4799646907768832 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 % _timestamp‚1.7843174642038794e+09 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 142456817 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚16699 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚4815355270828032 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚4799646907768832 JDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚4797852357386240 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 142465017 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 287099258 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚47283761 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 144634235 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 144634235 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 % _timestamp‚1.7843174655888736e+09 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚4815635568163840 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 287089265 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15683753013221 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15710633491456 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15709288094720 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 287099258 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚4799924934672384 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 144624248 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚4813534740371059 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚4813536110399461 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚42840485 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1316527803227136 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚20262955084800 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚90124241 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1296265216233472 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 142465023 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚47284000 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚42840480 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚15709288094720 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚4797852331958272 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 142465017 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15710633491456 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 142465023 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1316528205847552 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚239 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 144624248 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚4799924908498944 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995åYi4p ÄA£N WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 144624248 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚4813534740371059 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚4813536110399461 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚42840485 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1316527803227136 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚20262955084800 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚90124241 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1296265216233472 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 142465023 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚47284000 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚42840480 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚15709288094720 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚4797852331958272 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 142465017 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15710633491456 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 142465023 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1316528205847552 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚239 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 144624248 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚4799924908498944 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 trainer/global_step‚16699 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚4797852357386240 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 142465017 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 287099258 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚47283761 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 144634235 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 144634235 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 % _timestamp‚1.7843174655888736e+09 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚4815635568163840 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 287089265 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15683753013221 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15710633491456 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15709288094720 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 287099258 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚4799924934672384ß okÝ4ÅA/ _wandbruntime‚27649  _runtime‚27649$BðOÆA‘ epoch‚41 trainer/global_step‚16699 _runtime‚27649.892781702 3loss/SRqqt2/assignment_loss‚0.11225936397204342 4loss/FBt1/assignment_loss‚3.0819865961215015e-07 3loss/FBt2/assignment_loss‚7.452732733520496e-07 /loss/FRt2/detection_loss‚1.1944293924043807 0loss/FBt2/detection_loss‚0.08242729902892064 _step‚1376 0loss/FRt1/assignment_loss‚0.8797111966551804 4loss/SRqqt1/assignment_loss‚0.058845561237762446 /loss/FRt1/detection_loss‚0.4339511467555036 1loss/SRqqt1/detection_loss‚0.3488168728689858 0loss/FBt1/detection_loss‚0.40986793733974614 %loss/total_loss‚4.068768813903061 1loss/FRt2/assignment_loss‚0.07371427863836288 2loss/SRqqt2/detection_loss‚0.47474471153024184 % _timestamp‚1.7843174656758778e+09ýoûÇA– epoch‚41 trainer/global_step‚16699 _runtime‚27649.892781702 3loss/SRqqt2/assignment_loss‚0.11225936397204342 4loss/FBt1/assignment_loss‚3.0819865961215015e-07 3loss/FBt2/assignment_loss‚7.452732733520496e-07 /loss/FRt2/detection_loss‚1.1944293924043807 0loss/FBt2/detection_loss‚0.08242729902892064 _step‚1376 0loss/FRt1/assignment_loss‚0.8797111966551804 4loss/SRqqt1/assignment_loss‚0.058845561237762446 /loss/FRt1/detection_loss‚0.4339511467555036 1loss/SRqqt1/detection_loss‚0.3488168728689858 0loss/FBt1/detection_loss‚0.40986793733974614 %loss/total_loss‚4.068768813903061 1loss/FRt2/assignment_loss‚0.07371427863836288 2loss/SRqqt2/detection_loss‚0.47474471153024184 % _timestamp‚1.7843174656758778e+09à ˵Î/ÈA‚(:’ £ŒêÒ¡Ä’ø gpu.0.gpu‚89.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚331.35' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚66.27000000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ýÜÉA‚(: £Œêң̭ù disk.rbd1.in‚ 2.9619140625 disk.sda.in‚ 108.92578125 disk.drbd2165.out‚0 disk.sda.out‚ 1610.7421875& disk.rbd0.in‚1.759218604441423e+13$ memory_percent‚81.77676200866699 cpu‚25.585578747507245) disk./.usagePercent‚38.54861172439138# disk./.usageGB‚574.402172088623 disk.dm-1.in‚ 108.92578125 disk.dm-1.out‚ 1610.7421875 disk.dm-0.in‚65096.16796875* proc.memory.percent‚17.087507247924805 network.recv‚ 6.876707e+06" disk.nvme0n1.in‚65096.16796875% disk.nvme0n1.out‚170529.037109375 disk.sda1.in‚0 disk.sda3.in‚ 108.92578125 disk.sda3.out‚ 1610.7421875 disk.rbd0.out‚0 disk.sda1.out‚0" proc.memory.rssMB‚ 2799.6171875 network.sent‚ 3.2300079e+07$ disk.nvme0c0n1.in‚65096.16796875' disk.nvme0c0n1.out‚170529.037109375 disk.rbd1.out‚1737.044921875" disk.dm-0.out‚170529.037109375 proc.cpu.threads‚17 disk.sda2.in‚0( proc.memory.availableMB‚ 2985.6953125 disk.drbd2165.in‚0 disk.sda2.out‚0Fòåî”ÊA‚(:‰ ²ŒêÒºáŒø gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚275.567' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.1134 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0B•ËA‚(: ²ŒêÒ´á‚ú" disk.nvme0n1.in‚65096.58984375 disk.sda.out‚ 1611.1015625 disk.dm-1.out‚ 1611.1015625$ disk.dm-0.out‚170538.05029296875 disk.sda1.out‚0 disk.sda2.in‚0" proc.memory.rssMB‚ 2799.6171875 proc.cpu.threads‚17' disk.nvme0n1.out‚170538.05029296875 disk.drbd2165.in‚0$ disk.nvme0c0n1.in‚65096.58984375% proc.memory.availableMB‚ 2985.1875) disk./.usagePercent‚38.54924636706876$ disk./.usageGB‚574.4116287231445& disk.rbd0.in‚1.759218604441423e+13) disk.nvme0c0n1.out‚170538.05029296875$ memory_percent‚81.77986145019531 cpu‚25.585467952363295 network.sent‚ 3.236348e+07 network.recv‚ 6.882184e+06 disk.sda1.in‚0 disk.rbd0.out‚0 disk.dm-0.in‚65096.58984375 disk.rbd1.in‚ 2.9619140625 disk.sda.in‚ 108.92578125 disk.dm-1.in‚ 108.92578125 disk.sda3.in‚ 108.92578125 disk.sda3.out‚ 1611.1015625 disk.sda2.out‚0 disk.rbd1.out‚1737.044921875* proc.memory.percent‚17.087507247924805 disk.drbd2165.out‚0ûš{h”ÌA‚(:‰ ÁŒêÒ‹’åö gpu.0.gpu‚99.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚294.678' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚58.9356 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0/`ÍA‚(:û ÁŒêÒ£ÕÉù disk.dm-1.in‚ 108.92578125" proc.memory.rssMB‚ 2799.6171875 proc.cpu.threads‚17& disk.rbd0.in‚1.759218604441423e+13 disk.dm-0.in‚65096.80859375 network.sent‚ 3.2367072e+07& disk.nvme0c0n1.out‚170550.44140625$ disk.nvme0n1.out‚170550.44140625! disk.dm-0.out‚170550.44140625 disk.sda3.out‚1611.375 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.sda2.out‚0 disk.dm-1.out‚1611.375 disk.rbd0.out‚0 disk.sda1.out‚0 disk.sda3.in‚ 108.92578125) disk./.usagePercent‚38.54955869626054$ disk.nvme0c0n1.in‚65096.80859375 cpu‚25.58535937729428" disk.nvme0n1.in‚65096.80859375 disk.sda2.in‚0 disk.sda.out‚1611.375( proc.memory.availableMB‚ 2985.1953125 disk.rbd1.out‚ 1739.796875* proc.memory.percent‚17.087507247924805$ memory_percent‚81.77981376647949 disk.rbd1.in‚ 2.9619140625 disk.sda.in‚ 108.92578125 disk.sda1.in‚0 network.recv‚ 6.884825e+06$ disk./.usageGB‚574.4162826538086 ý·dŸÎA‚(:” ÐŒêÒ¥”÷ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚300.279' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚60.055800000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0.k7ÁÏA‚(:’ ÐŒêÒݽžú disk.sda1.in‚0 cpu‚25.58525784504703 disk.sda.out‚ 1611.91015625$ disk.dm-0.out‚170561.78271484375( proc.memory.availableMB‚ 2985.3671875( disk./.usagePercent‚38.5500010772797 disk.sda2.in‚0$ memory_percent‚81.77876472473145' disk.nvme0n1.out‚170561.78271484375 disk.rbd0.out‚0 disk.sda3.out‚ 1611.91015625 disk.drbd2165.in‚0) disk.nvme0c0n1.out‚170561.78271484375" proc.memory.rssMB‚ 2799.6171875 proc.cpu.threads‚17$ disk./.usageGB‚574.4228744506836" disk.nvme0n1.in‚65097.01171875$ disk.nvme0c0n1.in‚65097.01171875 disk.rbd1.out‚ 1739.796875& disk.rbd0.in‚1.759218604441423e+13 disk.drbd2165.out‚0 disk.sda2.out‚0 disk.rbd1.in‚ 2.9619140625 disk.dm-1.out‚ 1611.91015625 disk.dm-0.in‚65097.01171875 disk.sda1.out‚0 disk.sda3.in‚ 108.92578125* proc.memory.percent‚17.087507247924805 network.sent‚ 3.2369931e+07 network.recv‚ 6.887408e+06 disk.sda.in‚ 108.92578125 disk.dm-1.in‚ 108.92578125}(ˆÂŸÐA‚(:” ߌêÒ¯Þëö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚277.162' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚55.432399999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0÷™;¶ÑA‚(:† ߌêÒøçîø disk.sda.in‚ 108.92578125 disk.sda3.in‚ 108.92578125 disk.sda2.out‚0( proc.memory.availableMB‚ 2985.5859375 cpu‚25.585142289154653" disk.dm-0.out‚170574.580078125 disk.sda1.in‚0 disk.drbd2165.in‚0 disk.drbd2165.out‚0" proc.memory.rssMB‚ 2799.6171875 disk.sda.out‚ 1612.10546875 disk.dm-1.out‚ 1612.10546875 disk.sda2.in‚0* proc.memory.percent‚17.087507247924805$ disk./.usageGB‚574.4298210144043 disk.rbd0.out‚0 disk.sda1.out‚0 disk.sda3.out‚ 1612.10546875! disk.nvme0c0n1.in‚ 65212.15625 disk.nvme0n1.in‚ 65212.15625$ memory_percent‚81.77742958068848* disk./.usagePercent‚38.550467266999554 disk.dm-1.in‚ 108.92578125 disk.rbd1.out‚ 1739.921875 network.sent‚ 3.2373126e+07' disk.nvme0c0n1.out‚170574.580078125% disk.nvme0n1.out‚170574.580078125 disk.rbd1.in‚ 2.9619140625 proc.cpu.threads‚17 network.recv‚ 6.890626e+06& disk.rbd0.in‚1.759218604441423e+13 disk.dm-0.in‚ 65212.15625ô>±4ÒA/ _wandbruntime‚27726  _runtime‚27726j­†ÿËÓAÅ ' lr-AdamW/pg2‚4.5911671933426835e-05 trainer/global_step‚16749 $ _timestamp‚1.784317542734777e+09 _runtime‚27726.951941206 _step‚1377 ' lr-AdamW/pg1‚4.5911671933426835e-05ã¥ÔsÐÔAÊ ' lr-AdamW/pg2‚4.5911671933426835e-05 trainer/global_step‚16749 $ _timestamp‚1.784317542734777e+09 _runtime‚27726.951941206 _step‚1377 ' lr-AdamW/pg1‚4.5911671933426835e-05á Îî/ª4ÕA/ _wandbruntime‚27726  _runtime‚27726.ºår(ÖA”P JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚251 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 287908633 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 142867117 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚15754678989824 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚4829302994827264 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1320383542409216 trainer/global_step‚16749 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚4813546944274432 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 _step‚1378 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15754678989824 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚246 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1300062050209792 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚20321492199424 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 145041522 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 287918612 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 145051495 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚42965087 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚4813546970447872 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚90384425 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚90384174 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚4813546944274432 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚4829301623264256 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 145051495 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚47419087 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15727722059649 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 145041522 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚47419333 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚20321457662976 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 _runtime‚27726.953142522 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚42965092 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚4811468539088896 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15729066659971 % _timestamp‚1.7843175427358234e+09 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚4829301623264256 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚4813546970447872 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15756024379392 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚4829302994827264 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 142867111 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚4827197631176835 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚4811468564516864 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 142867111 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15756024379392 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚42965092 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚4811468539088896 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15729066659971 % _timestamp‚1.7843175427358234e+09 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚4829301623264256 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚4813546970447872 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15756024379392 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚4829302994827264 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 142867111 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚4827197631176835 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚4811468564516864 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 142867111 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15756024379392 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚251 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 287908633 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 142867117 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚15754678989824 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚4829302994827264 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1320383542409216 trainer/global_step‚16749 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚84pÞR1627 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚4813546944274432 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 _step‚1378 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15754678989824 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚246 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1300062050209792 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚20321492199424 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 145041522 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 287918612 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 145051495 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚42965087 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚4813546970447872 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚90384425 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚90384174 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚4813546944274432 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚4829301623264256 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 145051495 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚47419087 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15727722059649 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0â ù}=-4ØA/ _wandbruntime‚27728  _runtime‚277285\$'ÙAžN ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚4829580549029888 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚27728.336404238 _step‚1379 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 287935310 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚4813824945004544 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚240 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 287935310 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 142875323 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 145059987 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚47421846 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1300139536617472 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15729990126329 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚4829581920600064 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 142875317 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚42967635 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚42967630 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚4827476436421369 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚4813824945004544 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚90389476 trainer/global_step‚16749 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 287925317 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15756949422080 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 287925317 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚4811746446295040 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15728645525895 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚20322689553408 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0úms±)'ÚA£N _runtime‚27728.336404238 _step‚1379 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 287935310 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚4813824945004544 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚240 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 287935310 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 142875323 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 145059987 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚47421846 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1300139536617472 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15729990126329 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚4829581920600064 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 142875317 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚42967635 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚42967630 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚4827476436421369 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚4813824945004544 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚90389476 trainer/global_step‚16749 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 287925317 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15756949422080 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 287925317 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚4811746446295040 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15728645525895 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚20322689553408 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚4829580549029888 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0ã o¨4ÛA/ _wandbruntime‚27728  _runtime‚27728|b‘lÜA 3loss/SRqqt1/assignment_loss‚0.06005115168570628 2loss/SRqqt2/detection_loss‚0.46295511878783197 0loss/FRt1/assignment_loss‚0.8053245140695441 3loss/SRqqt2/assignment_loss‚0.11779764677466979 0loss/FBt2/detection_loss‚0.11572508900674029 % _timestamp‚1.7843175442057934e+09 _runtime‚27728.422724808 2loss/FBt1/assignment_loss‚2.36144350352114e-05 1loss/SRqqt1/detection_loss‚0.4118084312788657 /loss/FBt1/detection_loss‚0.2189860384887002 &loss/total_loss‚3.8166568856681122 _step‚1380 4loss/FBt2/assignment_loss‚0.00011748038232292402 /loss/FRt1/detection_loss‚0.4082728889419446 /loss/FRt2/detection_loss‚1.1244989517362773 epoch‚41 trainer/global_step‚16749 1loss/FRt2/assignment_loss‚0.09109596008047424¥iQÑÝA” 2loss/FBt1/assignment_loss‚2.36144350352114e-05 1loss/SRqqt1/detection_loss‚0.4118084312788657 /loss/FBt1/detection_loss‚0.2189860384887002 &loss/total_loss‚3.8166568856681122 _step‚1380 4loss/FBt2/assignment_loss‚0.00011748038232292402 /loss/FRt1/detection_loss‚0.4082728889419446 /loss/FRt2/detection_loss‚1.1244989517362773 epoch‚41 trainer/global_step‚16749 1loss/FRt2/assignment_loss‚0.09109596008047424 3loss/SRqqt1/assignment_loss‚0.06005115168570628 2loss/SRqqt2/detection_loss‚0.46295511878783197 0loss/FRt1/assignment_loss‚0.8053245140695441 3loss/SRqqt2/assignment_loss‚0.11779764677466979 0loss/FBt2/detection_loss‚0.11572508900674029 % _timestamp‚1.7843175442057934e+09 _runtime‚27728.422724808ä 26¸ŸÞA‚(:” îŒêÒ“à«ø gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚156.155' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚31.230999999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0aòÿ÷ßA‚(:ý îŒêÒëîÁù network.sent‚ 3.2381392e+07 disk.dm-1.out‚ 1612.4453125 disk.nvme0c0n1.in‚ 65212.375& disk.nvme0n1.out‚170586.7333984375 disk.sda2.in‚0) disk./.usagePercent‚38.55076653980708$ disk./.usageGB‚574.4342803955078 disk.sda.in‚ 108.92578125 disk.dm-0.in‚ 65212.375 disk.drbd2165.in‚0 disk.rbd1.in‚ 2.9619140625 disk.rbd1.out‚ 1739.921875 cpu‚25.585040991109043 disk.sda.out‚ 1612.4453125 disk.sda1.in‚0 disk.sda2.out‚0 proc.cpu.threads‚17& disk.rbd0.in‚1.759218604441423e+13# disk.dm-0.out‚170586.7333984375 disk.sda1.out‚0" memory_percent‚81.778883934021" proc.memory.rssMB‚ 2799.6171875( disk.nvme0c0n1.out‚170586.7333984375) proc.memory.availableMB‚ 2985.34765625* proc.memory.percent‚17.087507247924805 disk.drbd2165.out‚0 disk.dm-1.in‚ 108.92578125 disk.rbd0.out‚0 disk.sda3.in‚ 108.92578125 network.recv‚ 6.89445e+06 disk.sda3.out‚ 1612.4453125 disk.nvme0n1.in‚ 65212.375År'ŸàA‚(:” ýŒêÒæ´œ÷ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚295.159' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚59.031800000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ö(ŠàáA‚(:‡ ýŒêÒ€ùöø disk.sda1.in‚0 disk.drbd2165.in‚0$ disk.nvme0c0n1.in‚65212.58984375( disk.nvme0c0n1.out‚170595.6318359375) disk./.usagePercent‚38.55124757796392$ disk./.usageGB‚574.4414482116699 disk.rbd0.out‚0 disk.sda1.out‚0 disk.dm-0.in‚65212.58984375 disk.sda3.out‚ 1612.65625" disk.nvme0n1.in‚65212.58984375" proc.memory.rssMB‚ 2799.6171875 disk.sda.out‚ 1612.65625# disk.dm-0.out‚170595.6318359375 disk.sda2.out‚0 disk.rbd1.in‚ 2.9619140625 disk.rbd1.out‚ 1739.984375& disk.rbd0.in‚1.759218604441423e+13 disk.sda2.in‚0 disk.sda3.in‚ 108.92578125 disk.drbd2165.out‚0 network.sent‚ 3.2433779e+07 network.recv‚ 6.900089e+06 disk.dm-1.in‚ 108.92578125& disk.nvme0n1.out‚170595.6318359375$ memory_percent‚81.78036212921143* proc.memory.percent‚17.087507247924805 cpu‚25.5849241749396 proc.cpu.threads‚17 disk.sda.in‚ 108.92578125 disk.dm-1.out‚ 1612.65625) proc.memory.availableMB‚ 2985.10546875G¿‘XžâA‚(:“ ŒêÒÖ•óö gpu.0.gpu‚66.0 gpu.0.memory‚20, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚340.097' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚68.01939999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0α ŠãA‚(:ú ŒêÒÙ„üø disk.dm-1.out‚ 1612.91015625 disk.dm-0.in‚65213 disk.rbd1.out‚ 1739.984375 disk.sda.out‚ 1612.91015625 proc.cpu.threads‚17 disk.sda1.out‚0 disk.sda2.in‚0 disk.rbd1.in‚ 2.9619140625 disk.sda.in‚ 108.92578125 cpu‚25.584824290499643 disk.sda1.in‚0 disk.nvme0c0n1.in‚65213) disk.nvme0c0n1.out‚170611.99755859375 disk.sda2.out‚0$ memory_percent‚81.78009986877441" proc.memory.rssMB‚ 2799.6171875* proc.memory.percent‚17.087507247924805 network.sent‚ 3.2436638e+07 disk.dm-1.in‚ 108.92578125 disk.drbd2165.out‚0( proc.memory.availableMB‚ 2985.1484375$ disk./.usageGB‚574.4478416442871 disk.rbd0.out‚0 disk.sda3.out‚ 1612.91015625$ disk.dm-0.out‚170611.99755859375 disk.sda3.in‚ 108.92578125' disk.nvme0n1.out‚170611.99755859375* disk./.usagePercent‚38.551676646591304& disk.rbd0.in‚1.759218604441423e+13 disk.drbd2165.in‚0 disk.nvme0n1.in‚65213 network.recv‚ 6.902824e+06BQ‰×“äA‚(:ˆ ›êÒ‰øíö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚254.525' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚50.905 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0’Uq åA‚(:• ›êÒ‰˜æø& disk.rbd0.in‚1.759218604441423e+13 disk.sda1.in‚0 disk.rbd1.in‚ 2.9619140625 disk.sda3.out‚ 1613.14453125 disk.drbd2165.out‚0) disk./.usagePercent‚38.55212542779883$ disk./.usageGB‚574.4545288085938 disk.sda.in‚ 108.92578125 disk.sda1.out‚0 disk.sda2.out‚0) proc.memory.availableMB‚ 2985.58203125 network.sent‚ 3.2493626e+07 disk.sda.out‚ 1613.14453125 disk.dm-1.out‚ 1613.14453125 disk.sda3.in‚ 108.92578125$ disk.nvme0c0n1.in‚65315.87890625" disk.nvme0n1.in‚65315.87890625 disk.sda2.in‚0 proc.cpu.threads‚17 disk.dm-1.in‚ 108.92578125 disk.rbd0.out‚0 disk.dm-0.in‚65315.87890625) disk.nvme0c0n1.out‚170619.84814453125 network.recv‚ 6.909263e+06$ disk.dm-0.out‚170619.84814453125' disk.nvme0n1.out‚170619.84814453125 disk.rbd1.out‚ 1740.140625 disk.drbd2165.in‚0$ memory_percent‚81.77745342254639" proc.memory.rssMB‚ 2799.6171875 cpu‚25.584706912662313* proc.memory.percent‚17.0875072479248051Gj‘æA‚(:† ªêÒÛœóö gpu.0.gpu‚29.0 gpu.0.memory‚9, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚46.0 gpu.0.powerWatts‚341.58' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚68.316 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ïü rçA‚(:‚ ªêҩݾù disk.sda1.out‚0' disk.nvme0c0n1.out‚170628.958984375) disk./.usagePercent‚38.55240908414677 disk.sda.out‚ 1613.41015625& disk.rbd0.in‚1.759218604441423e+13* proc.memory.percent‚17.087507247924805$ disk./.usageGB‚574.4587554931641" disk.dm-0.out‚170628.958984375 disk.sda1.in‚0 disk.sda3.out‚ 1613.41015625$ proc.memory.availableMB‚2985.375 network.sent‚ 3.2497346e+07 disk.rbd1.in‚ 2.9619140625 disk.sda.in‚ 108.92578125 cpu‚25.584605980456086 proc.cpu.threads‚17 disk.dm-1.in‚ 108.92578125 network.recv‚ 6.912667e+06 disk.rbd1.out‚ 1740.1484375 disk.dm-0.in‚ 65316.09375 disk.nvme0n1.in‚ 65316.09375$ memory_percent‚81.77871704101562" proc.memory.rssMB‚ 2799.6171875þ@Šæù disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-1.out‚ 1613.41015625 disk.rbd0.out‚0 disk.drbd2165.in‚0% disk.nvme0n1.out‚170628.958984375 disk.sda3.in‚ 108.92578125 disk.drbd2165.out‚0! disk.nvme0c0n1.in‚ 65316.09375­º{¤4èA/ _wandbruntime‚27805  _runtime‚27805 þ<6ÊéAÄ _runtime‚27805.641885052 _step‚1381 & lr-AdamW/pg1‚4.454119015209731e-05 & lr-AdamW/pg2‚4.454119015209731e-05 trainer/global_step‚16799 % _timestamp‚1.7843176214247015e+09²SRCÏêAÉ _runtime‚27805.641885052 _step‚1381 & lr-AdamW/pg1‚4.454119015209731e-05 & lr-AdamW/pg2‚4.454119015209731e-05 trainer/global_step‚16799 % _timestamp‚1.7843176214247015e+09å Y¨I!4ëA/ _wandbruntime‚27805  _runtime‚27805ÏО˜(ìA“P D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚4843249347263488 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚47557013 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1303936002502656 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚4825362653425664 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 288744685 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 145477247 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _step‚1382 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15800994920448 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 143277417 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1303936370593792 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚4841136587170453 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚4843247975700480 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15775303773079 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚4843247975700480 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚4827447006953472 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚4825362627997696 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 145477247 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 trainer/global_step‚16799 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 288754664 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚43092237 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 143277417 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15802340310016 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚4841137957198743 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15773959172757 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚4827447006953472 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚90649255 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 143277411Ù¨ÔN(íA˜P YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚47557013 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1303936002502656 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚4825362653425664 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 288744685 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 145477247 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _step‚1382 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15800994920448 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 143277417 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1303936370593792 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚4841136587170453 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚4843247975700480 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15775303773079 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚4843247975700480 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚4827447006953472 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚4825362627997696 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 145477247 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 trainer/global_step‚16799 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 288754664 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚43092237 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 143277417 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15802340310016 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚4841137957198743 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15773959172757 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚4827447006953472 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚90649255 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 143277411 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚4843249347263488 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736æ Uù¤4îA/ _wandbruntime‚27807  _runtime‚27807ÓÁŽÇ$'ïAžN OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚90654504 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 288771362 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚90654252 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚4827725007683584 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚20382200438784 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚247 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15803265352704 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 288761369 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚4841416762443277 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚4825640535203840 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7843176228117332e+09 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 288761369 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 143285617 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚16799 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 145475752 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 _runtime‚27807.029153596 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚252 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚4827724981510144 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 143285617 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚20382234968064 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚43094780 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1304013857001472 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚4843528273036288 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚4843528273036288 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15803265352704 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 143285623 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚4825640509775872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚4843526901466112 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 145485739 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15776227239437 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15774882639003 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0³ÿ}1ðA£N OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚90654504 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 EDeviceStatsMonitor.on_trxj©|ø!ain_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 288771362 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚90654252 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚4827725007683584 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚20382200438784 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚247 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15803265352704 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 288761369 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚4841416762443277 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚4825640535203840 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7843176228117332e+09 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 288761369 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 143285617 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 QDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚16799 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 145475752 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 _runtime‚27807.029153596 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚252 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚4827724981510144 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 143285617 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚20382234968064 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚43094780 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1304013857001472 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚4843528273036288 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚4843528273036288 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15803265352704 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 143285623 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚4825640509775872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚4843526901466112 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 145485739 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15776227239437 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15774882639003 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0ç I«9–4ñA/ _wandbruntime‚27807  _runtime‚27807ŸžòAŽ /loss/FBt1/detection_loss‚0.1653455344965649 %loss/total_loss‚3.812375311503025 epoch‚41 % _timestamp‚1.7843176228982446e+09 _runtime‚ 27807.1151861 0loss/FRt1/assignment_loss‚0.8611064461944495 4loss/FBt1/assignment_loss‚3.5118285869718937e-07 2loss/SRqqt2/detection_loss‚0.44477962424501577 trainer/global_step‚16799 3loss/SRqqt1/assignment_loss‚0.04397789061371744 3loss/FBt2/assignment_loss‚7.687161638177014e-07 /loss/FRt1/detection_loss‚0.3877012491900348 1loss/SRqqt1/detection_loss‚0.4364954016159526 _step‚1384 1loss/FRt2/assignment_loss‚0.10777741622924805 3loss/SRqqt2/assignment_loss‚0.10290015911098635 /loss/FRt2/detection_loss‚1.2047861302526242 1loss/FBt2/detection_loss‚0.057504339655408715WHVRóA“ 3loss/FBt2/assignment_loss‚7.687161638177014e-07 /loss/FRt1/detection_loss‚0.3877012491900348 1loss/SRqqt1/detection_loss‚0.4364954016159526 _step‚1384 1loss/FRt2/assignment_loss‚0.10777741622924805 3loss/SRqqt2/assignment_loss‚0.10290015911098635 /loss/FRt2/detection_loss‚1.2047861302526242 1loss/FBt2/detection_loss‚0.057504339655408715 /loss/FBt1/detection_loss‚0.1653455344965649 %loss/total_loss‚3.812375311503025 epoch‚41 % _timestamp‚1.7843176228982446e+09 _runtime‚ 27807.1151861 0loss/FRt1/assignment_loss‚0.8611064461944495 4loss/FBt1/assignment_loss‚3.5118285869718937e-07 2loss/SRqqt2/detection_loss‚0.44477962424501577 trainer/global_step‚16799 3loss/SRqqt1/assignment_loss‚0.04397789061371744è à+U8’ôA‚(:‡ ¹êÒסÞö gpu.0.gpu‚10.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚343.705' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚68.741 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0• ¨ÝõA‚(:‰ ¹êÒ‡óÌú" proc.memory.rssMB‚ 2799.6171875 disk.sda2.in‚0 disk.sda2.out‚0$ memory_percent‚81.78069591522217* proc.memory.percent‚17.087507247924805 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.sda.in‚ 108.92578125 disk.rbd0.out‚0 disk.dm-0.in‚65316.50390625 disk.dm-1.in‚ 108.92578125 cpu‚25.584737910762858 proc.cpu.threads‚17 disk.sda.out‚ 1613.8828125) proc.memory.availableMB‚ 2985.05078125& disk.nvme0c0n1.out‚170642.23828125 network.recv‚ 6.916034e+06$ disk./.usageGB‚574.4654312133789! disk.dm-0.out‚170642.23828125 disk.sda3.out‚ 1613.8828125 disk.drbd2165.out‚0$ disk.nvme0n1.out‚170642.23828125 disk.rbd1.out‚ 1740.2109375& disk.rbd0.in‚1.759218604441423e+13$ disk.nvme0c0n1.in‚65316.50390625 network.sent‚ 3.250237e+07 disk.sda1.in‚0 disk.rbd1.in‚ 2.9619140625 disk.dm-1.out‚ 1613.8828125) disk./.usagePercent‚38.55285709733169 disk.sda3.in‚ 108.92578125" disk.nvme0n1.in‚65316.50390625bN6öA‚(:’ ÈêÒîñáö gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚36.0 gpu.0.powerWatts‚109.588' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.917599999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0MÀ˜u÷A‚(:ˆ ÈêÒÀ™±ø$ disk./.usageGB‚574.4723739624023 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd2165.out‚0" disk.nvme0n1.in‚65316.72265625& disk.rbd0.in‚1.759218604441423e+13& proc.memory.availableMB‚ 2982.09375$ disk.nvme0c0n1.in‚65316.72265625 network.recv‚ 6.921422e+06 disk.sda3.in‚ 108.92578125' disk.nvme0n1.out‚170653.70361328125 disk.rbd1.in‚ 2.9619140625 disk.rbd1.out‚ 1740.2109375 disk.dm-1.in‚ 108.92578125* proc.memory.percent‚17.105603218078613 network.sent‚ 3.2565656e+07 disk.dm-0.in‚65316.72265625$ disk.dm-0.out‚170653.70361328125 disk.sda.in‚ 108.92578125 disk.dm-1.out‚ 1614.1875) disk./.usagePercent‚38.55332303104401 disk.sda2.in‚0 disk.rbd0.out‚0# proc.memory.rssMB‚ 2802.58203125 disk.drbd2165.in‚0) disk.nvme0c0n1.out‚170653.70361328125 disk.sda2.out‚0 cpu‚25.586052074647867 disk.sda3.out‚ 1614.1875 proc.cpu.threads‚17 disk.sda.out‚ 1614.1875$ memory_percent‚81.79874420166016&儼4øA/ _wandbruntime‚27833  _runtime‚27833*ëœA>ùA–| 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6343187689781189 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9861670136451721 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 8"Purity/1FRt*SRqqt2FBt/event_purity‚0.670351505279541 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9841936826705933 +particle/specificity‚0.8959917426109314 _step‚1385 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.4837305545806885 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 _runtime‚27833.552551122 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9041630029678345 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.14402951300144196 8"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.743571937084198 /particle/accuracy_3_of_4‚0.9974937438964844 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7142857313156128 particle/accuracy_5_of_5‚1 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.26851850748062134 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9888888597488403 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7562643885612488 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 % _timestamp‚1.7843176493347528e+09 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.5541905760765076 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.7435752749443054 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9686591625213623 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9936507940292358 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.19591526687145233 ,jet/accuracy_3_of_3‚0.040591925382614136 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.22087149322032928 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.004297424107789993 6validation_average_jet_accuracy‚0.3238338828086853 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.10094588249921799 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.5565159916877747 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9939178228378296 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6047590970993042 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.7392290830612183 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 /particle/accuracy_4_of_4‚0.9399129152297974 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8448857665061951 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.10958608239889145 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7332874536514282 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7622554302215576 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.26923078298568726 jet/accuracy_4_of_5‚0 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6695405840873718 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6323516964912415 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.08833333104848862 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7297503352165222 9"Purity/*FRt2SRqqt0FBt/event_purity‚0.2150696963071823 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6916977763175964 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.12077710777521133 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.14477461576461792 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3530319035053253 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.21104548871517181 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.10406927019357681 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.09809001535177231 +jet/accuracy_2_of_2‚0.11477762460708618 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6908376812934875 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.017112625762820244 -"Purity/2FRt0SRqqt1FBt/event_purity‚0.1875 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.43981480598449707 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.16774454712867737 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6047590970993042 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.12889982759952545 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6552121639251709 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.8653846383094788 *loss/val_total_loss‚12.466998462951707 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.25653108954429626 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9892206788063049 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.15825895965099335 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9906229376792908 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.2222222238779068 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.027133163064718246 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.15332235395908356 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 particle/accuracy_2_of_5‚1 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.8657407164573669 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9838355183601379 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 particle/accuracy_3_of_3‚1 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9775912165641785 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6061415672302246 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6720970869064331 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8888888955116272 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.5832951068878174 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9827920198440552 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.02656625211238861 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.8500000238418579 +particle/sensitivity‚0.5844063758850098 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.26923078298568726 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9851953387260437 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.7735847234725952 *jet/accuracy_1_of_1‚0.2809669077396393 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.19587218761444092 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.008129189722239971 *jet/accuracy_1_of_4‚0.9616509675979614 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.5598958134651184 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9843295812606812 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7857142686843872 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.7407407760620117 - Purity/0FRt2SRqqt1FBt/FBt_purity‚0.703125 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.1198277696967125 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.2222222238779068 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9966966509819031 /"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.578125 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_5‚1 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.13561749458312988 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7546296119689941 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.02417675405740738 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7334835529327393 *jet/accuracy_2_of_4‚0.6791807413101196 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.12652379274368286 particle/accuracy_2_of_3‚1 particle/accuracy_2_of_4‚1 jet/accuracy_3_of_5‚1 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6453142166137695 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.4697258472442627 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.1318768709897995 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6813642978668213 'particle/f_score‚0.5948742628097534 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.7855890393257141 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.5486111044883728 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9863317608833313 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9909250140190125 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9966966509819031 particle/accuracy_2_of_2‚1 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6292302012443542 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5059515833854675 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.08730188757181168 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 8"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.619853675365448 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7454081773757935 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 6 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.997686505317688 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.027550866827368736 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.13278508186340332 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6682813167572021 epoch‚41 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 particle/accuracy_1_of_1‚1 particle/accuracy_1_of_4‚1 8"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.684313952922821 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8228169679641724 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.024405337870121002 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9916297197341919 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6690248847007751 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 +jet/accuracy_2_of_3‚0.47774967551231384 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5920861959457397 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 *jet/accuracy_1_of_3‚0.9046705961227417 (particle/accuracy‚0.8290745615959167 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.625 jet/accuracy_2_of_5‚1 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.5833280086517334 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6195029020309448 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7546296119689941 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.020288407802581787 *jet/accuracy_1_of_2‚0.7293843030929565 jet/accuracy_1_of_5‚1 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.13247820734977722 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.26851850748062134 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9113219976425171 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7578047513961792 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.023730015382170677 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 ) Purity/2FRt*SRqqt2FBt/FRt_purity‚0.75 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6359597444534302 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.49416667222976685 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.17902421951293945 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7477702498435974 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6660550236701965 particle/accuracy_3_of_5‚1 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5009506940841675 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9925084114074707 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.44729751348495483 particle/accuracy_1_of_3‚1 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4497929513454437 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.7407407760620117 - Purity/0FRt2SRqqt*FBt/FBt_purity‚0.703125 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.42500001192092896 jet/accuracy_5_of_5‚0 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.1318768709897995 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7392290830612183 particle/accuracy_1_of_2‚1 particle/accuracy_4_of_5‚1 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 trainer/global_step‚16799 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.14602752029895782 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9756953716278076 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9753750562667847 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6844531297683716 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.1287223845720291 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.017112625762820244 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.13393999636173248 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6550239324569702 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.13190573453903198 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9715638160705566 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.11447078734636307 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.8500000238418579 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.010551336221396923 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7422209978103638 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 )jet/accuracy_3_of_4‚0.043703593313694 ,jet/accuracy_4_of_4‚0.013899499550461769 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.2262207418680191 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3432900309562683 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9873457551002502 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8228169679641724 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.12619277834892273jv„´ úA›| 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8448857665061951 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.10958608239889145 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7332874536514282 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7622554302215576 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.26923078298568726 jet/accuracy_4_of_5‚0 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6695405840873718 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6323516964912415 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.08833333104848862 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7297503352165222 9"Purity/*FRt2SRqqt0FBt/event_purity‚0.2150696963071823 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6916977763175964 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.12077710777521133 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.14477461576461792 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3530319035053253 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.21104548871517181 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.10406927019357681 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.09809001535177231 +jet/accuracy_2_of_2‚0.11477762460708618 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6908376812934875 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.017112625762820244 -"Purity/2FRt0SRqqt1FBt/event_purity‚0.1875 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.43981480598449707 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.16774454712867737 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6047590970993042 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.12889982759952545 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6552121639251709 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.8653846383094788 *loss/val_total_loss‚12.466998462951707 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.25653108954429626 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9892206788063049 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.15825895965099335 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9906229376792908 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.2222222238779068 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.027133163064718246 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.15332235395908356 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 particle/accuracy_2_of_5‚1 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.8657407164573669 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9838355183601379 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 particle/accuracy_3_of_3‚1 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9775912165641785 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6061415672302246 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6720970869064331 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8888888955116272 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.5832951068878174 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9827920198440552 :"Purity/1FRt1SRqqt*FBt/µÂÖm0event_purity‚0.02656625211238861 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.8500000238418579 +particle/sensitivity‚0.5844063758850098 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.26923078298568726 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9851953387260437 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.7735847234725952 *jet/accuracy_1_of_1‚0.2809669077396393 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.19587218761444092 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.008129189722239971 *jet/accuracy_1_of_4‚0.9616509675979614 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.5598958134651184 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9843295812606812 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7857142686843872 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.7407407760620117 - Purity/0FRt2SRqqt1FBt/FBt_purity‚0.703125 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.1198277696967125 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.2222222238779068 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9966966509819031 /"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.578125 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_5‚1 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.13561749458312988 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7546296119689941 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.02417675405740738 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7334835529327393 *jet/accuracy_2_of_4‚0.6791807413101196 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.12652379274368286 particle/accuracy_2_of_3‚1 particle/accuracy_2_of_4‚1 jet/accuracy_3_of_5‚1 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6453142166137695 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.4697258472442627 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.1318768709897995 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6813642978668213 'particle/f_score‚0.5948742628097534 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.7855890393257141 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.5486111044883728 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9863317608833313 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9909250140190125 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9966966509819031 particle/accuracy_2_of_2‚1 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6292302012443542 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5059515833854675 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.08730188757181168 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 8"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.619853675365448 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7454081773757935 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 6 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.997686505317688 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.027550866827368736 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.13278508186340332 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6682813167572021 epoch‚41 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 particle/accuracy_1_of_1‚1 particle/accuracy_1_of_4‚1 8"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.684313952922821 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8228169679641724 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.024405337870121002 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9916297197341919 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6690248847007751 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 +jet/accuracy_2_of_3‚0.47774967551231384 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5920861959457397 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 *jet/accuracy_1_of_3‚0.9046705961227417 (particle/accuracy‚0.8290745615959167 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.625 jet/accuracy_2_of_5‚1 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.5833280086517334 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6195029020309448 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7546296119689941 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.020288407802581787 *jet/accuracy_1_of_2‚0.7293843030929565 jet/accuracy_1_of_5‚1 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.13247820734977722 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.26851850748062134 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9113219976425171 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7578047513961792 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.023730015382170677 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 ) Purity/2FRt*SRqqt2FBt/FRt_purity‚0.75 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6359597444534302 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.49416667222976685 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.17902421951293945 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7477702498435974 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6660550236701965 particle/accuracy_3_of_5‚1 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5009506940841675 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9925084114074707 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.44729751348495483 particle/accuracy_1_of_3‚1 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4497929513454437 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.7407407760620117 - Purity/0FRt2SRqqt*FBt/FBt_purity‚0.703125 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.42500001192092896 jet/accuracy_5_of_5‚0 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.1318768709897995 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7392290830612183 particle/accuracy_1_of_2‚1 particle/accuracy_4_of_5‚1 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 trainer/global_step‚16799 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.14602752029895782 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9756953716278076 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9753750562667847 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6844531297683716 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.1287223845720291 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.017112625762820244 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.13393999636173248 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6550239324569702 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.13190573453903198 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9715638160705566 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.11447078734636307 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.8500000238418579 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.010551336221396923 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7422209978103638 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 )jet/accuracy_3_of_4‚0.043703593313694 ,jet/accuracy_4_of_4‚0.013899499550461769 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.2262207418680191 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3432900309562683 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9873457551002502 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8228169679641724 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.12619277834892273 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6343187689781189 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9861670136451721 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 8"Purity/1FRt*SRqqt2FBt/event_purity‚0.670351505279541 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9841936826705933 +particle/specificity‚0.8959917426109314 _step‚1385 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.4837305545806885 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 _runtime‚27833.552551122 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9041630029678345 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.14402951300144196 8"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.743571937084198 /particle/accuracy_3_of_4‚0.9974937438964844 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7142857313156128 particle/accuracy_5_of_5‚1 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.26851850748062134 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9888888597488403 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7562643885612488 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 % _timestamp‚1.7843176493347528e+09 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.5541905760765076 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.7435752749443054 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9686591625213623 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9936507940292358 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.19591526687145233 ,jet/accuracy_3_of_3‚0.040591925382614136 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.22087149322032928 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.004297424107789993 6validation_average_jet_accuracy‚0.3238338828086853 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.10094588249921799 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.5565159916877747 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9939178228378296 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6047590970993042 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.7392290830612183 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 /particle/accuracy_4_of_4‚0.9399129152297974é r]V”ûA‚(:‰ ×êÒçÒëö gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚277.663' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.5326 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0pÙüA‚(: ×êҬȶù disk.drbd2165.out‚0 disk.sda1.in‚0# disk.nvme0c0n1.in‚ 65416.0546875& disk.nvme0n1.out‚170661.8876953125! disk.nvme0n1.in‚ 65416.0546875 disk.rbd1.out‚ 1740.3046875 disk.dm-1.in‚ 108.92578125$ memory_percent‚81.79795742034912) proc.memory.availableMB‚ 2982.22265625 cpu‚25.586802285964826# disk./.usageGB‚574.479434967041( disk.nvme0c0n1.out‚170661.8876953125& disk.rbd0.in‚1.759218604441423e+13 disk.rbd0.out‚0# proc.memory.rssMB‚ 2802.58203125 network.sent‚ 3.2569766e+07) disk./.usagePercent‚38.55379690098989 disk.sda1.out‚0 disk.rbd1.in‚ 2.9619140625 disk.dm-1.out‚ 1614.36328125 disk.sda2.out‚0 disk.sda.out‚ 1614.36328125 proc.cpu.threads‚17 network.recv‚ 6.924203e+06 disk.sda3.in‚ 108.92578125 disk.sda2.in‚0 disk.sda.in‚ 108.92578125# disk.dm-0.out‚170661.8876953125 disk.sda3.out‚ 1614.36328125 disk.dm-0.in‚ 65416.0546875* proc.memory.percent‚17.105603218078613 disk.drbd2165.in‚0"Klc”ýA‚(:‰ æêÒ¢Òìö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚302.696' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚60.5392 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¼£d¿þA‚(:“ æêÒÕŒÚù* disk./.usagePercent‚38.554088749578916 disk.drbd2165.in‚0 disk.dm-1.out‚ 1614.66796875 disk.sda3.in‚ 108.92578125' disk.nvme0n1.out‚170676.80419921875 disk.sda2.out‚0 disk.dm-1.in‚ 108.92578125 network.recv‚ 6.929281e+06# disk.nvme0c0n1.in‚ 65416.2578125) disk.nvme0c0n1.out‚170676.80419921875 disk.rbd0.out‚0$ disk.dm-0.out‚170676.80419921875 disk.rbd1.in‚ 2.9619140625 disk.sda.in‚ 108.92578125 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda.out‚ 1614.66796875# proc.memory.rssMB‚ 2802.58203125 disk.drbd2165.out‚0 disk.dm-0.in‚ 65416.2578125 proc.cpu.threads‚17 network.sent‚ 3.2627696e+07$ disk./.usageGB‚574.4837837219238 disk.sda3.out‚ 1614.66796875! disk.nvme0n1.in‚ 65416.2578125& disk.rbd0.in‚1.759218604441423e+13$ memory_percent‚81.79812431335449* proc.memory.percent‚17.105603218078613 cpu‚25.58670124265792 disk.rbd1.out‚ 1740.3046875( proc.memory.availableMB‚ 2982.1953125­x(ŸÿA‚(:” õêÒ„„ÿö gpu.0.gpu‚99.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚222.206' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚44.441199999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0üùxЀB‚(:‰ õêÒðÙÚù# disk.nvme0c0n1.in‚ 65416.4765625% disk.nvme0n1.out‚170685.103515625 disk.rbd0.out‚0 disk.sda2.in‚0 disk.rbd1.out‚ 1740.3671875" disk.dm-0.out‚170685.103515625 disk.sda1.in‚0# proc.memory.rssMB‚ 2802.58203125 proc.cpu.threads‚17 disk.sda3.out‚ 1614.9765625 disk.drbd2165.out‚0 network.sent‚ 3.2632547e+07) disk./.usagePercent‚38.55454751508028$ disk./.usageGB‚574.4906196594238 disk.drbd2165.in‚0 disk.sda.out‚ 1614.9765625 disk.dm-1.out‚ 1614.9765625& disk.rbd0.in‚1.759218604441423e+13* proc.memory.percent‚17.105603218078613 network.recv‚ 6.931626e+06 disk.rbd1.in‚ 2.9619140625 disk.dm-1.in‚ 108.92578125' disk.nvme0c0n1.out‚170685.103515625 disk.sda1.out‚0# memory_percent‚81.7981481552124 disk.sda3.in‚ 108.92578125! disk.nvme0n1.in‚ 65416.4765625 disk.sda2.out‚0 disk.sda.in‚ 108.92578125 disk.dm-0.in‚ 65416.4765625 cpu‚25.58660086351693) proc.memory.availableMB‚ 2982.19140625úÉ”B‚(:‰ „ŽêÒ¨íë÷ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚270.152' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚54.0304 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0bu-D‚B‚(: „ŽêÒ½û½ø disk.sda.out‚ 1615.1015625 disk.dm-1.out‚ 1615.1015625* proc.memory.percent‚17.105603218078613$ disk./.usageGB‚574.4973068237305 disk.rbd0.out‚0 disk.drbd2165.out‚0# disk.nvme0c0n1.in‚ 65416.8984375) disk.nvme0c0n1.out‚170700.04638671875 disk.sda3.out‚ 1615.1015625' proc.memory.availableMB‚ 2982.109375 network.recv‚ 6.936102e+06 disk.sda2.out‚0$ memory_percent‚81.79864883422852# proc.memory.rssMB‚ 2802.58203125$ disk.dm-0.out‚170700.04638671875 disk.sda1.out‚0 disk.dm-1.in‚ 108.92578125 disk.drbd2165.in‚0* disk./.usagePercent‚38.554996296287804 disk.dm-0.in‚ 65416.8984375 disk.sda2.in‚0 disk.rbd1.in‚ 2.9619140625 proc.cpu.threads‚17& disk.rbd0.in‚1.759218604441423e+13 disk.sda1.in‚0 disk.sda3.in‚ 108.92578125! disk.nvme0n1.in‚ 65416.8984375 disk.rbd1.out‚ 1740.3671875 cpu‚25.58648498330695 network.sent‚ 3.2639288e+07' disk.nvme0n1.out‚170700.04638671875 disk.sda.in‚ 108.92578125ïÚ)µƒB‚(:Ž “ŽêÒÖ”„ù$ memory_percent‚81.79879188537598 network.recv‚ 6.93952e+06) disk./.usagePercent‚38.55530632141176 disk.dm-0.in‚65449.01953125 disk.sda3.in‚ 108.92578125 disk.drbd2165.in‚0# proc.memory.rssMB‚ 2802.58203125 cpu‚25.58637479075678$ disk./.usageGB‚574.5019264221191 disk.sda.out‚ 1615.42578125& disk.rbd0.in‚1.759218604441423e+13 disk.sda2.in‚0" disk.nvme0n1.in‚65449.01953125( proc.memory.availableMB‚ 2982.0859375 disk.dm-1.out‚ 1615.42578125 disk.sda3.out‚ 1615.42578125 proc.cpu.threads‚17 disk.sda.in‚ 108.92578125 disk.dm-1.in‚ 108.92578125 disk.rbd0.out‚0 disk.rbd1.out‚ 1740.4296875" disk.dm-0.out‚170707.712890625 disk.sda2.out‚0* proc.memory.percent‚17.105603218078613 network.sent‚ 3.2643046e+07 disk.sda1.in‚0 disk.sda1.out‚0% disk.nvme0n1.out‚170707.712890625 disk.rbd1.in‚ 2.9619140625 disk.drbd2165.out‚0$ disk.nvme0c0n1.in‚65449.01953125' disk.nvme0c0n1.out‚170707.712890625zCü’„B‚(:‡ “ŽêÒö½à‰ gpu.0.gpu‚9.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚340.229' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚68.0458 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0­†ÏU4…B/ _wandbruntime‚27910  _runtime‚27910òXP ̆BÆ ' lr-AdamW/pg1‚4.3190065639976505e-05 ' lr-AdamW/pg2‚4.3190065639976505e-05 trainer/global_step‚16849 % _timestamp‚1.7843177265652401e+09 _runtime‚27910.782418299 _step‚1386÷¯OчBË trainer/global_step‚16849 % _timestamp‚1.7843177265652401e+09 _runtime‚27910.782418299 _step‚1386 ' lr-AdamW/pg1‚4.3190065639976505e-05 ' lr-AdamW/pg2‚4.3190065639976505e-05ê -4ªæ4ˆB/ _wandbruntime‚27910  _runtime‚27910äà(‰B”P QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚239 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15854498852352 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 400530432 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 143819717 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚90956514 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚47727965 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1328307272541184 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 143819723 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚15853153462784 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚4845175024893952 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15854498852352 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚4845175024893952 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚4861028152183296 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚90956270 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚47728204 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 289771725 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚4845174998720512 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚20448469316608 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 _runtime‚27910.783657268 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1328306872010752 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚20448436877312 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 143819717 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚4858908651256406 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 143819723 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 289781704 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚43228310 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚4861029523746304 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚4845174998720512 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚4843082625839104 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 trainer/global_step‚16849 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚4843082651267072 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚244 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1307858435133440 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚4858910021284696 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚43228305 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 145961981 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 289781704 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚43228310 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚4861029523746304 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚4845174998720512 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚4843082625839104 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 trainer/global_step‚16849 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 aLDeviceStatsMonitor.on_train_batch_st`p¥art/requested_bytes.large_pool.allocated‚4843082651267072 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚244 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1307858435133440 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚4858910021284696 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚43228305 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 145961981 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 143819717 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚90956514 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚47727965 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1328307272541184 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 143819723 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚15853153462784 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚4845175024893952 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15854498852352 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚4845175024893952 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚4861028152183296 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚90956270 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚47728204 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 289771725 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚4845174998720512 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚20448469316608 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0ë ˆFRi4‹B/ _wandbruntime‚27912  _runtime‚27912Ójÿó$'ŒBžN ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15828293483982 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1307935921541120 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚20449674780672 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚234 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚4861308449519104 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚43230853 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚43230848 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚15854078498304 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚27912.168044036 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 _step‚1388 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 143827929 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 145960486 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 143827923 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚90961576 trainer/global_step‚16849 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 289788409 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15855423895040 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚4859187456500828 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 289788409 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15854078498304 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 289798402 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 289798402 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1328385964412928 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 145960486 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 145970473 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚47730962 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚20449642348544 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 143827929 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚4845453025624064 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚90961815 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15826948883548 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚32432128 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚4843360533045248 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚4845452999450624 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1307936289632256 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 400523264 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚4861308449519104 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 145970473 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1328385563889664 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 % _timestamp‚1.7843177279506307e+09 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 143827923 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚47730728 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚4859188826529230 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚4845453025624064 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚4845452999450624 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15855423895040öt%)'B£N YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 _step‚1388 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 143827929 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 145960486 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 143827923 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚90961576 trainer/global_step‚16849 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 289788409 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15855423895040 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚4859187456500828 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 289788409 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15854078498304 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 289798402 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 289798402 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1328385964412928 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 145960486 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 145970473 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚47730962 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚20449642348544 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 143827929 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚4845453025624064 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚90961815 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15826948883548 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚32432128 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚4843360533045248 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚4845452999450624 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1307936289632256 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 400523264 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚4861308449519104 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 145970473 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1328385563889664 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 % _timestamp‚1.7843177279506307e+09 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 143827923 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚47730728 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚4859188826529230 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚4845453025624064 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚4845452999450624 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15855423895040 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15828293483982 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1307935921541120 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚20449674780672 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚234 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚4861308449519104 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚43230853 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚43230848 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚15854078498304 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚27912.168044036 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736ì Õwu=4ŽB/ _wandbruntime‚27912  _runtime‚27912mªJBŽ /loss/FRt2/detection_loss‚1.1901186300118232 1loss/SRqqt2/detection_loss‚0.4453979332352451 epoch‚42 _runtime‚27912.254235714 /loss/FRt1/detection_loss‚0.4412542214114534 0loss/FBt1/detection_loss‚0.34446735091025404 0loss/FBt2/detection_loss‚0.12341574690678299 %loss/total_loss‚3.979650678692093 % _timestamp‚1.7843177280373144e+09 /loss/FRt1/assignment_loss‚0.846158388604099 0loss/FRt2/assignment_loss‚0.0681902177613275 3loss/SRqqt2/assignment_loss‚0.08291205353103435 2loss/SRqqt1/detection_loss‚0.39363541432481336 3loss/SRqqt1/assignment_loss‚0.04352178333844362 4loss/FBt1/assignment_loss‚4.7683712978141555e-09 3loss/FBt2/assignment_loss‚0.0005789338884449749 trainer/global_step‚16849 _step‚1389‘–ëµB“ trainer/global_step‚16849 _step‚1389 /loss/FRt2/detection_loss‚1.1901186300118232 1loss/SRqqt2/detection_loss‚0.4453979332352451 epoch‚42 _runtime‚27912.254235714 /loss/FRt1/detection_loss‚0.4412542214114534 0loss/FBt1/detection_loss‚0.34446735091025404 0loss/FBt2/detection_loss‚0.12341574690678299 %loss/total_loss‚3.979650678692093 % _timestamp‚1.7843177280373144e+09 /loss/FRt1/assignment_loss‚0.846158388604099 0loss/FRt2/assignment_loss‚0.0681902177613275 3loss/SRqqt2/assignment_loss‚0.08291205353103435 2loss/SRqqt1/detection_loss‚0.39363541432481336 3loss/SRqqt1/assignment_loss‚0.04352178333844362 4loss/FBt1/assignment_loss‚4.7683712978141555e-09 3loss/FBt2/assignment_loss‚0.0005789338884449749í —u4²”‘B‚(:‰ ¢ŽêÒ¼å¢ø gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚275.279' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.0558 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0mÖ‰}!’B‚(:– ¢ŽêÒ”•Öù' disk.nvme0n1.out‚170718.76416015625) disk./.usagePercent‚38.55591997147134$ disk./.usageGB‚574.5110702514648 disk.sda.in‚ 108.92578125& disk.rbd0.in‚1.759218604441423e+13 disk.sda1.in‚0 disk.drbd2165.out‚0) disk.nvme0c0n1.out‚170718.76416015625 disk.rbd1.in‚ 2.9619140625 disk.sda.out‚ 1615.69921875 disk.rbd1.out‚ 1740.4296875$ memory_percent‚81.80196285247803* proc.memory.percent‚17.105603218078613 network.recv‚ 6.943081e+06 disk.sda3.in‚ 108.92578125 disk.drbd2165.in‚0$ disk.nvme0c0n1.in‚65515.90234375 disk.sda2.out‚0# proc.memory.rssMB‚ 2802.58203125 cpu‚25.58626455775799 proc.cpu.threads‚17" disk.nvme0n1.in‚65515.90234375) proc.memory.availableMB‚ 2981.56640625 network.sent‚ 3.2647539e+07 disk.dm-1.out‚ 1615.69921875 disk.dm-0.in‚65515.90234375 disk.sda3.out‚ 1615.69921875 disk.dm-1.in‚ 108.92578125 disk.rbd0.out‚0$ disk.dm-0.out‚170718.76416015625 disk.sda1.out‚0 disk.sda2.in‚0a§_ž“B‚(:“ ±ŽêÒ˜‚êö gpu.0.gpu‚48.0 gpu.0.memory‚15, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚330.913' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚66.18260000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0}ÚPï”B‚(:” ±ŽêÒÍñŸù network.recv‚ 6.947864e+06 disk.sda.in‚ 108.92578125 disk.sda.out‚ 1616.02734375 disk.dm-1.in‚ 108.92578125$ memory_percent‚81.80372714996338 disk.sda3.out‚ 1616.02734375" disk.nvme0n1.in‚65516.12109375) proc.memory.availableMB‚ 2981.27734375 cpu‚25.5861564758931 disk.dm-0.in‚65516.12109375 disk.sda1.out‚0' disk.nvme0n1.out‚170732.25634765625 disk.rbd1.in‚ 2.9619140625) disk./.usagePercent‚38.55621770823366$ disk./.usageGB‚574.5155067443848 disk.sda1.in‚0 disk.drbd2165.out‚0 disk.sda2.out‚0# proc.memory.rssMB‚ 2802.58203125 disk.dm-1.out‚ 1616.02734375& disk.rbd0.in‚1.759218604441423e+13 disk.rbd1.out‚ 1740.5546875* proc.memory.percent‚17.105603218078613 disk.sda3.in‚ 108.92578125$ disk.nvme0c0n1.in‚65516.12109375 network.sent‚ 3.271086e+07$ disk.dm-0.out‚170732.25634765625 disk.drbd2165.in‚0 disk.sda2.in‚0 proc.cpu.threads‚17 disk.rbd0.ouã*2K0t‚0) disk.nvme0c0n1.out‚170732.256347656251BC‡”•B‚(:‰ ÀŽêÒŸõåö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚253.982' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚50.7964 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Uè0Y–B‚(: ÀŽêÒ†ÄÈù disk.sda3.out‚ 1616.1796875 disk.rbd1.in‚ 2.9619140625# proc.memory.rssMB‚ 2802.58203125 disk.dm-1.in‚ 108.92578125$ disk.nvme0c0n1.in‚65516.32421875 disk.sda2.in‚0# memory_percent‚81.8030834197998 disk.sda3.in‚ 108.92578125* proc.memory.percent‚17.105603218078613 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.sda2.out‚0 disk.sda1.in‚0 network.sent‚ 3.2716433e+07$ disk./.usageGB‚574.5221061706543 disk.sda.out‚ 1616.1796875( proc.memory.availableMB‚ 2981.3828125 disk.rbd0.out‚0" disk.nvme0n1.in‚65516.32421875 cpu‚25.586038142743753) disk./.usagePercent‚38.55666060126789 disk.dm-0.in‚65516.32421875& disk.nvme0n1.out‚170743.3173828125 disk.rbd1.out‚ 1740.5546875( disk.nvme0c0n1.out‚170743.3173828125 disk.sda1.out‚0 disk.dm-1.out‚ 1616.1796875& disk.rbd0.in‚1.759218604441423e+13# disk.dm-0.out‚170743.3173828125 proc.cpu.threads‚17 network.recv‚ 6.951129e+06 disk.sda.in‚ 108.92578125R%䌓—B‚(:ˆ ÏŽêÒáöö gpu.0.gpu‚100.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚223.56' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚44.712 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0(Ðr#˜B‚(:˜ ÏŽêÒƒØÑù) proc.memory.availableMB‚ 2981.37890625# proc.memory.rssMB‚ 2802.58203125* proc.memory.percent‚17.105603218078613 network.sent‚ 3.2721463e+07$ disk./.usageGB‚574.5288467407227 disk.sda.in‚ 108.92578125 disk.rbd0.out‚0$ disk.dm-0.out‚170756.78955078125 network.recv‚ 6.954019e+06 disk.rbd1.out‚ 1740.6171875 disk.sda3.in‚ 108.92578125 disk.sda3.out‚ 1616.51953125 proc.cpu.threads‚17* disk./.usagePercent‚38.557112966580895 disk.sda2.in‚0 disk.sda.out‚ 1616.51953125 cpu‚25.585938278970087 disk.drbd2165.out‚0" disk.nvme0n1.in‚65516.74609375& disk.rbd0.in‚1.759218604441423e+13 disk.dm-0.in‚65516.74609375$ disk.nvme0c0n1.in‚65516.74609375 disk.rbd1.in‚ 2.9619140625 disk.dm-1.in‚ 108.92578125 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd2165.in‚0$ memory_percent‚81.80310726165771 disk.dm-1.out‚ 1616.51953125) disk.nvme0c0n1.out‚170756.78955078125' disk.nvme0n1.out‚170756.78955078125 disk.sda2.out‚0£ß%{”™B‚(:‰ ÞŽêÒƒ†÷ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚275.279' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.0558 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ð·išB‚(:‰ ÞŽêÒÚí°ù disk.dm-1.in‚ 108.92578125" disk.nvme0c0n1.in‚ 65615.234375) disk.nvme0c0n1.out‚170766.75244140625' disk.nvme0n1.out‚170766.75244140625 disk.sda1.out‚0) disk./.usagePercent‚38.55744680040555& disk.rbd0.in‚1.759218604441423e+13 disk.sda3.out‚ 1616.8984375 disk.sda2.out‚0 disk.sda.out‚ 1616.8984375 disk.rbd0.out‚0 disk.dm-0.in‚ 65615.234375 disk.sda3.in‚ 108.92578125 disk.nvme0n1.in‚ 65615.234375 disk.rbd1.out‚ 1740.6171875 network.recv‚ 6.957861e+06$ disk.dm-0.out‚170766.75244140625 disk.drbd2165.in‚0 disk.rbd1.in‚ 2.9619140625$ memory_percent‚81.80179595947266# proc.memory.rssMB‚ 2802.58203125* proc.memory.percent‚17.105603218078613 proc.cpu.threads‚17 disk.sda1.in‚0 network.sent‚ 3.2726529e+07 disk.sda.in‚ 108.92578125 disk.dm-1.out‚ 1616.8984375 disk.sda2.in‚0 cpu‚25.58581237012511# disk./.usageGB‚574.533821105957 disk.drbd2165.out‚0& proc.memory.availableMB‚ 2981.59375çì1Ê4›B/ _wandbruntime‚27989  _runtime‚27989:€‰«ÊœBÄ trainer/global_step‚16899 % _timestamp‚1.7843178050227275e+09 _runtime‚27989.239966286 _step‚1390 & lr-AdamW/pg1‚4.185838517726694e-05 & lr-AdamW/pg2‚4.185838517726694e-05öâÏBÉ % _timestamp‚1.7843178050227275e+09 _runtime‚27989.239966286 _step‚1390 & lr-AdamW/pg1‚4.185838517726694e-05 & lr-AdamW/pg2‚4.185838517726694e-05 trainer/global_step‚16899î ºÝž4žB/ _wandbruntime‚27989  _runtime‚27989̆T(ŸB”P F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 290607777 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚4859075035226112 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚4859075061399552 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚4874975876182528 trainer/global_step‚16899 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚4859075035226112 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15899469393408 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚20508533643264 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 144230023 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 144230017 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚4872850347306604 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚47865216 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚4874975876182528 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 146377760 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15900814782976 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚15899469393408 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚43355455 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1311733123608576 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚4872848977278314 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚43355460 % _timestamp‚1.7843178050238447e+09 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚47865463 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 144230023 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 146377760 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15873607130732 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 146387733 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚4856976714747904 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15872262530410 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1332241289160704 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 146387733 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 144230017 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚4874974504619520 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 _runtime‚27989.241216726 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175»"wÝ( B™P _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1332241691788288 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚47865216 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚4874975876182528 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 146377760 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15900814782976 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚15899469393408 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚43355455 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1311733123608576 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚4872848977278314 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚43355460 % _timestamp‚1.7843178050238447e+09 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚47865463 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 144230023 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 146377760 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15873607130732 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 146387733 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚4856976714747904 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15872262530410 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1332241289160704 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 146387733 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 144230017 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚4874974504619520 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 _runtime‚27989.241216726 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 290607777 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚4859075035226112 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚4859075061399552 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚4874975876182528 trainer/global_step‚16899 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚4859075035226112 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15899469393408 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚20508533643264 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 144230023 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 144230017 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚4872850347306604 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 B,A¤BœN D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 _step‚1392 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚47867948 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 146386238 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚4859353035956224 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 trainer/global_step‚16899 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚4875254801955328 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 146396225 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚4857254596526080 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚4859353062129664 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚4859353035956224 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 144238229 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚43358003 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚91225946 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1332320372060160 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 _runtime‚27990.62428192 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 290624461 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚246 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1311810241925120 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚4857254621954048 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 290634454 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1311810610016256 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚4873129152551138 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚43357998 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 290624461 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 146396225 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 290634454 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15901739825664Ñ1¢''¥B¡N _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚4857254621954048 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 290634454 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1311810610016256 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚4873129152551138 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚43357998 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 290624461 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 146396225 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 290634454 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15901739825664 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 _step‚1392 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚47867948 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 146386238 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚4859353035956224 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 trainer/global_step‚16899 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚4875254801955328 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 146396225 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚4857254596526080 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚4859353062129664 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚4859353035956224 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 144238229 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚43358003 QDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚91225946 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1332320372060160 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 _runtime‚27990.62428192 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 290624461 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚246 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1311810241925120 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987ð §;Š4¦B/ _wandbruntime‚27990  _runtime‚27990û.› §B 0loss/FRt1/assignment_loss‚0.8350084493946097 3loss/SRqqt1/assignment_loss‚0.03039516255422168 0loss/FBt1/detection_loss‚0.06598430234224001 0loss/FBt2/detection_loss‚0.13803528327532397 trainer/global_step‚16899 3loss/SRqqt2/assignment_loss‚0.07776446668868907 3loss/FBt1/assignment_loss‚0.0028339625439417976 .loss/FRt2/detection_loss‚1.119871794292308 1loss/SRqqt1/detection_loss‚0.3573937240584632 2loss/SRqqt2/detection_loss‚0.37432010433615226 %loss/total_loss‚3.469849765566911 _runtime‚27990.709838278 _step‚1393 1loss/FRt2/assignment_loss‚0.04873265131020252 4loss/FBt2/assignment_loss‚5.6925699680134666e-05 0loss/FRt1/detection_loss‚0.41945293907107906 epoch‚42 % _timestamp‚1.7843178064928932e+09^°¡¨B• epoch‚42 % _timestamp‚1.7843178064928932e+09 0loss/FRt1/assignment_loss‚0.8350084493946097 3loss/SRqqt1/assignment_loss‚0.03039516255422168 0loss/FBt1/detection_loss‚0.06598430234224001 0loss/FBt2/detection_loss‚0.13803528327532397 trainer/global_step‚16899 3loss/SRqqt2/assignment_loss‚0.07776446668868907 3loss/FBt1/assignment_loss‚0.0028339625439417976 .loss/FRt2/detection_loss‚1.119871794292308 1loss/SRqqt1/detection_loss‚0.3573937240584632 2loss/SRqqt2/detection_loss‚0.37432010433615226 %loss/total_loss‚3.469849765566911 _runtime‚27990.709838278 _step‚1393 1loss/FRt2/assignment_loss‚0.04873265131020252 4loss/FBt2/assignment_loss‚5.6925699680134666e-05 0loss/FRt1/detection_loss‚0.41945293907107906ñ )ÍД©B‚(:‰ üŽêÒ´ÿö gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚286.068' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚57.2136 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0®` çªB‚(:† üŽêÒ¾øÐù disk.drbd2165.out‚0 disk.sda.out‚ 1618.2734375 disk.sda1.out‚0 disk.sda3.out‚ 1618.2734375 disk.rbd1.in‚ 2.9619140625 disk.sda.in‚ 108.92578125 disk.dm-1.out‚ 1618.2734375) disk./.usagePercent‚38.55848414293429 disk.sda2.in‚0 disk.rbd1.out‚ 1740.6796875& disk.rbd0.in‚1.759218604441423e+13* proc.memory.percent‚17.105603218078613 network.sent‚ 3.27843e+07$ disk./.usageGB‚574.5492782592773& disk.nvme0n1.out‚170793.8291015625 disk.dm-0.in‚ 65615.859375' proc.memory.availableMB‚ 2981.421875 disk.drbd2165.in‚0" disk.nvme0c0n1.in‚ 65615.859375( disk.nvme0c0n1.out‚170793.8291015625 disk.rbd0.out‚0# proc.memory.rssMB‚ 2802.58203125 proc.cpu.threads‚17 disk.sda1.in‚0# memory_percent‚81.8028450012207 disk.nvme0n1.in‚ 65615.859375 disk.sda2.out‚0 disk.dm-1.in‚ 108.92578125# disk.dm-0.out‚170793.8291015625 cpu‚25.585621952703136 network.recv‚ 6.966468e+06 disk.sda3.in‚ 108.92578125rS”«B‚(:‰ ‹êÒ†…÷ gpu.0.gpu‚59.0 gpu.0.memory‚19, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚336.034' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚67.2068 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¹ó#w¬B‚(: ‹ê҃ߋú$ memory_percent‚81.80279731750488( proc.memory.availableMB‚ 2981.4296875) disk./.usagePercent‚38.55895570881236 disk.dm-0.in‚ 65616.078125 cpu‚25.585655480274188 proc.cpu.threads‚17 disk.sda3.out‚ 1619.6796875 disk.nvme0n1.in‚ 65616.078125' disk.nvme0n1.out‚170802.79248046875 disk.dm-1.in‚ 108.92578125 disk.sda1.out‚0 network.sent‚ 3.2836916e+07" disk.nvme0c0n1.in‚ 65616.078125 disk.sda2.out‚0 disk.rbd0.out‚0 network.recv‚ 6.971489e+06 disk.sda3.in‚ 108.92578125 disk.drbd2165.out‚0 disk.rbd1.in‚ 2.9619140625 disk.rbd1.out‚ 1740.7421875 disk.dm-1.out‚ 1619.6796875$ disk./.usageGB‚574.5563049316406) disk.nvme0c0n1.out‚170802.79248046875 disk.sda2.in‚0 disk.sda.out‚ 1619.6796875$ disk.dm-0.out‚170802.79248046875 disk.sda1.in‚0* proc.memory.percent‚17.105603218078613 disk.drbd2165.in‚0 disk.sda.in‚ 108.92578125& disk.rbd0.in‚1.759218604441423e+13# proc.memory.rssMB‚ 2802.58203125‚8ì”­B‚(:‰ šêÒŒ¢ëö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚263.713' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.7426 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¨¨5ª®B‚(:ø šêÒÞýçø disk.rbd1.out‚ 1740.7421875# memory_percent‚81.8047046661377( proc.memory.availableMB‚ 2981.1171875* disk./.usagePercent‚38.559398089831525 disk.rbd0.out‚0* proc.memory.percent‚17.105603218078613 network.sent‚ 3.2841931e+07 disk.dm-0.in‚65698.75 disk.sda3.out‚ 1620.140625 disk.dm-1.out‚ 1620.140625 disk.drbd2165.in‚0 disk.drbd2165.out‚0$ disk./.usageGB‚574.5628967285156" disk.dm-0.out‚170819.482421875' disk.nvme0c0n1.out‚170819.482421875 disk.sda2.in‚0 disk.rbd1.in‚ 2.9619140625 disk.sda.out‚ 1620.140625 proc.cpu.threads‚17 network.recv‚ 6.975039e+06 disk.nvme0n1.in‚65698.75& disk.rbd0.in‚1.759218604441423e+13# proc.memory.rssMB‚ 2802.58203125 disk.sda3.in‚ 108.92578125 disk.sda.in‚ 108.92578125 cpu‚25.585621046981895 disk.sda1.in‚0 disk.sda1.out‚0 disk.nvme0c0n1.in‚65698.75% disk.nvme0n1.out‚170819.482421875 disk.dm-1.in‚ 108.92578125 disk.sda2.out‚05vž¯B‚(:“ ©êÒÊúíö gpu.0.gpu‚36.0 gpu.0.memory‚12, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚329.609' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚65.92179999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¾Ç•z°B‚(: ©êÒ¥­íø disk.sda.in‚ 108.92578125 disk.sda3.in‚ 108.92578125 disk.sda3.out‚ 1620.71875 disk.rbd1.out‚ 1740.8671875& disk.rbd0.in‚1.759218604441423e+13 disk.rbd0.out‚0 disk.sda1.out‚0$ disk.nvme0c0n1.in‚65703.91015625" disk.nvme0n1.in‚65703.91015625 cpu‚25.58561083882136 network.recv‚ 6.978673e+06) disk./.usagePercent‚38.55982946252672 disk.drbd2165.out‚0) disk.nvme0c0n1.out‚170830.75439453125 disk.sda2.out‚0) proc.memory.availableMB‚ 2981.10546875* proc.memory.percent‚17.105603218078613 proc.cpu.threads‚17$ disk./.usageGB‚574.5693244934082 disk.sda.out‚ 1620.71875 disk.dm-0.in‚65703.91015625 disk.sda1.in‚0# proc.memory.rssMB‚ 2802.58203125 network.sent‚ 3.2846994e+07 disk.dm-1.in‚ 108.92578125 disk.drbd2165.in‚0 disk.sda2.in‚0$ memory_percent‚81.80477619171143$ disk.dm-0.out‚170830.75439453125 disk.rbd1.in‚ 2.9619140625 disk.dm-1.out‚ 1620.71875' disk.nvme0n1.out‚170830.75439453125ÉCÓB”±B‚(:‰ ¸êÒ÷øö gpu.0.gpu‚90.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚281.671' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚56.3342 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ûª„²B‚(:” ¸êÒ©»Òù disk.dm-1.out‚ 1624.1328125 disk.sda3.out‚ 1624.1328125# proc.memory.rssMB‚ 2802.58203125 cpu‚25.585590312684417 proc.cpu.threads‚17 disk.drbd2165.in‚0) proc.memory.availableMB‚ 2981.09765625 network.recv‚ 6.982505e+06$ disk.nvme0c0n1.in‚65704.33203125 disk.rbd1.out‚ 1740.8671875 disk.rbd0.out‚0 network.sent‚ 3.2852005e+07$ disk./.usageGB‚574.5779266357422 disk.rbd1.in‚ 2.9619140625& disk.rbd0.in‚1.759218604441423e+13* proc.memory.percent‚17.105603218078613" disk.nvme0n1.in‚65704.33203125' disk.nvme0n1.out‚170847.25341796875 disk.sda2.in‚0 disk.sda3.in‚ 108.92578125) disk./.usagePercent‚38.56040675951643 disk.dm-0.in‚65704.33203125$ disk.dm-0.out‚170847.25341796875 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd2165.out‚0$ memory_percent‚81.80482387542725 disk.sda.out‚ 1624.1328125) disk.nvme0c0n1.out‚170847.25341796875 disk.sda.in‚ 108.92578125 disk.dm-1.in‚ 108.92578125 disk.sda2.out‚0(”4³B/ _wandbruntime‚28068  _runtime‚28068>¤n½Ç´BÁ % lr-AdamW/pg1‚4.05462342953164e-05 % lr-AdamW/pg2‚4.05462342953164e-05 trainer/global_step‚16949 % _timestamp‚1.7843178846887298e+09 _runtime‚28068.90596264 _step‚1394Gf×̵BÆ % _timestamp‚1.7843178846887298e+09 _runtime‚28068.90596264 _step‚1394 % lr-AdamW/pg1‚4.05462342953164e-05 % lr-AdamW/pg2‚4.05462342953164e-05 trainer/global_step‚16949ò u6=À4¶B/ _wandbruntime‚28068  _runtime‚28068¼ë8»"·B”P WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚4872975097905152 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚48002726 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 144640323 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚43482605 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1336175282129920 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚91485596 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36633600 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 146813485 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚4888922228618752 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚4886790673328512 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 146803512 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15945785324032 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚4888920857055744 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚91485331 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚48002986 trainer/global_step‚16949 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚4872975071731712 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚265 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚20568206228480 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15947130713600 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 144640317 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚43482610 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1315607075901440 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚15945785324032 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚4870870803656704 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 _step‚1395 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 146803512 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404724736 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15947130713600 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚4872975097905152 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 144640323 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 291453808 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 146813485 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚260 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1336175686854656 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 % _timestamp‚1.7843178846899397e+09 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 291443829 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 144640317 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 A8DeviceStatsMonitor.on_tra i_in_batch_start/segment.all.peak‚1995 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 _runtime‚28068.907528296 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚4872975071731712 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚4870870829084672 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚4888920857055744 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1315607443992576 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚20568242862080.:!Ì(¸B™P F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 291443829 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 144640317 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 _runtime‚28068.907528296 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚4872975071731712 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚4870870829084672 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚4888920857055744 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1315607443992576 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚20568242862080 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚4872975097905152 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚48002726 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 144640323 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚43482605 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1336175282129920 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚91485596 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36633600 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 146813485 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚4888922228618752 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚4886790673328512 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 146803512 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15945785324032 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚4888920857055744 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚91485331 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚48002986 trainer/global_step‚16949 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚4872975071731712 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚265 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚20568206228480 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15947130713600 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 144640317 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚43482610 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1315607075901440 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚15945785324032 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚4870870803656704 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 _step‚1395 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 146803512 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404724736 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15947130713600 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚4872975097905152 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 144640323 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 291453808 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 146813485 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚260 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1336175686854656 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 % _timestamp‚1.7843178846899397e+09 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0ó W±N4¹B/ _wandbruntime‚28070  _runtime‚28070«,ÁO#'ºBN YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚20569437396992 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚91490606 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15920767710198 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚254 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚4873253098635264 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚4871148685434880 _step‚1396 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 291460513 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 144648523 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚4889199782821376 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚4873253072461824 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1336254367797248 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 146821977 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 146811990 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚4887068108544644 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚4887069478573046 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 291470506 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚28070.293744873 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 144648523 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1315684930400256 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15919423109764 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 146811990 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15948055756288 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚4889201154391552 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1336253965176832 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚4871148710862848 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 291460513 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 291470506 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1315684562309120 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚4873253072461824 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚91490860 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚48005707º éã('»B¢N YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15946710359552 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 144648529 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚43485153 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚249 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15919423109764 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 146811990 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15948055756288 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚4889201154391552 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1336253965176832 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚4871148710862848 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 291460513 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 291470506 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1315684562309120 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚4873253072461824 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚91490860 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚48005707 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚20569437396992 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚91490606 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15920767710198 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚254 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚4873253098635264 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚4871148685434880 _step‚1396 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 291460513 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 144648523 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚4889199782821376 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚4873253072461824 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1336254367797248 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 146821977 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 146811990 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚4887068108544644 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚4887069478573046 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 291470506 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚28070.293744873 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 144648523 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1315684930400256 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 QDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚4900729629322130 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚4886875108237312 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚4902867209491968 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 _step‚1399 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15992101254656 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 147229264 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚20627751930880 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 145050617 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 147239237 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 147229264 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚4886875108237312 trainer/global_step‚16999 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 292279881 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚4886875134410752 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚91749692 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 145050623 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚266 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚48139937 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15964736756626 $ _timestamp‚1.784317964393791e+09 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6ÒÛ«¬(ÎB˜P XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚48140203 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚4902868581054976 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1340109111582720 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1319481764376576 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚4900730999350420 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚4884764892565504 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚43609755 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚4886875134410752 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15993446644224 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 147239237 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚271 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚4902867209491968 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36633600 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚4900729629322130 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚4886875108237312 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚4902867209491968 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 _step‚1399 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15992101254656 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 147229264 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚20627751930880 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 145050617 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 147239237 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 147229264 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚4886875108237312 trainer/global_step‚16999 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 292279881 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚4886875134410752 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚91749692 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 145050623 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚266 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚48139937 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15964736756626 $ _timestamp‚1.784317964393791e+09 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 145058829 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 145058823 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚4901008434566552 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1340187785906176 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 292306558 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1319559250784256 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚91754974 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36626432 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1340188190623744 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 % _timestamp‚1.7843179657787542e+09 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚4887153108967424 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚15993026290176 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 292296565 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 145058823 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚4887153108967424 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 147237742 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15993026290176 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 147237742 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚20628939839488 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968ÕÑ_)'ÑB£N RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15993026290176 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 147237742 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚20628939839488 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚4885042799771648 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 145058829 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 145058823 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚4901008434566552 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1340187785906176 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 292306558 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1319559250784256 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚91754974 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36626432 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1340188190623744 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 % _timestamp‚1.7843179657787542e+09 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚4887153108967424 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚15993026290176 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 292296565 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 145058823 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚4887153108967424 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 147237742 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0ø …Ñ4ÒB/ _wandbruntime‚28150  _runtime‚28150n\C‚ÓB &loss/total_loss‚3.4071139611212873 epoch‚42 % _timestamp‚1.7843179658653328e+09 _runtime‚28150.082254644 0loss/FRt1/assignment_loss‚0.8017060748011481 0loss/FRt1/detection_loss‚0.39998395203709464 trainer/global_step‚16999 /loss/FRt2/detection_loss‚1.0480144988470355 1loss/FRt2/assignment_loss‚0.09384513157364292 3loss/SRqqt2/assignment_loss‚0.10235857186312912 2loss/FBt1/assignment_loss‚7.94727851846288e-08 3loss/FBt2/assignment_loss‚8.118812297448364e-06 2loss/SRqqt1/detection_loss‚0.34343205535551563 1loss/SRqqt2/detection_loss‚0.4516581900181011 0loss/FBt2/detection_loss‚0.03986942739533456 _step‚1401 3loss/SRqqt1/assignment_loss‚0.04980742849620518 0loss/FBt1/detection_loss‚0.07643043244899808µŒ>`ÔB• 3loss/SRqqt2/assignment_loss‚0.10235857186312912 2loss/FBt1/assignment_loss‚7.94727851846288e-08 3loss/FBt2/assignment_loss‚8.118812297448364e-06 2loss/SRqqt1/detection_loss‚0.34343205535551563 1loss/SRqqt2/detection_loss‚0.4516581900181011 0loss/FBt2/detection_loss‚0.03986942739533456 _step‚1401 3loss/SRqqt1/assignment_loss‚0.04980742849620518 0loss/FBt1/detection_loss‚0.07643043244899808 &loss/total_loss‚3.4071139611212873 epoch‚42 % _timestamp‚1.7843179658653328e+09 _runtime‚28150.082254644 0loss/FRt1/assignment_loss‚0.8017060748011481 0loss/FRt1/detection_loss‚0.39998395203709464 trainer/global_step‚16999 /loss/FRt2/detection_loss‚1.0480144988470355 1loss/FRt2/assignment_loss‚0.09384513157364292ù Ù‚ŸÕB‚(:” ’êÒÏÏÇö gpu.0.gpu‚84.0 gpu.0.memory‚25, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚125.935' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚25.186999999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0I8u7ÖB‚(:† ’êÒÝÒÌù$ disk.dm-0.out‚170953.84326171875 network.recv‚ 7.004954e+06 disk.dm-1.in‚ 108.92578125& disk.rbd0.in‚1.759218604441423e+13 disk.rbd0.out‚0 disk.sda1.out‚0! disk.nvme0c0n1.in‚ 65874.53125) disk.nvme0c0n1.out‚170953.84326171875 network.sent‚ 3.2940708e+07 disk.rbd1.in‚ 2.9619140625 disk.sda2.out‚0& proc.memory.availableMB‚ 2980.90625# proc.memory.rssMB‚ 2802.58203125 cpu‚25.585476724421337 disk.rbd1.out‚ 1741.0546875 disk.sda.out‚ 1634.453125 disk.sda3.in‚ 108.92578125 disk.drbd2165.in‚0 disk.drbd2165.out‚0$ memory_percent‚81.80599212646484* disk./.usagePercent‚38.565396346358774$ disk./.usageGB‚574.6522750854492 disk.sda1.in‚0 disk.sda3.out‚ 1634.453125 proc.cpu.threads‚17 disk.nvme0n1.in‚ 65874.53125* proc.memory.percent‚17.105603218078613 disk.sda.in‚ 108.92578125 disk.dm-1.out‚ 1634.453125 disk.dm-0.in‚ 65874.53125' disk.nvme0n1.out‚170953.84326171875 disk.sda2.in‚0üVW’×B‚(:‡ ¡êÒˆê¿÷ gpu.0.gpu‚91.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚279.58' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.916 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0®ëmØB‚(:ü ¡êÒãÙÿù network.sent‚ 3.3003121e+07 network.recv‚ 7.009442e+06* proc.memory.percent‚17.105603218078613 cpu‚25.58542162627591$ disk./.usageGB‚574.6296119689941& disk.rbd0.in‚1.759218604441423e+13 disk.sda3.in‚ 108.92578125 disk.drbd2165.out‚0 disk.nvme0n1.in‚65874.75 disk.sda.in‚ 108.92578125 disk.sda1.out‚0 disk.nvme0c0n1.in‚65874.75' disk.nvme0n1.out‚170973.79736328125& proc.memory.availableMB‚ 2981.28125 disk.sda3.out‚ 1637.890625) disk.nvme0c0n1.out‚170973.79736328125 disk.sda2.out‚0 disk.rbd1.out‚ 1741.1171875 disk.dm-1.in‚ 108.92578125 disk.drbd2165.in‚0# proc.memory.rssMB‚ 2802.58203125 proc.cpu.threads‚17* disk./.usagePercent‚38.563875405597855 disk.rbd1.in‚ 2.9619140625 disk.sda.out‚ 1637.890625 disk.dm-1.out‚ 1637.890625 disk.rbd0.out‚0$ memory_percent‚81.80370330810547 disk.dm-0.in‚65874.75$ disk.dm-0.out‚170973.79736328125 disk.sda1.in‚0 disk.sda2.in‚0çz™ŸÙB‚(:” °êÒõ¤÷ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚300.629' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚60.125800000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‹™H–ÚB‚(:ˆ °êÒ–Ãúø disk.rbd1.out‚ 1741.1171875" disk.dm-0.out‚171030.423828125 disk.sda1.in‚0" disk.nvme0n1.in‚65874.96484375% disk.nvme0n1.out‚171030.423828125 disk.sda3.in‚ 108.92578125& disk.rbd0.in‚1.759218604441423e+13 cpu‚25.585360456164 network.recv‚ 7.013152e+06 disk.drbd2165.out‚0 disk.rbd0.out‚0$ memory_percent‚81.80246353149414 network.sent‚ 3.3008235e+07$ disk./.usageGB‚574.7144660949707$ disk.nvme0c0n1.in‚65874.96484375 disk.sda2.out‚0 disk.dm-1.out‚ 1638.6484375' disk.nvme0c0n1.out‚171030.423828125 disk.sda.out‚ 1638.6484375 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.dm-1.in‚ 108.92578125 disk.dm-0.in‚65874.96484375 disk.sda3.out‚ 1638.6484375* proc.memory.percent‚17.105603218078613( disk./.usagePercent‚38.5695700371895 disk.sda.in‚ 108.92578125# proc.memory.rssMB‚ 2802.58203125 proc.cpu.threads‚17' proc.memory.availableMB‚ 2981.484375 disk.sda2.in‚0 disk.rbd1.in‚ 2.9619140625À‹æ”ÛB‚(:‰ ¿êÒ÷åö gpu.0.gpu‚100.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚342.225' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚68.445 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0AN£¤ ÜB‚(: ¿êÒ¸¹Òù disk.sda3.in‚ 108.92578125# disk.nvme0n1.out‚171044.5234375$ memory_percent‚81.80255889892578& proc.memory.availableMB‚ 2981.46875 cpu‚25.585349206867374 disk.sda.out‚ 1640.58203125& disk.rbd0.in‚1.759218604441423e+13" disk.nvme0c0n1.in‚ 65875.515625 disk.rbd1.in‚ 2.9619140625 network.recv‚ 7.016405e+06% disk.nvme0c0n1.out‚171044.5234375 disk.sda.in‚ 108.92578125 disk.dm-0.in‚ 65875.515625 disk.dm-0.out‚171044.5234375 disk.nvme0n1.in‚ 65875.515625 proc.cpu.threads‚17 disk.dm-1.in‚ 108.92578125 disk.dm-1.out‚ 1640.58203125 disk.rbd1.out‚ 1741.2421875 disk.sda3.out‚ 1640.58203125 disk.drbd2165.out‚0) disk./.usagePercent‚38.56438486059099$ disk./.usageGB‚574.6372032165527 disk.drbd2165.in‚0# proc.memory.rssMB‚ 2802.58203125 disk.sda2.out‚0 network.sent‚ 3.301329e+07 disk.rbd0.out‚0 disk.sda1.in‚0 disk.sda2.in‚0* proc.memory.percent‚17.105603218078613 disk.sda1.out‚0¦“§F”ÝB‚(:‰ ÎêÒ‰¢ëö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚253.989' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚50.7978 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0O†Uý ÞB‚(:• ÎêÒã›øø network.sent‚ 3.3018308e+07$ disk./.usageGB‚574.6410255432129 disk.sda.out‚ 1643.16015625 disk.dm-1.in‚ 108.92578125 disk.sda3.out‚ 1643.16015625) disk.nvme0c0n1.out‚171061.17919921875' disk.nvme0n1.out‚171061.17919921875$ disk.dm-0.out‚171061.17919921875 proc.cpu.threads‚17 disk.sda.in‚ 108.92578125 disk.dm-1.out‚ 1643.16015625$ memory_percent‚81.80108070373535( proc.memory.availableMB‚ 2981.7109375* proc.memory.percent‚17.105603218078613 disk.rbd1.in‚ 2.9619140625# proc.memory.rssMB‚ 2802.58203125 network.recv‚ 7.019952e+06 disk.dm-0.in‚65995.63671875 disk.rbd1.out‚ 1741.2421875& disk.rbd0.in‚1.759218604441423e+13 disk.drbd2165.in‚0 disk.drbd2165.out‚0" disk.nvme0n1.in‚65995.63671875 disk.sda2.out‚0 disk.rbd0.out‚0 disk.sda1.in‚0$ disk.nvme0c0n1.in‚65995.63671875( disk./.usagePercent‚38.5646413801403 disk.sda1.out‚0 disk.sda3.in‚ 108.92578125 disk.sda2.in‚0 cpu‚25.585305220976707¢¦G4ßB/ _wandbruntime‚28228  _runtime‚28228o&nSÌàBÆ _step‚1402 ' lr-AdamW/pg1‚3.7980857121927686e-05 ' lr-AdamW/pg2‚3.7980857121927686e-05 trainer/global_step‚17049 % _timestamp‚1.7843180438427796e+09 _runtime‚28228.061067586‰~ ›ÑáBË ' lr-AdamW/pg1‚3.7980857121927686e-05 ' lr-AdamW/pg2‚3.7980857121927686e-05 trainer/global_step‚17049 % _timestamp‚1.7843180438427796e+09 _runtime‚28228.061067586 _step‚1402ú /£ßÊ4âB/ _wandbruntime‚28228  _runtime‚28228T]ÉfmãB”P JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚4916813561928192 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 147664989 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚4916814933491200 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 145460917 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚4898658981474304 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 145460923 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚4900775144742912 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1323355716669440 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚4914669955344038 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 293125912 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚16012318470056 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 147664989 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1323356084760576 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚4900775170916352 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1344043125190656 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 trainer/global_step‚17049 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16038417185280 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚270 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚48277463 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36633600 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008Hq`(äB™P VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚4914669955344038 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 293125912 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚16012318470056 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 147664989 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1323356084760576 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚4900775170916352 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1344043125190656 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 trainer/global_step‚17049 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16038417185280 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚270 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚48277463 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36633600 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚4916813561928192 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 147664989 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚4916814933491200 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 145460917 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚4898658981474304 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 145460923 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚4900775144742912 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1323355716669440 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175û À)94åB/ _wandbruntime‚28229  _runtime‚28229‹Þ´Q$'æBžN K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 147663494 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚92019301 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚43739448 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚20688190990336 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1323433203077120 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16039342220800 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 293132617 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 145469123 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚48279853 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1344121798785024 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 _runtime‚28229.447236601 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚92019569 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚48280116 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16013241936414 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚20688227616768 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 147673481 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚4898936863252480 % _timestamp‚1.7843180452298503e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 293142610 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚268 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 145469129 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16039342220800 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚4901053171646464 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚4917093859264000 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚4901053145473024 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 293132617 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 _step‚1404 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚4901053171646464 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚4917093859264000 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36626432 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚4914950130616862 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚16040687617536 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚4898936888680448 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 293142610 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 145469129 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404717568 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚4917092487693824 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 147673481 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚4901053171646464 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚4917093859264000 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚4901053145473024 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 293132617 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 _step‚1404 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚4901053171646464 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚4917093859264000 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36626432 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚4914950130616862 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚16040687617536 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚4898936888680448 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 293142610 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 145469129 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404717568 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚4917092487693824 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 147673481 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚92019301 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚43739448 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚20688190990336 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1323433203077120 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16039342220800 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 293132617 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 145469123 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚48279853 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1344121798785024 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 _runtime‚28229.447236601 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚92019569 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚48280116 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16013241936414 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚20688227616768 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 147673481 RBDeviceStatsMonitor.on_train_batch_end/6Ýpreserved_bytes.all.allocated‚ 49071259648 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚4898936863252480 % _timestamp‚1.7843180452298503e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 293142610 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚268 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440ü rLŠ4èB/ _wandbruntime‚28229  _runtime‚28229¶Ó·˜éB 1loss/FRt2/assignment_loss‚0.07870935041729997 /loss/FRt1/detection_loss‚0.4665464437536275 0loss/FBt2/detection_loss‚0.06707691296789672 % _timestamp‚1.7843180453161142e+09 3loss/SRqqt1/assignment_loss‚0.05249089531740483 3loss/FBt1/assignment_loss‚2.607702676016288e-08 /loss/FRt2/detection_loss‚1.1366490494682848 1loss/SRqqt2/detection_loss‚0.4253209908862781 epoch‚42 trainer/global_step‚17049 _step‚1405 &loss/total_loss‚3.9549654632710367 _runtime‚28229.533054251 3loss/SRqqt2/assignment_loss‚0.08823968207048578 4loss/FBt2/assignment_loss‚1.5727465408815056e-07 0loss/SRqqt1/detection_loss‚0.394216231774469 0loss/FBt1/detection_loss‚0.39036585363447784 0loss/FRt1/assignment_loss‚0.85534986962913085bêB” /loss/FRt2/detection_loss‚1.1366490494682848 1loss/SRqqt2/detection_loss‚0.4253209908862781 epoch‚42 trainer/global_step‚17049 _step‚1405 &loss/total_loss‚3.9549654632710367 _runtime‚28229.533054251 3loss/SRqqt2/assignment_loss‚0.08823968207048578 4loss/FBt2/assignment_loss‚1.5727465408815056e-07 0loss/SRqqt1/detection_loss‚0.394216231774469 0loss/FBt1/detection_loss‚0.39036585363447784 0loss/FRt1/assignment_loss‚0.8553498696291308 1loss/FRt2/assignment_loss‚0.07870935041729997 /loss/FRt1/detection_loss‚0.4665464437536275 0loss/FBt2/detection_loss‚0.06707691296789672 % _timestamp‚1.7843180453161142e+09 3loss/SRqqt1/assignment_loss‚0.05249089531740483 3loss/FBt1/assignment_loss‚2.607702676016288e-08ý 9©Ó#’ëB‚(:‡ ÝêÒ¬Úóö gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚41.0 gpu.0.powerWatts‚340.229' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚68.0458 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Dõ!ìB‚(:– ÝêÒŠÙù disk.drbd2165.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda3.out‚ 1644.59765625) disk./.usagePercent‚38.56533823264851 disk.sda2.in‚0( proc.memory.availableMB‚ 2981.4140625# proc.memory.rssMB‚ 2802.58203125 cpu‚25.585276192259204 network.recv‚ 7.023431e+06 disk.drbd2165.out‚0 proc.cpu.threads‚17 disk.dm-0.in‚65996.05859375 disk.rbd1.in‚ 2.9619140625 disk.sda.out‚ 1644.59765625& disk.rbd0.in‚1.759218604441423e+13$ disk.nvme0c0n1.in‚65996.05859375" disk.nvme0n1.in‚65996.05859375 disk.sda3.in‚ 108.92578125$ memory_percent‚81.80289268493652* proc.memory.percent‚17.105603218078613) disk.nvme0c0n1.out‚171075.12158203125 disk.dm-1.out‚ 1644.59765625' disk.nvme0n1.out‚171075.12158203125 disk.sda2.out‚0 disk.sda.in‚ 108.92578125$ disk.dm-0.out‚171075.12158203125 network.sent‚ 3.3023315e+07$ disk./.usageGB‚574.6514091491699 disk.rbd1.out‚ 1851.1796875 disk.dm-1.in‚ 108.92578125 disk.rbd0.out‚0#KÝg”íB‚(:‰ ìêÒ…‡ìö gpu.0.gpu‚90.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚278.726' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.7452 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ìLîB‚(:‘ ìêÒðå’ú disk.sda1.out‚0$ disk./.usageGB‚574.6571426391602 disk.sda1.in‚0 disk.dm-1.in‚ 108.92578125 disk.drbd2165.in‚0* disk./.usagePercent‚38.565723011972466 disk.rbd0.out‚0 disk.dm-0.in‚65996.26171875 disk.sda2.out‚0 network.recv‚ 7.028967e+06$ disk.dm-0.out‚171101.05126953125 disk.sda3.in‚ 108.92578125 disk.drbd2165.out‚0 disk.rbd1.in‚ 2.9619140625' proc.memory.availableMB‚ 2981.640625* proc.memory.percent‚17.105603218078613 cpu‚25.58524781126271 disk.sda.out‚ 1646.3828125& disk.rbd0.in‚1.759218604441423e+13) disk.nvme0c0n1.out‚171101.05126953125" disk.nvme0n1.in‚65996.26171875 disk.sda2.in‚0$ memory_percent‚81.80150985717773# proc.memory.rssMB‚ 2802.58203125 network.sent‚ 3.308664e+07 disk.rbd1.out‚ 1874.1015625 proc.cpu.threads‚17 disk.dm-1.out‚ 1646.3828125 disk.sda3.out‚ 1646.3828125$ disk.nvme0c0n1.in‚65996.26171875 disk.sda.in‚ 108.92578125' disk.nvme0n1.out‚171101.05126953125ÁÍ$ð’ïB‚(:‡ ûêÒ—‹Ýö gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚287.91' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚57.582 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Õø¿!ðB‚(:– ûêÒÃÉàø disk.sda.out‚ 1649.12890625 disk.sda1.in‚0 disk.rbd1.out‚ 1923.8984375$ memory_percent‚81.80267810821533 proc.cpu.threads‚17 disk.dm-1.out‚ 1649.12890625$ disk.nvme0c0n1.in‚65996.48046875 disk.sda1.out‚0' disk.nvme0n1.out‚171129.81884765625 disk.sda.in‚ 108.92578125 disk.sda3.out‚ 1649.12890625 disk.drbd2165.out‚0 disk.sda2.in‚0 disk.sda2.out‚0 network.sent‚ 3.3091672e+07 disk.sda3.in‚ 108.92578125 disk.drbd2165.in‚0) disk.nvme0c0n1.out‚171129.81884765625& disk.rbd0.in‚1.759218604441423e+13 disk.rbd0.out‚0" disk.nvme0n1.in‚65996.48046875) proc.memory.availableMB‚ 2981.44921875# proc.memory.rssMB‚ 2802.58203125 cpu‚25.58519635545941* disk./.usagePercent‚38.566314389376565$ disk./.usageGB‚574.6659545898438 disk.dm-0.in‚65996.48046875 disk.rbd1.in‚ 2.9619140625 network.recv‚ 7.03219e+06 disk.dm-1.in‚ 108.92578125$ disk.dm-0.out‚171129.81884765625* proc.memory.percent‚17.105603218078613aéWŸñB‚(:” Š‘êÒ瑆÷ gpu.0.gpu‚99.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚284.382' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚56.876400000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¶sÓ„òB‚(:ˆ Š‘êÒÕÃÆù& disk.rbd0.in‚1.759218604441423e+13# disk.dm-0.out‚171149.8349609375) disk./.usagePercent‚38.56654018802176# disk./.usageGB‚574.669319152832 disk.sda.out‚ 1650.55078125 disk.sda2.out‚0 disk.sda1.out‚0* proc.memory.percent‚17.105603218078613 network.recv‚ 7.035894e+06( disk.nvme0c0n1.out‚171149.8349609375 disk.drbd2165.in‚0$ memory_percent‚81.80274963378906 disk.nvme0n1.in‚ 66079.171875 disk.dm-1.out‚ 1650.55078125 disk.rbd0.out‚0 disk.dm-0.in‚ 66079.171875 network.sent‚ 3.3096733e+07 disk.sda2.in‚0 disk.dm-1.in‚ 108.92578125% proc.memory.availableMB‚ 2981.4375& disk.nvme0n1.out‚171149.8349609375 disk.sda.in‚ 108.92578125# proc.memory.rssMB‚ 2802.58203125 disk.rbd1.out‚ 2080.6015625 disk.sda3.in‚ 108.92578125 disk.sda3.out‚ 1650.55078125 disk.drbd2165.out‚0 cpu‚25.58516735739025 proc.cpu.threads‚17 disk.sda1.in‚0" disk.nvme0c0n1.in‚ 66079.171875 disk.rbd1.in‚ 2.96191406258>ÜžóB‚(:“ ™‘êÒ•»ï÷ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚276.921' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚55.38419999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ó⌠ôB‚(:ÿ ™‘êÒ¥©òù) disk./.usagePercent‚38.56667894410532 disk.sda2.in‚0 disk.sda2.out‚0% proc.memory.availableMB‚ 2981.1875# disk.dm-0.out‚171157.8544921875 disk.sda1.in‚0" disk./.usageGB‚574.67138671875 disk.dm-1.out‚ 1652.66796875 cpu‚25.585174571481797 disk.sda1.out‚0 disk.sda3.in‚ 108.92578125 disk.nvme0c0n1.in‚ 66079.375* proc.memory.percent‚17.105603218078613 disk.sda.in‚ 108.92578125 disk.sda.out‚ 1652.66796875 disk.rbd0.out‚0 disk.drbd2165.out‚0 disk.nvme0n1.in‚ 66079.375& disk.nvme0n1.out‚171157.8544921875 disk.rbd1.out‚ 2102.0859375# proc.memory.rssMB‚ 2802.58203125 proc.cpu.threads‚17 disk.dm-0.in‚ 66079.375 network.recv‚ 7.039342e+06 disk.dm-1.in‚ 108.92578125( disk.nvme0c0n1.out‚171157.8544921875& disk.rbd0.in‚1.759218604441423e+13 disk.sda3.out‚ 1652.66796875 disk.drbd2165.in‚0 disk.rbd1.in‚ 2.9619140625$ memory_percent‚81.80427551269531 network.sent‚ 3.3101877e+07h+{»”õB‚(:‰ ¨‘êÒëÜž÷ gpu.0.gpu‚81.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚46.0 gpu.0.powerWatts‚327.587' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚65.5174 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚06pÆÛ öB‚(: ¨‘êÒ–½§ú disk.dm-0.in‚ 66079.796875 disk.sda1.in‚0 disk.drbd2165.in‚0 disk.nvme0n1.in‚ 66079.796875* proc.memory.percent‚17.105603218078613 disk.dm-1.in‚ 108.92578125 disk.dm-1.out‚ 1654.3203125 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda.out‚ 1654.3203125 proc.cpu.threads‚17 disk.sda.in‚ 108.92578125! disk.dm-0.out‚171173.13671875$ disk.nvme0n1.out‚171173.13671875 disk.rbd1.in‚ 2.9619140625 network.sent‚ 3.3106463e+07* disk./.usagePercent‚38.567301042413526" disk.nvme0c0n1.in‚ 66079.796875% proc.memory.availableMB‚ 2981.1875$ memory_percent‚81.80427551269531& disk.rbd0.in‚1.759218604441423e+13 disk.sda1.out‚0 disk.drbd2165.out‚0& disk.nvme0c0n1.out‚171173.13671875 network.recv‚ 7.043034e+06 disk.rbd0.out‚0 disk.sda3.in‚ 108.92578125 disk.sda3.out‚ 1654.3203125 disk.rbd1.out‚ 2283.0703125# proc.memory.rssMB‚ 2802.58203125$ disk./.usageGB‚574.6806564331055 cpu‚25.58517283212699×*qÑ4÷B/ _wandbruntime‚28307  _runtime‚28307|ÎÞHÊøBÄ trainer/global_step‚17099 % _timestamp‚1.7843181234747543e+09 _runtime‚28307.695125571 _step‚1406 & lr-AdamW/pg1‚3.672779559987159e-05 & lr-AdamW/pg2‚3.672779559987159e-05›ä_€ÏùBÉ _runtime‚28307.695125571 _step‚1406 & lr-AdamW/pg1‚3.672779559987159e-05 & lr-AdamW/pg2‚3.672779559987159e-05 trainer/global_step‚17099 % _timestamp‚1.7843181234747543e+09þ W˜b4úB/ _wandbruntime‚28307  _runtime‚28307ÏÜÓŠ(ûB”P ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚4914675181248512 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚248 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 148080768 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚4914675207421952 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16084733115904 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚92278680 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _runtime‚28307.695976798 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚4928611651394236 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚4930759914364416 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚16058555583164 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚253 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚4928610281365946 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16086078505472 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚48414873 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 145871223 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚92278933 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16086078505472 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 148090741 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚4930761285927424 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚20746721562624 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 148080768 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568úAÍø(üB™P VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚4914675181248512 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚248 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 148080768 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚4914675207421952 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16084733115904 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚92278680 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _runtime‚28307.695976798 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚4928611651394236 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚4930759914364416 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚16058555583164 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚253 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚4928610281365946 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16086078505472 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚48414873 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 145871223 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚92278933 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16086078505472 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 148090741 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚4930761285927424 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚20746721562624 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 148080768 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097ÿ ÚiHM4ýB/ _wandbruntime‚28309  _runtime‚28309§LE_þBžN Qàßp'dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1327307891552256 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚20747949381632 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚92283912 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1348055438313472 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚4931040211700224 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 148089246 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚241 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚4912830977589248 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 145879429 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚4914953208152064 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚4914953181978624 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 _runtime‚28309.080883539 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚16087003548160 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚4914953208152064 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚17099 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 145879429 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚4931038840130048 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 293968669 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚236 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 293978662 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 293978662 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1348055840933888 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚16087003548160 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16085658151424 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1327307523461120 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 145879423 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 % _timestamp‚1.7843181248635056e+09 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 _step‚1408 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚92284153 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚4912830952161280 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16059479049522 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚43866603 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚48417550 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 148099233 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚43866598 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008Kä°Æ)'ÿB£N 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚4912830977589248 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 145879429 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚4914953208152064 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚4914953181978624 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 _runtime‚28309.080883539 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚16087003548160 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚4914953208152064 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚17099 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 145879429 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚4931038840130048 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 293968669 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚236 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 293978662 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 293978662 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1348055840933888 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚16087003548160 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16085658151424 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1327307523461120 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 145879423 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 % _timestamp‚1.7843181248635056e+09 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 _step‚1408 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚92284153 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚4912830952161280 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16059479049522 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚43866603 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚48417550 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 148099233 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚43866598 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1327307891552256 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚20747949381632 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚92283912 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1348055438313472 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚4931040211700224 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 148089246 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚241€ JÇÐ4€C/ _wandbruntime‚28309  _runtime‚28309~(,¿C‹ 0loss/FRt2/assignment_loss‚0.0971482157406939 4loss/SRqqt1/assignment_loss‚0.056799862708558635 3loss/FBt1/assignment_loss‚3.325292582109054e-07 1loss/SRqqt1/detection_loss‚0.3778109921787937 2loss/SRqqt2/detection_loss‚0.43905486825313056 .loss/FBt1/detection_loss‚0.317692287728541 _step‚1409 2loss/SRqqt2/assignment_loss‚0.1074456416985485 4loss/FBt2/assignment_loss‚1.1275708115266686e-05 /loss/FBt2/detection_loss‚0.1121461798403862 %loss/total_loss‚3.936030470277352 epoch‚42 _runtime‚28309.167259036 trainer/global_step‚17099 0loss/FRt1/assignment_loss‚0.8004809913259966 /loss/FRt1/detection_loss‚0.4236979696128544 /loss/FRt2/detection_loss‚1.2037418529524748 $ _timestamp‚1.784318124950321e+09§#†‚C 0loss/FRt2/assignment_loss‚0.0971482157406939 4loss/SRqqt1/assignment_loss‚0.056799862708558635 3loss/FBt1/assignment_loss‚3.325292582109054e-07 1loss/SRqqt1/detection_loss‚0.3778109921787937 2loss/SRqqt2/detection_loss‚0.43905486825313056 .loss/FBt1/detection_loss‚0.317692287728541 _step‚1409 2loss/SRqqt2/assignment_loss‚0.1074456416985485 4loss/FBt2/assignment_loss‚1.1275708115266686e-05 /loss/FBt2/detection_loss‚0.1121461798403862 %loss/total_loss‚3.936030470277352 epoch‚42 _runtime‚28309.167259036 trainer/global_step‚17099 0loss/FRt1/assignment_loss‚0.8004809913259966 /loss/FRt1/detection_loss‚0.4236979696128544 /loss/FRt2/detection_loss‚1.2037418529524748 $ _timestamp‚1.784318124950321e+09 !ö§žƒC‚(:“ ·‘êÒ‹Žæö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚294.623' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚58.92459999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0&ÁÁa„C‚(:‡ ·‘êÒ­˜²ù) disk./.usagePercent‚38.56394094352662 disk.rbd0.out‚0 disk.sda3.in‚ 108.92578125 disk.drbd2165.out‚0 disk.sda2.out‚0 disk.dm-0.out‚ 171196.609375 disk.drbd2165.in‚0" disk.nvme0n1.in‚66080.05859375* proc.memory.percent‚17.105603218078613$ disk./.usageGB‚574.6305885314941" disk.nvme0n1.out‚ 171196.609375$ memory_percent‚81.80413246154785# proc.memory.rssMB‚ 2802.58203125 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-1.in‚ 108.92578125 disk.sda2.in‚0 proc.cpu.threads‚17 disk.sda.in‚ 108.92578125 disk.dm-1.out‚ 1655.52734375 disk.dm-0.in‚66080.05859375$ disk.nvme0c0n1.in‚66080.05859375 disk.rbd1.in‚ 2.9619140625 cpu‚25.585120641434614 network.sent‚ 3.3112056e+07 network.recv‚ 7.046551e+06 disk.sda.out‚ 1655.52734375& disk.rbd0.in‚1.759218604441423e+13$ disk.nvme0c0n1.out‚ 171196.609375 disk.rbd1.out‚ 2308.3984375( proc.memory.availableMB‚ 2981.2109375 disk.sda3.out‚ 1655.52734375‘Ô-””…C‚(:‰ Æ‘êÒ¸Šéö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚336.329' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚67.2658 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0#µP*†C‚(:„ Æ‘êÒÿ¯½ù disk.sda1.in‚0 disk.rbd1.in‚ 2.9619140625 disk.sda.in‚ 108.92578125 disk.sda.out‚ 1655.7734375 disk.nvme0c0n1.in‚ 66150.125) disk.nvme0c0n1.out‚171209.72119140625' disk.nvme0n1.out‚171209.72119140625# proc.memory.rssMB‚ 2802.58203125 disk.sda2.in‚0 disk.dm-1.out‚ 1655.7734375 disk.nvme0n1.in‚ 66150.125* proc.memory.percent‚17.105603218078613) disk./.usagePercent‚38.56439458887729 disk.rbd0.out‚0 disk.dm-0.in‚ 66150.125 disk.drbd2165.in‚0( proc.memory.availableMB‚ 2981.1953125 network.sent‚ 3.3175521e+07 network.recv‚ 7.051959e+06 disk.sda1.out‚0 disk.drbd2165.out‚0 cpu‚25.585128345734184 disk.rbd1.out‚ 2444.9453125 disk.sda3.in‚ 108.92578125 disk.sda3.out‚ 1655.7734375$ memory_percent‚81.80422782897949 proc.cpu.threads‚17$ disk./.usageGB‚574.6373481750488 disk.dm-1.in‚ 108.92578125& disk.rbd0.in‚1.759218604441423e+13 disk.sda2.out‚0$ disk.dm-0.out‚171209.72119140625#u¯””‡C‚(:‰ Õ‘êÒ€©èö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚287.951' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚57.5902 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0&é‹ÏˆC‚(:‘ Õ‘êÒÒÇÀù# disk.nvme0c0n1.in‚ 66150.6328125 proc.cpu.threads‚17 network.sent‚ 3.3180576e+07) disk./.usagePercent‚38.56486513072523 disk.dm-1.in‚ 108.92578125 disk.sda1.out‚0 disk.sda3.in‚ 108.92578125 disk.sda2.in‚0 disk.rbd1.in‚ 2.9619140625 disk.rbd1.out‚ 2453.1328125# disk./.usageGB‚574.644359588623 disk.sda.out‚ 1656.09765625' disk.nvme0n1.out‚171221.45361328125$ memory_percent‚81.80432319641113# proc.memory.rssMB‚ 2802.58203125& disk.rbd0.in‚1.759218604441423e+13 disk.sda3.out‚ 1656.09765625 disk.sda.in‚ 108.92578125 disk.dm-1.out‚ 1656.09765625( proc.memory.availableMB‚ 2981.1796875 disk.drbd2165.in‚0 disk.drbd2165.out‚0) disk.nvme0c0n1.out‚171221.45361328125 disk.sda2.out‚0* proc.memory.percent‚17.105603218078613 network.recv‚ 7.055287e+06 disk.dm-0.in‚ 66150.6328125$ disk.dm-0.out‚171221.45361328125 disk.sda1.in‚0! disk.nvme0n1.in‚ 66150.6328125 cpu‚25.58513616024932 disk.rbd0.out‚0½n󗞉C‚(:“ ä‘êÒ¹‹÷ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚189.283' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚37.85659999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0cïxŠC‚(: ä‘êÒ ©´ø disk.dm-0.in‚ 66150.8515625$ memory_percent‚81.79984092712402 disk.rbd1.in‚ 2.9619140625 proc.cpu.threads‚17# disk.nvme0c0n1.in‚ 66150.8515625( disk.nvme0c0n1.out‚171238.5517578125! disk.nvme0n1.in‚ 66150.8515625 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda3.in‚ 108.92578125 disk.dm-1.in‚ 108.92578125 disk.rbd0.out‚0# proc.memory.rssMB‚ 2802.58203125 network.recv‚ 7.05885e+06 disk.sda3.out‚ 1656.38671875 disk.sda2.in‚0 disk.dm-1.out‚ 1656.38671875# disk.dm-0.out‚171238.5517578125# disk./.usageGB‚574.648509979248 disk.drbd2165.in‚0 disk.drbd2165.out‚0& disk.rbd0.in‚1.759218604441423e+13) disk./.usagePercent‚38.56514366692249 disk.sda.out‚ 1656.38671875 cpu‚25.58513728349061 network.sent‚ 3.3185585e+07* proc.memory.percent‚17.105603218078613( proc.memory.availableMB‚ 2981.9140625& disk.nvme0n1.out‚171238.5517578125 disk.sda2.out‚0 disk.rbd1.out‚ 2649.3671875 disk.sda.in‚ 108.92578125€X¥›ž‹C‚(:“ ó‘êÒµêí÷ gpu.0.gpu‚90.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚291.384' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚58.27680000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÖH{ì"ŒC‚(:— ó‘êÒÁåù network.recv‚ 7.062111e+06 disk.sda.in‚ 108.92578125 cpu‚25.585080904470015 disk.drbd2165.in‚0 disk.drbd2165.out‚0$ disk.nvme0c0n1.in‚66151.06640625) disk.nvme0c0n1.out‚171245.78564453125 disk.rbd1.out‚ 2651.5078125" disk.nvme0n1.in‚66151.06640625# proc.memory.rssMB‚ 2802.58203125 network.sent‚ 3.3190659e+07) disk./.usagePercent‚38.56560934462727$ disk./.usageGB‚574.6554489135742 disk.dm-1.out‚ 1656.87109375 disk.dm-0.in‚66151.06640625) proc.memory.availableMB‚ 2981.67578125 disk.dm-1.in‚ 108.92578125 disk.sda1.out‚0 proc.cpu.threads‚17 disk.sda.out‚ 1656.87109375& disk.rbd0.in‚1.759218604441423e+13 disk.rbd0.out‚0$ disk.dm-0.out‚171245.78564453125' disk.nvme0n1.out‚171245.78564453125 disk.sda1.in‚0 disk.sda3.out‚ 1656.87109375 disk.rbd1.in‚ 2.9619140625$ memory_percent‚81.80129528045654* proc.memory.percent‚17.105603218078613 disk.sda3.in‚ 108.92578125 disk.sda2.in‚0 disk.sda2.out‚0uøXå4C/ _wandbruntime‚28387  _runtime‚28387‰¸0pÊŽCÄ & lr-AdamW/pg1‚3.549459318675565e-05 & lr-AdamW/pg2‚3.549459318675565e-05 trainer/global_step‚17149 % _timestamp‚1.7843182031307633e+09 _runtime‚28387.348003787 _step‚1410–0 ~ÏCÉ trainer/global_step‚17149 % _timestamp‚1.7843182031307633e+09 _runtime‚28387.348003787 _step‚1410 & lr-AdamW/pg1‚3.549459318675565e-05 & lr-AdamW/pg2‚3.549459318675565e-05‚ KÉC4C/ _wandbruntime‚28387  _runtime‚283879‹”Y ‘C”P QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚251 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 trainer/global_step‚17149 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚4944706266800640 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚4944706266800640 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚48551297 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16132394436096 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚92542502 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚16104792696272 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16131049046528 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 148506520 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 % _timestamp‚1.7843182031317396e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚4928575217754112 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 WJDeviceStatsMonitor.on_train_batch_stagÀÁrt/allocated_bytes.large_pool.current‚26173440 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1331104357437440 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚43991205 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _runtime‚28387.349110747 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚4928575243927552 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚92542758 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚4944707638363648 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1331104725528576 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚20806530594816 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚4942550607387854 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16132394436096 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 146281517 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚43991210 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1351911290659840 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚4926447184719872 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 294798016 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚4926447159291904 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16131049046528 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 148516493 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1351910888032256 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 294788037 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚4944706266800640 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚4944706266800640 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚48551297 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16132394436096 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚92542502 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚16104792696272 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16131049046528 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 148506520 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 % _timestamp‚1.7843182031317396e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚4928575217754112 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1331104357437440 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚43991205 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _runtime‚28387.349110747 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚4928575243927552 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚92542758 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚4944707638363648 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1331104725528576 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚20806530594816 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚4942550607387854 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16132394436096 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 146281517 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚43991210 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1351911290659840 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚4926447184719872 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 294798016 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚4926447159291904 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16131049046528 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 148516493 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1351910888032256 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 294788037 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 JDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 294814714 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚4942830782660678 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 148524985 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 294804721 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 148514998 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚4944986564136448 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚92548002 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16105716162630 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 148524985 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚4928853218484224 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 146289729 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1331182211936256 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚20807716724736 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 294804721 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 146289723 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚92547755 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1331181843845120 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚4926725041070080 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚16133319478784 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 146289723 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚1412 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚43993748 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚48554007 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 146289729 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16104371562196 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚4928853244657664 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0ôc­;¥•C£N WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 trainer/global_step‚17149 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚242 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 _runtime‚28388.733943034 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 294814714 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚4942830782660678 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 148524985 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 294804721 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 148514998 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚4944986564136448 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚92548002 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16105716162630 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 148524985 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚4928853218484224 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 146289729 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1331182211936256 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚20807716724736 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 294804721 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 146289723 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚92547755 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1331181843845120 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚4926725041070080 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚16133319478784 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 146289723 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚1412 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚43993748 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚48554007 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 146289729 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16104371562196 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚4928853244657664 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440„ úvN4–C/ _wandbruntime‚28388  _runtime‚28388¨¦7ì—C 0loss/FRt1/assignment_loss‚0.7760829247702333 0loss/FBt1/detection_loss‚0.13214980848645125 % _timestamp‚1.7843182046036115e+09 _step‚1413 1loss/FRt2/assignment_loss‚0.05643964587868034 3loss/SRqqt1/assignment_loss‚0.04930047061524193 3loss/SRqqt2/assignment_loss‚0.07891907622776362 3loss/FBt2/assignment_loss‚3.364849388498255e-07 /loss/FRt2/detection_loss‚1.1565629971288194 trainer/global_step‚17149 3loss/FBt1/assignment_loss‚8.063144648346949e-06 0loss/FRt1/detection_loss‚0.39521300582746516 1loss/SRqqt1/detection_loss‚0.3688524085803884 1loss/SRqqt2/detection_loss‚0.3723323716578096 %loss/total_loss‚3.501280476984734 0loss/FBt2/detection_loss‚0.11541936818229342 epoch‚42 _runtime‚28388.820507879åÖ↘C” 0loss/FRt1/detection_loss‚0.39521300582746516 1loss/SRqqt1/detection_loss‚0.3688524085803884 1loss/SRqqt2/detection_loss‚0.3723323716578096 %loss/total_loss‚3.501280476984734 0loss/FBt2/detection_loss‚0.11541936818229342 epoch‚42 _runtime‚28388.820507879 0loss/FRt1/assignment_loss‚0.7760829247702333 0loss/FBt1/detection_loss‚0.13214980848645125 % _timestamp‚1.7843182046036115e+09 _step‚1413 1loss/FRt2/assignment_loss‚0.05643964587868034 3loss/SRqqt1/assignment_loss‚0.04930047061524193 3loss/SRqqt2/assignment_loss‚0.07891907622776362 3loss/FBt2/assignment_loss‚3.364849388498255e-07 /loss/FRt2/detection_loss‚1.1565629971288194 trainer/global_step‚17149 3loss/FBt1/assignment_loss‚8.063144648346949e-06… ¸¿ Mž™C‚(:“ ‚’êÒ¬Òìö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚282.255' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚56.45099999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ë°.ÀšC‚(: ‚’êÒÍËàù disk.dm-1.in‚ 108.92578125 disk.rbd0.out‚0 disk.sda3.out‚ 1657.078125$ disk./.usageGB‚574.6620674133301) disk.nvme0c0n1.out‚171262.16943359375 disk.rbd1.out‚ 2782.1796875 disk.sda.out‚ 1657.078125& disk.rbd0.in‚1.759218604441423e+13 disk.sda3.in‚ 108.92578125 network.sent‚ 3.3195791e+07 network.recv‚ 7.065851e+06( proc.memory.availableMB‚ 2982.0546875 proc.cpu.threads‚17$ disk.nvme0c0n1.in‚66242.80078125' disk.nvme0n1.out‚171262.16943359375 disk.rbd1.in‚ 2.9619140625$ disk.dm-0.out‚171262.16943359375$ memory_percent‚81.79898262023926 disk.sda2.out‚0 disk.sda1.out‚0 cpu‚25.58504486660196* disk./.usagePercent‚38.566053517699174" disk.nvme0n1.in‚66242.80078125 disk.sda.in‚ 108.92578125 disk.dm-1.out‚ 1657.078125 disk.dm-0.in‚66242.80078125 disk.sda1.in‚0* proc.memory.percent‚17.105603218078613 disk.sda2.in‚0 disk.drbd2165.in‚0 disk.drbd2165.out‚0# proc.memory.rssMB‚ 2802.58203125GÊ—ž›C‚(:“ ‘’êÒõŠèö gpu.0.gpu‚33.0 gpu.0.memory‚11, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚334.292' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚66.85839999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0†o«¥œC‚(: ‘’êÒÏÞüø! disk.nvme0n1.in‚ 66243.0546875 disk.sda2.out‚0 disk.sda.out‚ 1659.265625 disk.dm-0.in‚ 66243.0546875 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.rbd1.out‚ 2918.5234375 cpu‚25.585001480848074 disk.sda1.in‚0 disk.sda3.in‚ 108.92578125 disk.rbd0.out‚0# disk.dm-0.out‚171285.9091796875* proc.memory.percent‚17.105603218078613 disk.sda3.out‚ 1659.29296875 disk.dm-1.in‚ 108.92578125 disk.sda1.out‚0# disk.nvme0c0n1.in‚ 66243.0546875 disk.sda.in‚ 108.92578125$ memory_percent‚81.79981708526611 network.sent‚ 3.3259087e+07) disk./.usagePercent‚38.56667459197724( disk.nvme0c0n1.out‚171285.8740234375 disk.sda2.in‚0 disk.rbd1.in‚ 2.9619140625 network.recv‚ 7.070431e+06$ disk./.usageGB‚574.6713218688965 disk.dm-1.out‚ 1659.29296875& disk.nvme0n1.out‚171285.8740234375& disk.rbd0.in‚1.759218604441423e+13) proc.memory.availableMB‚ 2981.91796875# proc.memory.rssMB‚ 2802.58203125 proc.cpu.threads‚17‘em}ŸC‚(:”  ’êÒšç‰ø gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚291.919' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚58.383799999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¤O:C!žC‚(:–  ’êÒ¯ê¾ù$ disk./.usageGB‚574.6787490844727 disk.sda1.out‚0 disk.sda3.in‚ 108.92578125" disk.nvme0n1.in‚66243.26953125) proc.memory.availableMB‚ 2982.07421875* proc.memory.percent‚17.105603218078613 network.recv‚ 7.073884e+06 disk.sda.out‚ 1659.87109375& disk.rbd0.in‚1.759218604441423e+13 disk.dm-1.in‚ 108.92578125 disk.rbd0.out‚0 disk.dm-0.in‚66243.26953125 disk.sda1.in‚0 disk.sda2.out‚0 network.sent‚ 3.3264162e+07 disk.sda.in‚ 108.92578125 disk.dm-1.out‚ 1659.87109375# memory_percent‚81.7988634109497# proc.memory.rssMB‚ 2802.58203125 disk.sda3.out‚ 1659.87109375$ disk.nvme0c0n1.in‚66243.26953125) disk.nvme0c0n1.out‚171301.01318359375' disk.nvme0n1.out‚171301.01318359375 disk.sda2.in‚0 disk.drbd2165.out‚0 disk.rbd1.in‚ 2.9619140625 disk.rbd1.out‚ 3029.6015625$ disk.dm-0.out‚171301.01318359375 disk.drbd2165.in‚0 proc.cpu.threads‚17) disk./.usagePercent‚38.56717303864641 cpu‚25.584938120768445¦N®“ŸC‚(:ˆ ¯’êҊ߈÷ gpu.0.gpu‚27.0 gpu.0.memory‚9, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚326.764' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚65.3528 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0t'öE  C‚(:• ¯’êÒ¤ÈÒù disk.dm-0.in‚66243.26953125 disk.sda3.out‚ 1662.82421875$ disk.nvme0c0n1.in‚66243.26953125& disk.nvme0n1.out‚171313.4345703125 network.sent‚ 3.3268696e+07 disk.dm-1.out‚ 1662.82421875 disk.drbd2165.out‚0 disk.sda2.out‚0$ memory_percent‚81.80034160614014 disk.sda1.in‚0( disk.nvme0c0n1.out‚171313.4345703125# proc.memory.rssMB‚ 2802.58203125 network.recv‚ 7.077199e+06$ disk./.usageGB‚574.6738548278809 disk.sda.in‚ 108.92578125 disk.drbd2165.in‚0 disk.sda.out‚ 1662.82421875 disk.sda3.in‚ 108.92578125" disk.nvme0n1.in‚66243.26953125* proc.memory.percent‚17.105603218078613& disk.rbd0.in‚1.759218604441423e+13# disk.dm-0.out‚171313.4345703125 disk.sda1.out‚0 disk.rbd1.in‚ 2.9619140625* disk./.usagePercent‚38.566844580979975 disk.sda2.in‚0) proc.memory.availableMB‚ 2981.83203125 cpu‚25.584945747821802 proc.cpu.threads‚17 disk.dm-1.in‚ 108.92578125 disk.rbd1.out‚ 3172.2578125 disk.rbd0.out‚0਷ðž¡C‚(:“ ¾’êÒ¬÷ö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚294.095' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚58.81900000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0±ð»ü¢C‚(:ñ ¾’êÒ׸Åù network.recv‚ 7.080789e+06 disk.sda.out‚ 1663.21875 disk.drbd2165.out‚0# proc.memory.rssMB‚ 2802.58203125* proc.memory.percent‚17.105603218078613 disk.rbd1.in‚ 2.9619140625 disk.rbd1.out‚ 3293.5546875 disk.sda.in‚ 108.92578125 disk.dm-1.in‚ 108.92578125 disk.sda3.out‚ 1663.21875" disk.nvme0n1.in‚66311.05078125 disk.dm-1.out‚ 1663.21875& disk.rbd0.in‚1.759218604441423e+13 disk.nvme0n1.out‚ 171318.75) disk./.usagePercent‚38.56686429356011 disk.sda3.in‚ 108.92578125 disk.sda2.out‚0# memory_percent‚81.7988634109497 proc.cpu.threads‚17 disk.sda1.in‚0 disk.drbd2165.in‚0 disk.sda2.in‚0) proc.memory.availableMB‚ 2982.07421875 disk.rbd0.out‚0 disk.nvme0c0n1.out‚ 171318.75$ disk./.usageGB‚574.6741485595703 disk.dm-0.in‚66311.05078125 cpu‚25.584909978035803 network.sent‚ 3.327427e+07 disk.dm-0.out‚ 171318.75 disk.sda1.out‚0$ disk.nvme0c0n1.in‚66311.05078125Ú7ý64£C/ _wandbruntime‚28466  _runtime‚28466÷kQ}ɤCà $ _timestamp‚1.784318282539798e+09 _runtime‚28466.757173508 _step‚1414 & lr-AdamW/pg1‚3.428132908886661e-05 & lr-AdamW/pg2‚3.428132908886661e-05 trainer/global_step‚17199* &§Î¥CÈ _runtime‚28466.757173508 _step‚1414 & lr-AdamW/pg1‚3.428132908886661e-05 & lr-AdamW/pg2‚3.428132908886661e-05 trainer/global_step‚17199 $ _timestamp‚1.784318282539798e+09† ‡Úb4¦C/ _wandbruntime‚28466  _runtime‚28466Õ\8ú(§C”P XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚48688782 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚4958652619236864 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36633600 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚273 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 146691817 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 295624089 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚92807142 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1355844792459264 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 146691823 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚4958652619236864 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚4958653990799872 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚4942475280433152 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚4942475254259712 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1355845197184000 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404724736 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚4958653990799872 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 146691817 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚44118355 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚4940341273628672 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚4956490933409762 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚44118360 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1334979045912576 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 148942245 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚4942475254259712 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚48688514 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0¥J%Þö¨C™P aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚4942475280433152 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚20866151271424 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚4958652619236864 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36633600 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚273 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 146691817 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 295624089 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚92807142 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1355844792459264 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 146691823 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚4958652619236864 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚4958653990799872 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚4942475280433152 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚4942475254259712 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1355845197184000 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404724736 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚4958653990799872 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 146691817 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚44118355 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚4940341273628672 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚4956490933409762 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.Rãlê)large_pool.allocated‚44118360 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1334979045912576 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 148942245 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚4942475254259712 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚48688514 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚48688782 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 JDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 146700029 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚20867302111232 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 146700023 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚4958932916572672 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚264 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 295650766 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 295640773 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚4942753281163264 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚259 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 trainer/global_step‚17199 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 295650766 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 148950737 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16178290012672 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚4942753254989824 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚44120903 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16151953275738 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚4956769738654184 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 148940750 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚4958931545002496 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚16179635409408 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 % _timestamp‚1.7843182839301598e+09 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16150608675304 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚4940619129978880 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚20867338737664 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 295640773 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚92812427 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 148950737 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚48691524 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1335056532320256 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚4942753254989824 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 146700029 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1355923871057920 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404717568 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚92812163 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚4956771108682586—èv)'«C£N PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 148950737 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚48691524 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1335056532320256 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚4942753254989824 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 146700029 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1355923871057920 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404717568 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚92812163 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚4956771108682586 _step‚1416 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16178290012672 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36626432 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1355923466340352 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 _runtime‚28468.147524148 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚44120898 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 146700029 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚20867302111232 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 146700023 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚4958932916572672 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚264 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 295650766 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 295640773 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚4942753281163264 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚259 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 trainer/global_step‚17199 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 295650766 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 148950737 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16178290012672 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚4942753254989824 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚44120903 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16151953275738 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚4956769738654184 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 148940750 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚4958931545002496 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚16179635409408 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 % _timestamp‚1.7843182839301598e+09 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16150608675304 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚4940619129978880 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚20867338737664 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 295640773 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚92812427 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175ˆ ŠEãþ4¬C/ _wandbruntime‚28468  _runtime‚28468wÀ§­CŽ .loss/FRt1/assignment_loss‚0.84971082953565 2loss/SRqqt1/assignment_loss‚0.0291193316044278 3loss/SRqqt2/assignment_loss‚0.07793636906424525 3loss/FBt1/assignment_loss‚0.0004225952498855021 /loss/FRt2/detection_loss‚1.0886306347350836 _runtime‚28468.232101939 1loss/FBt2/detection_loss‚0.055812670456065264 &loss/total_loss‚3.5116712039815425 trainer/global_step‚17199 % _timestamp‚1.7843182840151598e+09 _step‚1417 0loss/SRqqt1/detection_loss‚0.378817500289987 2loss/SRqqt2/detection_loss‚0.38585472566612194 0loss/FBt1/detection_loss‚0.21022983191273348 1loss/FRt2/assignment_loss‚0.06600370188963782 3loss/FBt2/assignment_loss‚2.362258736332204e-06 0loss/FRt1/detection_loss‚0.36913065131896866 epoch‚42Ïu“û®C“ .loss/FRt1/assignment_loss‚0.84971082953565 2loss/SRqqt1/assignment_loss‚0.0291193316044278 3loss/SRqqt2/assignment_loss‚0.07793636906424525 3loss/FBt1/assignment_loss‚0.0004225952498855021 /loss/FRt2/detection_loss‚1.0886306347350836 _runtime‚28468.232101939 1loss/FBt2/detection_loss‚0.055812670456065264 &loss/total_loss‚3.5116712039815425 trainer/global_step‚17199 % _timestamp‚1.7843182840151598e+09 _step‚1417 0loss/SRqqt1/detection_loss‚0.378817500289987 2loss/SRqqt2/detection_loss‚0.38585472566612194 0loss/FBt1/detection_loss‚0.21022983191273348 1loss/FRt2/assignment_loss‚0.06600370188963782 3loss/FBt2/assignment_loss‚2.362258736332204e-06 0loss/FRt1/detection_loss‚0.36913065131896866 epoch‚42‰ G‰œ¯C‚(:‘ Í’êÒñòÚø gpu.0.gpu‚1.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚322.848' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚64.56960000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0i\"’ °C‚(:• Í’êÒ½Ýú disk.sda2.in‚0 disk.dm-1.in‚ 108.92578125& disk.rbd0.in‚1.759218604441423e+13 disk.rbd0.out‚0$ disk./.usageGB‚574.6742706298828 disk.drbd2165.out‚0" disk.nvme0n1.in‚66318.51171875$ disk.dm-0.out‚171322.04541015625 disk.sda.out‚ 1663.4921875 cpu‚25.584987020801776 disk.dm-1.out‚ 1663.4921875$ memory_percent‚81.80081844329834 disk.sda2.out‚0 disk.rbd1.in‚ 2.9619140625 disk.rbd1.out‚ 3429.9609375) proc.memory.availableMB‚ 2981.75390625 network.recv‚ 7.083472e+06* disk./.usagePercent‚38.566872485801206$ disk.nvme0c0n1.in‚66318.51171875) disk.nvme0c0n1.out‚171322.04541015625' disk.nvme0n1.out‚171322.04541015625* proc.memory.percent‚17.105603218078613 network.sent‚ 3.3279282e+07 disk.dm-0.in‚66318.51171875 disk.sda1.in‚0 disk.sda.in‚ 108.92578125 proc.cpu.threads‚17 disk.sda3.in‚ 108.92578125# proc.memory.rssMB‚ 2802.58203125 disk.sda1.out‚0 disk.sda3.out‚ 1663.4921875 disk.drbd2165.in‚0o&Á•±C‚(:Š Ü’êÒݯéö gpu.0.gpu‚100.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚340.338' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚68.0676 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0î,µ¡²C‚(: Ü’êÒŒ£ù proc.cpu.threads‚17) disk.nvme0c0n1.out‚171327.08447265625 disk.rbd1.in‚ 2.9619140625 disk.sda1.out‚0 disk.drbd2165.in‚0$ memory_percent‚81.80115222930908$ disk.nvme0c0n1.in‚66318.57421875 disk.dm-1.in‚ 108.92578125 disk.rbd0.out‚0 disk.sda3.out‚ 1663.859375) proc.memory.availableMB‚ 2981.69921875# proc.memory.rssMB‚ 2802.58203125 network.sent‚ 3.334125e+07( disk./.usagePercent‚38.5668881022608" disk.nvme0n1.in‚66318.57421875 disk.sda2.in‚0 disk.dm-0.in‚66318.57421875$ disk.dm-0.out‚171327.08447265625 disk.sda1.in‚0' disk.nvme0n1.out‚171327.08447265625 disk.sda3.in‚ 108.92578125 network.recv‚ 7.088228e+06 disk.rbd1.out‚ 3529.0390625 disk.sda.in‚ 108.92578125 disk.sda.out‚ 1663.859375& disk.rbd0.in‚1.759218604441423e+13 disk.drbd2165.out‚0# disk./.usageGB‚574.674503326416 cpu‚25.58594369576197 disk.sda2.out‚0 disk.dm-1.out‚ 1663.859375* proc.memory.percent‚17.105603218078613®2 x4³C/ _wandbruntime‚28496  _runtime‚28496ˆKúký´C˜| 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.11949094384908676 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.1941004991531372 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.26851850748062134 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8888888955116272 *jet/accuracy_3_of_4‚0.0407274030148983 epoch‚42 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.011118228547275066 6 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.147202730178833 7validation_average_jet_accuracy‚0.32346323132514954 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.7871062755584717 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 8"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.668899416923523 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6846222877502441 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.09666667133569717 particle/accuracy_3_of_5‚1 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.12454982846975327 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.12979845702648163 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.004297424107789993 particle/accuracy_2_of_3‚1 particle/accuracy_3_of_3‚1 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.15000000596046448 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.13263212144374847 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.5817930102348328 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.25582650303840637 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.16007518768310547 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.21370874345302582 particle/accuracy_4_of_5‚1 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.5254629850387573 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9837929606437683 *jet/accuracy_2_of_4‚0.6802229285240173 particle/accuracy_1_of_3‚1 particle/accuracy_2_of_5‚1 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7361111044883728 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.2256380319595337 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9891700744628906 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6068931221961975 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.46509024500846863 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6553196907043457 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.26851850748062134 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6909278035163879 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6734115481376648 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4388211667537689 -"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.5625 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.13072934746742249 +particle/sensitivity‚0.5818952322006226 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.15559789538383484 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.2083333283662796 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.14322616159915924 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.20966431498527527 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9705640077590942 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 +jet/accuracy_2_of_2‚0.11422982811927795 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.8978373408317566 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.8500000238418579 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.028694240376353264 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 +jet/accuracy_1_of_1‚0.28101056814193726 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7305902242660522 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9852820038795471 *jet/accuracy_1_of_3‚0.9071160554885864 particle/accuracy_1_of_2‚1 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 8"Purity/1FRt*SRqqt0FBt/event_purity‚0.099540576338768 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.2222222238779068 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9987687468528748 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9712021350860596 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.1084504947066307 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.48000001907348633 particle/accuracy_2_of_2‚1 /particle/accuracy_4_of_4‚0.9378424882888794 (particle/accuracy‚0.8288612365722656 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9644036293029785 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.26923078298568726 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9949828386306763 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.1303175836801529 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9812816381454468 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5936183929443359 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9888888597488403 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.26923078298568726 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.2141120582818985 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9764701128005981 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.008129189722239971 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.13511133193969727 & Purity/2FRt0SRqqt*FBt/FBt_purity‚Rç© !'1 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5038152933120728 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.030064871534705162 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.6920067667961121 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 trainer/global_step‚17199 _step‚1418 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6571744680404663 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.625 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.8500000238418579 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 particle/accuracy_1_of_1‚1 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.12457793205976486 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.47914332151412964 8"Purity/1FRt*SRqqt2FBt/event_purity‚0.627097487449646 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7345055341720581 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6449117064476013 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 *jet/accuracy_1_of_4‚0.9590054750442505 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6343035101890564 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7619304656982422 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6304526329040527 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.15640604496002197 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7386856079101562 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.7592592835426331 8"Purity/0FRt0SRqqt*FBt/event_purity‚0.992652177810669 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.020872022956609726 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8197734355926514 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9768246412277222 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.34411826729774475 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.02797875739634037 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.12979845702648163 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9841936826705933 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.09712576866149902 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.7726761102676392 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 particle/accuracy_1_of_5‚1 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6549845933914185 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6068931221961975 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5132067203521729 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7570046186447144 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8197734355926514 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7857142686843872 6 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.700264573097229 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6224874258041382 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.13337847590446472 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.1920185089111328 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8381319642066956 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7142857313156128 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 8"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.689154326915741 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.024097785353660583 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 *jet/accuracy_1_of_2‚0.7277117967605591 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9981850981712341 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.07677040249109268 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9906229376792908 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.020872022956609726 +jet/accuracy_3_of_3‚0.03754432499408722 8"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.632012665271759 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9740923047065735 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.5849193334579468 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.10244670510292053 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.7592592835426331 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.027099229395389557 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.14199841022491455 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.1132182702422142 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6087892651557922 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.17697477340698242 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6606621742248535 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7583213448524475 ,jet/accuracy_4_of_4‚0.013899499550461769 jet/accuracy_2_of_5‚1 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.12320412695407867 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.8461538553237915 ) Purity/2FRt*SRqqt2FBt/FRt_purity‚0.75 *loss/val_total_loss‚12.686022996104116 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.8564814329147339 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9936507940292358 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 +particle/specificity‚0.8964065313339233 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6218414902687073 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9939450025558472 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.6920067667961121 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 jet/accuracy_1_of_5‚1 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.15000000596046448 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6881820559501648 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7361111044883728 particle/accuracy_1_of_4‚1 - Purity/0FRt2SRqqt*FBt/FBt_purity‚0.734375 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7330008745193481 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 particle/accuracy_2_of_4‚1 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.13081671297550201 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.39814814925193787 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 jet/accuracy_3_of_5‚1 % _timestamp‚1.7843183119392276e+09 _runtime‚28496.157298642 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.2222222238779068 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 jet/accuracy_5_of_5‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 - Purity/0FRt2SRqqt1FBt/FBt_purity‚0.734375 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7084278464317322 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6617644429206848 particle/accuracy_3_of_4‚1 particle/accuracy_5_of_5‚1 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9981850981712341 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.5688120126724243 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4399922788143158 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9130856394767761 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6883578896522522 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 *jet/accuracy_2_of_3‚0.4702773690223694 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 jet/accuracy_4_of_5‚0 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9870997667312622 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9916297197341919 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.11811605840921402 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 'particle/f_score‚0.5935425758361816 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3543374538421631 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7021542191505432 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 . Purity/2FRt0SRqqt1FBt/FRt_purity‚ 0.5546875 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.5620932579040527 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.02139582857489586 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9873369336128235 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.02645712159574032 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6657368540763855 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7367451190948486 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9861281514167786ˆÇ^ª#>µC| 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.13072934746742249 +particle/sensitivity‚0.5818952322006226 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.15559789538383484 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.2083333283662796 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.14322616159915924 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.20966431498527527 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9705640077590942 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 +jet/accuracy_2_of_2‚0.11422982811927795 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.8978373408317566 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.8500000238418579 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.028694240376353264 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 +jet/accuracy_1_of_1‚0.28101056814193726 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7305902242660522 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9852820038795471 *jet/accuracy_1_of_3‚0.9071160554885864 particle/accuracy_1_of_2‚1 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 8"Purity/1FRt*SRqqt0FBt/event_purity‚0.099540576338768 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.2222222238779068 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9987687468528748 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9712021350860596 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.1084504947066307 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.48000001907348633 particle/accuracy_2_of_2‚1 /particle/accuracy_4_of_4‚0.9378424882888794 (particle/accuracy‚0.8288612365722656 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9644036293029785 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.26923078298568726 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9949828386306763 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.1303175836801529 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9812816381454468 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5936183929443359 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9888888597488403 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.26923078298568726 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.2141120582818985 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9764701128005981 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.008129189722239971 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.13511133193969727 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5038152933120728 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.030064871534705162 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.6920067667961121 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 trainer/global_step‚17199 _step‚1418 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6571744680404663 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.625 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.8500000238418579 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 particle/accuracy_1_of_1‚1 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.12457793205976486 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.47914332151412964 8"Purity/1FRt*SRqqt2FBt/event_purity‚0.627097487449646 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7345055341720581 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6449117064476013 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 *jet/accuracy_1_of_4‚0.9590054750442505 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6343035101890564 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7619304656982422 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6304526329040527 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.15640604496002197 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7386856079101562 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.7592592835426331 8"Purity/0FRt0SRqqt*FBt/event_purity‚0.992652177810669 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.020872022956609726 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8197734355926514 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9768246412277222 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.34411826729774475 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.02797875739634037 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.12979845702648163 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9841936826705933 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.09712576866149902 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.7726761102676392 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 particle/accuracy_1_of_5‚1 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6549845933914185 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6068931221961975 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5132067203521729 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7570046186447144 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8197734355926514 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7857142686843872 6 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.700264573097229 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6224874258041382 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.13337847590446472 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.1920185089111328 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8381319642066956 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7142857313156128 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 8"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.689154326915741 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.024097785353660583 ,"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.375 *jet/accuracy_1_of_2‚0.7277117967605591 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9981850981712341 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.07677040249109268 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9906229376792908 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.020872022956609726 +jet/accuracy_3_of_3‚0.03754432499408722 8"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.632012665271759 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9740923047065735 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.5849193334579468 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.10244670510292053 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.7592592835426331 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.027099229395389557 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.14199841022491455 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.1132182702422142 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6087892651557922 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.17697477340698242 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6606621742248535 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7583213448524475 ,jet/accuracy_4_of_4‚0.013899499550461769 jet/accuracy_2_of_5‚1 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.12320412695407867 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.8461538553237915 ) Purity/2FRt*SRqqt2FBt/FRt_purity‚0.75 *loss/val_total_loss‚12.686022996104116 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.8564814329147339 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9936507940292358 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 +particle/specificity‚0.8964065313339233 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6218414902687073 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9939450025558472 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.6920067667961121 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 jet/accuracy_1_of_5‚1 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.15000000596046448 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6881820559501648 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7361111044883728 particle/accuracy_1_of_4‚1 - Purity/0FRt2SRqqt*FBt/FBt_purity‚0.734375 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7330008745193481 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 particle/accuracy_2_of_4‚1 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.13081671297550201 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.39814814925193787 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 jet/accuracy_3_of_5‚1 % _timestamp‚1.7843183119392276e+09 _runtime‚28496.157298642 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.2222222238779068 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 jet/accuracy_5_of_5‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 - Purity/0FRt2SRqqt1FBt/FBt_purity‚0.734375 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7084278464317322 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6617644429206848 particle/accuracy_3_of_4‚1 particle/accuracy_5_of_5‚1 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9981850981712341 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.5688120126724243 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4399922788143158 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9130856394767761 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6883578896522522 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 *jet/accuracy_2_of_3‚0.4702773690223694 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 jet/accuracy_4_of_5‚0 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9870997667312622 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9916297197341919 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.11811605840921402 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 'particle/f_score‚0.5935425758361816 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3543374538421631 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7021542191505432 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 . Purity/2FRt0SRqqt1FBt/FRt_purity‚ 0.5546875 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.5620932579040527 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.02139582857489586 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9873369336128235 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.02645712159574032 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6657368540763855 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7367451190948486 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9861281514167786 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.11949094384908676 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.1941004991531372 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.26851850748062134 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8888888955116272 *jet/accuracy_3_of_4‚0.0407274030148983 epoch‚42 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.011118228547275066 6 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.147202730178833 7validation_average_jet_accuracy‚0.32346323132514954 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.7871062755584717 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 8"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.668899416923523 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6846222877502441 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.09666667133569717 particle/accuracy_3_of_5‚1 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.12454982846975327 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.12979845702648163 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.004297424107789993 particle/accuracy_2_of_3‚1 particle/accuracy_3_of_3‚1 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.15000000596046448 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.13263212144374847 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.5817930102348328 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.25582650303840637 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.16007518768310547 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.21370874345302582 particle/accuracy_4_of_5‚1 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.5254629850387573 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9837929606437683 *jet/accuracy_2_of_4‚0.6802229285240173 particle/accuracy_1_of_3‚1 particle/accuracy_2_of_5‚1 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7361111044883728 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.2256380319595337 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9891700744628906 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6068931221961975 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.46509024500846863 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6553196907043457 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.26851850748062134 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6909278035163879 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6734115481376648 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4388211667537689 -"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.5625 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1Š ­‡Oœ¶C‚(:‘ ë’êÒ³¼îö gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚333.764' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚66.75280000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ê4šP·C‚(:‹ ë’êÒ“ÉŠù disk.dm-0.out‚171334.8984375$ disk./.usageGB‚574.6745185852051 disk.dm-1.out‚ 1664.16796875 disk.rbd0.out‚0 disk.dm-0.in‚66318.57421875 disk.sda1.in‚0) disk./.usagePercent‚38.56688912629093 disk.dm-1.in‚ 108.92578125 disk.sda3.in‚ 108.92578125$ memory_percent‚81.79996013641357 disk.sda2.out‚0 disk.sda3.out‚ 1664.16796875) proc.memory.availableMB‚ 2981.89453125 network.recv‚ 7.091536e+06 disk.sda2.in‚0 disk.rbd1.in‚ 2.9619140625 network.sent‚ 3.3346864e+07# disk.nvme0n1.out‚171334.8984375 disk.sda.out‚ 1664.16796875 disk.drbd2165.out‚0* proc.memory.percent‚17.105603218078613$ disk.nvme0c0n1.in‚66318.57421875" disk.nvme0n1.in‚66318.57421875& disk.rbd0.in‚1.759218604441423e+13 disk.sda1.out‚0 cpu‚25.586731440915216 proc.cpu.threads‚17 disk.sda.in‚ 108.92578125 disk.drbd2165.in‚0# proc.memory.rssMB‚ 2802.58203125% disk.nvme0c0n1.out‚171334.8984375 disk.rbd1.out‚ 3529.0390625=¹ô6Ÿ¸C‚(:” ú’êÒ«¯Øö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚263.406' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚52.681200000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÞIŹC‚(: ú’êÒ‰Îõø) disk.nvme0c0n1.out‚171492.68017578125 disk.sda.out‚ 1664.3515625( proc.memory.availableMB‚ 2981.9140625 proc.cpu.threads‚17 disk.drbd2165.out‚0 disk.sda1.out‚0 disk.sda2.out‚0 cpu‚25.586694918381962 network.recv‚ 7.095985e+06 disk.drbd2165.in‚0' disk.nvme0n1.out‚171492.68017578125$ disk./.usageGB‚574.6731872558594 disk.dm-0.in‚ 66318.578125 disk.dm-1.in‚ 108.92578125 disk.rbd0.out‚0$ memory_percent‚81.79984092712402" disk.nvme0c0n1.in‚ 66318.578125) disk./.usagePercent‚38.56679977966148$ disk.dm-0.out‚171492.68017578125 disk.sda3.in‚ 108.92578125 disk.nvme0n1.in‚ 66318.578125 disk.sda2.in‚0 disk.rbd1.out‚ 3676.9140625# proc.memory.rssMB‚ 2802.58203125 network.sent‚ 3.3404696e+07 disk.sda.in‚ 108.92578125 disk.dm-1.out‚ 1664.3515625& disk.rbd0.in‚1.759218604441423e+13* proc.memory.percent‚17.105603218078613 disk.sda1.in‚0 disk.sda3.out‚ 1664.3515625 disk.rbd1.in‚ 2.96191406259ÕÆ„ŸºC‚(:” ‰“êÒŸ³úö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚279.015' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚55.803000000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¬SGŒ »C‚(:ÿ ‰“êÒÕüøø network.recv‚ 7.099161e+06 disk.sda.in‚ 108.92578125% disk.nvme0n1.out‚172015.091796875' disk./.usagePercent‚38.566796195556 disk.rbd1.out‚ 3762.1484375 disk.sda3.in‚ 108.92578125 proc.cpu.threads‚17 network.sent‚ 3.3411137e+07! disk.nvme0n1.in‚ 66417.6171875* proc.memory.percent‚17.105603218078613 cpu‚25.58668434405885 disk.rbd1.in‚ 2.9619140625 disk.sda.out‚ 1664.6875 disk.rbd0.out‚0 disk.dm-0.in‚ 66417.6171875$ disk./.usageGB‚574.6731338500977 disk.dm-1.in‚ 108.92578125" disk.dm-0.out‚172015.091796875 disk.sda1.out‚0 disk.sda3.out‚ 1664.6875 disk.drbd2165.in‚0$ memory_percent‚81.79986476898193# proc.memory.rssMB‚ 2802.58203125 disk.sda1.in‚0 disk.drbd2165.out‚0 disk.dm-1.out‚ 1664.6875& disk.rbd0.in‚1.759218604441423e+13 disk.sda2.in‚0# disk.nvme0c0n1.in‚ 66417.6171875' disk.nvme0c0n1.out‚172015.091796875 disk.sda2.out‚0) proc.memory.availableMB‚ 2981.91015625ŸZ»&”¼C‚(:‰ ˜“êÒ‚Е÷ gpu.0.gpu‚99.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚207.691' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚41.5382 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0»ÉÃœ½C‚(:“ ˜“êÒ°‡Ÿú disk.sda3.out‚ 1664.96484375 disk.sda2.out‚0 network.recv‚ 7.103983e+06 disk.drbd2165.out‚0' disk.nvme0n1.out‚172539.56689453125 disk.sda2.in‚0$ disk.dm-0.out‚172539.56689453125 disk.sda1.out‚0) disk.nvme0c0n1.out‚172539.56689453125* proc.memory.percent‚17.105603218078613 disk.sda.out‚ 1664.96484375& disk.rbd0.in‚1.759218604441423e+13 disk.dm-0.in‚ 66417.8359375 disk.drbd2165.in‚0 disk.rbd1.out‚ 3841.3671875) proc.memory.availableMB‚ 2981.63671875 network.sent‚ 3.341637e+07 disk.sda.in‚ 108.92578125 disk.dm-1.out‚ 1664.96484375 cpu‚25.586732710320526$ disk./.usageGB‚574.6700668334961 disk.dm-1.in‚ 108.92578125 disk.rbd0.out‚0 disk.sda3.in‚ 108.92578125 proc.cpu.threads‚17# disk.nvme0c0n1.in‚ 66417.8359375$ memory_percent‚81.80153369903564! disk.nvme0n1.in‚ 66417.8359375 disk.rbd1.in‚ 2.9619140625# proc.memory.rssMB‚ 2802.58203125) disk./.usagePercent‚38.56659036549848 disk.sda1.in‚0&ä-ož¾C‚(:“ §“êÒ¶˜òö gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚272.302' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚54.46040000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0HòŠ&¿C‚(:† §“êÒ½»†ù) disk./.usagePercent‚38.56649743476355( disk.nvme0c0n1.out‚172578.4169921875 disk.sda2.out‚0 disk.sda1.in‚0 disk.rbd0.out‚0# memory_percent‚81.8030595779419) proc.memory.availableMB‚ 2981.38671875* proc.memory.percent‚17.105603218078613# disk.dm-0.out‚172578.4169921875 disk.sda.in‚ 109.4921875 disk.sda.out‚ 1667.80859375! disk.nvme0c0n1.in‚ 66417.84375 disk.rbd1.out‚ 3920.5859375# proc.memory.rssMB‚ 2802.58203125 network.sent‚ 3Ïvý .3420087e+07 disk.nvme0n1.in‚ 66417.84375& disk.rbd0.in‚1.759218604441423e+13 disk.sda1.out‚0 cpu‚25.58673310101083 proc.cpu.threads‚17 disk.rbd1.in‚ 2.9619140625 disk.dm-1.in‚ 109.4921875 disk.sda3.out‚ 1667.80859375 disk.drbd2165.out‚0& disk.nvme0n1.out‚172578.4169921875 disk.sda3.in‚ 109.4921875 network.recv‚ 7.107009e+06 disk.sda2.in‚0 disk.dm-1.out‚ 1667.80859375 disk.dm-0.in‚ 66417.84375$ disk./.usageGB‚574.6686820983887 disk.drbd2165.in‚0”¿Q4ÀC/ _wandbruntime‚28574  _runtime‚28574ú;ŠõËÁCÅ ' lr-AdamW/pg2‚3.3088081231890286e-05 trainer/global_step‚17249 $ _timestamp‚1.784318390397798e+09 _runtime‚28574.614913811 _step‚1419 ' lr-AdamW/pg1‚3.3088081231890286e-05MANÐÂCÊ ' lr-AdamW/pg2‚3.3088081231890286e-05 trainer/global_step‚17249 $ _timestamp‚1.784318390397798e+09 _runtime‚28574.614913811 _step‚1419 ' lr-AdamW/pg1‚3.3088081231890286e-05‹ ó†Ô4ÃC/ _wandbruntime‚28574  _runtime‚28574E~`(ÄC’P ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚4960203298373632 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚44254423 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚4974264367524005 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 147234129 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚256 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚4976434167282688 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚4960203272200192 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16201751453603 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16229523519488 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚93117065 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚4976432795719680 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚20933396702720 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 147234123 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚48862637 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1359834875246080 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚16203096053925 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 $ _timestamp‚1.784318390398592e+09 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1338901478543360 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 149426979 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 trainer/global_step‚17249 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚20933362166272 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚4974262997495715 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚4958061271470080 _runtime‚28574.61596339 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16229523519488 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚48862381 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 149417006 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚93116804 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚261 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚4976432795719680 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 149417006 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚44254428 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1338901110452224 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 147234129 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 296651129 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚4976434167282688 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 _step‚1420 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16230868909056 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16230868909056 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0"p=¼(ÅC—P KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚4958061246042112 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚4960203272200192 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 149426979 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚4976432795719680 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 149417006 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚44254428 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1338901110452224 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 147234129 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 296651129 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚4976434167282688 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 _step‚1420 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16230868909056 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16230868909056 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚4960203298373632 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚44254423 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚4974264367524005 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 147234129 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚256 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚4976434167282688 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚4960203272200192 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16201751453603 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16229523519488 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚93117065 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚4976432795719680 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚20933396702720 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 147234123 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚48862637 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1359834875246080 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚16203096053925 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 $ _timestamp‚1.784318390398592e+09 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1338901478543360 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 149426979 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 trainer/global_step‚17249 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚20933362166272 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚4974262997495715 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚4958061271470080 _runtime‚28574.61596339 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16229523519488 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚48862381 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 149417006 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚93116804 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚261 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 JDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16230448555008 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚4976711721485312 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16230448555008 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16204019520283 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 149435471 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 147242329 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚251 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚16231793951744 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 147242335 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚4974541802740137 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚4958339153248256 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚93122101 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚246 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚44256966 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 296667813 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 149435471 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚4958339127820288 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚4960481299103744 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚20934567315968 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚48865381 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚16231793951744 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚4974543172768539 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚93122352 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 296677806 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1359913164175872 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚44256971 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 P@DeviceStatsMonitor.on+ÒÕG_train_batch_end/reserved_bytes.all.current‚ 49071259648 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 147242329 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚4976713093055488 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0'€G('ÊC¢N OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 147242335 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚4976713093055488 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚4960481272930304 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 _runtime‚28576.002526292 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚4960481272930304 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 296677806 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1359913164175872 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚44256971 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 147242329 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚4976713093055488 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16230448555008 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚4976711721485312 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16230448555008 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16204019520283 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 149435471 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 147242329 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚251 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚16231793951744 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 147242335 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚4974541802740137 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚4958339153248256 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚93122101 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚246 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚44256966 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 296667813 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 149435471 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚4958339127820288 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚4960481299103744 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚20934567315968 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚48865381 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚16231793951744 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚4974543172768539 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚93122352 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 QDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚44381573 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1342775430836224 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚4990379148155904 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚4990379148155904 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1363768728846848 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚20993332547072 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _runtime‚28654.135067849 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 149842758 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16275839450112 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚49000891 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚4971955334950912 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚93382469 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216»‡ñ”ê ÝC™P F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 297487181 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚4974103334879232 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚44381578 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 297497160 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 147644429 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 149852731 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚93382203 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16277184839680 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚44381573 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1342775430836224 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚4990379148155904 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚4990379148155904 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1363768728846848 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚20993332547072 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _runtime‚28654.135067849 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 149842758 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16275839450112 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚49000891 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚4971955334950912 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚93382469 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚4988204693545913 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚4990380519718912 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚4988482128762045 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚93387757 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚256 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚4990659445491712 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 147652635 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1342852917243904 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16276764485632 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚20994493200896 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚93387501 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚16278109882368 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 147652635 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 147652629 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚4972233216729088 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 297513858 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 297513858 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚4990658073921536 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 149861223 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚251 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚4988483498790447 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402'O")'àC£N SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 149861223 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚251 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚4988483498790447 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚4972233242157056 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚44384116 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16248912032957 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚4988482128762045 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚93387757 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚256 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚4990659445491712 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 147652635 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1342852917243904 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16276764485632 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚20994493200896 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚93387501 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚16278109882368 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 147652635 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 147652629 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚4972233216729088 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 297513858 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 297513858 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚4990658073921536 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0‘ ûq04áC/ _wandbruntime‚28655  _runtime‚28655úÇ×âCŒ 3loss/SRqqt1/assignment_loss‚0.05279764446444489 4loss/FBt1/assignment_loss‚1.3245475827261544e-08 0loss/FBt1/detection_loss‚0.06991500637556353 /loss/FBt2/detection_loss‚0.0708855597006461 _step‚1426 3loss/SRqqt2/assignment_loss‚0.07430037298257852 3loss/FBt2/assignment_loss‚6.333876481474161e-06 1loss/SRqqt1/detection_loss‚0.4194263269408149 epoch‚43 trainer/global_step‚17299 /loss/FRt1/assignment_loss‚0.762166417294661 1loss/FRt2/assignment_loss‚0.07676092916405293 .loss/FRt1/detection_loss‚0.398201831568658 /loss/FRt2/detection_loss‚1.1608128684220786 1loss/SRqqt2/detection_loss‚0.4304699535025321 %loss/total_loss‚3.515743257537988 % _timestamp‚1.7843184715787966e+09 _runtime‚28655.795740849 +FãC‘ /loss/FRt1/assignment_loss‚0.762166417294661 1loss/FRt2/assignment_loss‚0.07676092916405293 .loss/FRt1/detection_loss‚0.398201831568658 /loss/FRt2/detection_loss‚1.1608128684220786 1loss/SRqqt2/detection_loss‚0.4304699535025321 %loss/total_loss‚3.515743257537988 % _timestamp‚1.7843184715787966e+09 _runtime‚28655.795740849 3loss/SRqqt1/assignment_loss‚0.05279764446444489 4loss/FBt1/assignment_loss‚1.3245475827261544e-08 0loss/FBt1/detection_loss‚0.06991500637556353 /loss/FBt2/detection_loss‚0.0708855597006461 _step‚1426 3loss/SRqqt2/assignment_loss‚0.07430037298257852 3loss/FBt2/assignment_loss‚6.333876481474161e-06 1loss/SRqqt1/detection_loss‚0.4194263269408149 epoch‚43 trainer/global_step‚17299’ qiç’äC‚(:‡ ”êҽЃ÷ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚269.59' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.918 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0€ åC‚(:ˆ ”êÒÓÒ…ú& disk.rbd0.in‚1.759218604441423e+13 disk.rbd0.out‚0# memory_percent‚81.8063735961914 disk.drbd2165.out‚0* proc.memory.percent‚17.105603218078613 disk.dm-1.in‚ 109.4921875 disk.dm-1.out‚ 1672.828125 disk.sda1.out‚0& proc.memory.availableMB‚ 2980.84375) disk./.usagePercent‚38.56687683792929$ disk.nvme0c0n1.in‚66616.51171875 disk.sda.in‚ 109.4921875 network.sent‚ 3.3514183e+07 disk.sda3.in‚ 109.4921875' disk.nvme0n1.out‚172644.56298828125 disk.rbd1.out‚ 4473.1015625 disk.sda1.in‚0# proc.memory.rssMB‚ 2802.58203125 cpu‚25.5866142361068 disk.sda2.in‚0 disk.drbd2165.in‚0" disk.nvme0n1.in‚66616.51171875 disk.sda.out‚ 1672.828125 disk.dm-0.in‚66616.51171875 disk.sda2.out‚0$ disk.dm-0.out‚172644.56298828125 proc.cpu.threads‚17 network.recv‚ 7.133423e+06$ disk./.usageGB‚574.6743354797363 disk.sda3.out‚ 1672.828125) disk.nvme0c0n1.out‚172644.56298828125 disk.rbd1.in‚ 2.9619140625ýJ’æC‚(:‡ Ÿ”êÒíø“÷ gpu.0.gpu‚24.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚39.0 gpu.0.powerWatts‚110.125' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.025 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0½·[øçC‚(:ý Ÿ”êÒ”•¤ù disk.sda.in‚ 109.4921875 network.recv‚ 7.138756e+06 disk.sda2.out‚0 disk.dm-1.in‚ 109.4921875& disk.rbd0.in‚1.759218604441423e+13 disk.drbd2165.out‚0# memory_percent‚81.8063735961914 network.sent‚ 3.3577365e+07' disk.nvme0n1.out‚172651.54248046875 disk.sda1.in‚0$ disk.dm-0.out‚172651.54248046875 disk.sda3.in‚ 109.4921875 disk.sda2.in‚0 disk.rbd1.in‚ 2.9619140625 disk.dm-1.out‚ 1673.1875 disk.rbd0.out‚0 disk.dm-0.in‚ 66616.671875# proc.memory.rssMB‚ 2802.58203125$ disk./.usageGB‚574.6740226745605& proc.memory.availableMB‚ 2980.84375 disk.rbd1.out‚ 4656.0234375" disk.nvme0c0n1.in‚ 66616.671875 disk.sda1.out‚0 disk.sda3.out‚ 1673.1875 disk.drbd2165.in‚0) disk./.usagePercent‚38.56685584531148) disk.nvme0c0n1.out‚172651.54248046875 disk.nvme0n1.in‚ 66616.671875* proc.memory.percent‚17.105603218078613 disk.sda.out‚ 1673.1875 cpu‚25.58662268979702 proc.cpu.threads‚17ÌygZèC‚(:” ®”êÒáÈëö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚295.399' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚59.079800000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! õm1àEgpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¨â–í éC‚(:þ ®”êÒðâñø disk.rbd0.out‚0) proc.memory.availableMB‚ 2980.85546875* disk./.usagePercent‚38.566895782486824& disk.rbd0.in‚1.759218604441423e+13 disk.dm-0.out‚ 172661.203125 disk.sda2.out‚0 disk.sda1.in‚0 disk.sda1.out‚0 network.recv‚ 7.142451e+06" disk.nvme0c0n1.in‚ 66617.515625 disk.sda3.in‚ 109.4921875 network.sent‚ 3.3582418e+07* proc.memory.percent‚17.105603218078613# disk./.usageGB‚574.674617767334 disk.dm-1.in‚ 109.4921875 disk.dm-1.out‚ 1673.37109375 disk.dm-0.in‚ 66617.515625 disk.sda3.out‚ 1673.37109375" disk.nvme0n1.out‚ 172661.203125 disk.rbd1.in‚ 2.9619140625 disk.rbd1.out‚ 4749.0859375 disk.sda.in‚ 109.4921875 disk.sda.out‚ 1673.37109375# proc.memory.rssMB‚ 2802.58203125 cpu‚25.58658684186761 disk.drbd2165.in‚0 disk.drbd2165.out‚0$ disk.nvme0c0n1.out‚ 172661.203125 disk.nvme0n1.in‚ 66617.515625 disk.sda2.in‚0$ memory_percent‚81.80630207061768 proc.cpu.threads‚17B%:ŸêC‚(:” ½”êÒÊ“éö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚41.0 gpu.0.powerWatts‚112.143' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.428600000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0nÁûëC‚(:… ½”êÒØÅžù disk.rbd0.out‚0 disk.sda1.in‚0 cpu‚25.586575799915295 network.recv‚ 7.14529e+06 disk.sda1.out‚0 proc.cpu.threads‚17 disk.sda3.in‚ 109.4921875( disk.nvme0c0n1.out‚172669.0732421875 disk.sda2.out‚0 disk.rbd1.out‚ 4893.0234375 network.sent‚ 3.3587362e+07$ disk.nvme0c0n1.in‚66711.22265625" disk.nvme0n1.in‚66711.22265625 disk.dm-1.out‚ 1673.84375& disk.nvme0n1.out‚172669.0732421875$ memory_percent‚81.80627822875977 disk.sda.out‚ 1673.84375 disk.dm-1.in‚ 109.4921875 disk.dm-0.in‚66711.22265625 disk.drbd2165.out‚0 disk.rbd1.in‚ 2.9619140625# proc.memory.rssMB‚ 2802.58203125* proc.memory.percent‚17.105603218078613) disk./.usagePercent‚38.56689629450189$ disk./.usageGB‚574.6746253967285 disk.sda.in‚ 109.4921875# disk.dm-0.out‚172669.0732421875 disk.sda3.out‚ 1673.84375 disk.drbd2165.in‚0 disk.sda2.in‚0' proc.memory.availableMB‚ 2980.859375& disk.rbd0.in‚1.759218604441423e+13&yñtžìC‚(:“ Ì”êÒ–°¢÷ gpu.0.gpu‚99.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚259.417' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚51.88339999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ô+F)íC‚(:‹ Ì”êÒÊŠÒù disk.sda2.out‚0 network.sent‚ 3.3594017e+07$ disk./.usageGB‚574.6748085021973 disk.sda3.in‚ 109.4921875 disk.drbd2165.in‚0) disk.nvme0c0n1.out‚172675.49169921875 disk.drbd2165.out‚0' disk.nvme0n1.out‚172675.49169921875# proc.memory.rssMB‚ 2802.58203125 network.recv‚ 7.149468e+06 disk.rbd1.in‚ 2.9619140625 disk.dm-0.in‚66711.22265625 disk.sda1.out‚0$ disk.nvme0c0n1.in‚66711.22265625 disk.sda2.in‚0 disk.sda.in‚ 109.4921875$ disk.dm-0.out‚172675.49169921875# memory_percent‚81.8028450012207" disk.nvme0n1.in‚66711.22265625) disk./.usagePercent‚38.56690858286353 disk.rbd1.out‚ 4961.6328125 disk.dm-1.out‚ 1674.203125& disk.rbd0.in‚1.759218604441423e+13 disk.rbd0.out‚0 disk.sda1.in‚0' proc.memory.availableMB‚ 2981.421875 disk.sda3.out‚ 1674.203125 disk.dm-1.in‚ 109.4921875 cpu‚25.586564657026667 proc.cpu.threads‚17* proc.memory.percent‚17.105603218078613 disk.sda.out‚ 1674.203125ýu\m4îC/ _wandbruntime‚28733  _runtime‚28733£ÕØÎÊïCÄ _step‚1427 & lr-AdamW/pg1‚3.076193951046773e-05 & lr-AdamW/pg2‚3.076193951046773e-05 trainer/global_step‚17349 % _timestamp‚1.7843185497029333e+09 _runtime‚28733.921078698}úàªÏðCÉ & lr-AdamW/pg1‚3.076193951046773e-05 & lr-AdamW/pg2‚3.076193951046773e-05 trainer/global_step‚17349 % _timestamp‚1.7843185497029333e+09 _runtime‚28733.921078698 _step‚1427“ á'Á„4ñC/ _wandbruntime‚28733  _runtime‚28733Ìüå(òC”P GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚5004325500592128 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 148054723 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 % _timestamp‚1.7843185497047873e+09 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 _step‚1428 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚49138863 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 150268510 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1367703243867648 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚5004326872155136 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚93647591 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚4988003371384832 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚281 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚44508728 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 148054723 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚21053087922688 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16323500770304 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚4988003345211392 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚5004325500592128 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚93647310 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚16295570280141 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404724736 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 150278483 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16323500770304 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚4985849423859712 _runtime‚28733.925397041 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 150278483 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1367702839142912 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 148054729 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚5004326872155136 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚44508723 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚276 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36633600 trainer/global_step‚17349 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚4988003371384832 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚4985849449287680 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1346650119311360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1346649751220224 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 150268510 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 148054729 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚5002145019567821 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚5002143649539531 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16322155380736 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚4988003345211392 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16322155380736 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 298333212 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚21053124556288 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚49138587 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16294225679819™èÆ(óC™P ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚4988003345211392 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚5004325500592128 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚93647310 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚16295570280141 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404724736 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 150278483 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16323500770304 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚4985849423859712 _runtime‚28733.925397041 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 150278483 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1367702839142912 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 148054729 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚5004326872155136 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚44508723 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚276 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36633600 trainer/global_step‚17349 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚4988003371384832 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚4985849449287680 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1346650119311360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1346649751220224 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 150268510 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 148054729 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚5002145019567821 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚5002143649539531 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16322155380736 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚4988003345211392 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16322155380736 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 298333212 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚21053124556288 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚49138587 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16294225679819 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚5004325500592128 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 148054723 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 % _timestamp‚1.7843185497047873e+09 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 _step‚1428 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚49138863 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 150268510 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1367703243867648 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚5004326872155136 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚93647591 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚4988003371384832 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚281 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚44508728 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 148054723 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚21053087922688 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16323500770304” O¶¸Ï4ôC/ _wandbruntime‚28735  _runtime‚287359'VõõC›N _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚4986127331065856 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 298339917 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚4988281345941504 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚21054308261376 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚93652609 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 148062929 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 _runtime‚ 28735.3102858 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚16324425812992 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚5002423824812355 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚49141602 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚49141343 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1367781913980416 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16323080416256 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 $ _timestamp‚1.784318551092876e+09 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚4988281372114944 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚4988281372114944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 298349910 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 trainer/global_step‚17349 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16323080416256 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36626432 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 298339917 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚4986127305637888 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 150276988 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 148062935 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 150276988 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚264 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1346727605719040 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0¦ïÅS¢  C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 148062935 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 150286975 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚5004605797927936 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚259 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 150286975 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404717568 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1367781509262848 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1346727237627904 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16295149146065 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136jSÜ&'öC N KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚16324425812992 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚5002423824812355 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚49141602 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚49141343 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1367781913980416 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16323080416256 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 $ _timestamp‚1.784318551092876e+09 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚4988281372114944 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚4988281372114944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 298349910 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 trainer/global_step‚17349 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16323080416256 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36626432 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 298339917 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚4986127305637888 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 150276988 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 148062935 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 150276988 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚264 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1346727605719040 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 148062935 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 150286975 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚5004605797927936 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚259 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 150286975 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404717568 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1367781509262848 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1346727237627904 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16295149146065 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚4986127331065856 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 298339917 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚4988281345941504 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚21054308261376 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚93652609 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 148062929 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 _runtime‚ 28735.3102858 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119• »¤ŠJ4÷C/ _wandbruntime‚28735  _runtime‚28735ÜRõ–ÿøCù 0loss/FRt1/detection_loss‚0.42467122141005015 2loss/SRqqt1/detection_loss‚0.37751189321206424 0loss/FRt1/assignment_loss‚0.7909536627347996 3loss/SRqqt1/assignment_loss‚0.05485124916473858 /loss/FRt2/detection_loss‚1.0358793271779347 2loss/SRqqt2/detection_loss‚0.42440558961700514 .loss/FBt1/detection_loss‚0.213658467440547 &loss/total_loss‚3.5159532808491125 % _timestamp‚1.7843185511785421e+09 /loss/FBt2/detection_loss‚0.0611569562228104 epoch‚43 trainer/global_step‚17349 _step‚1430 1loss/FRt2/assignment_loss‚0.05385097982314638 2loss/FBt1/assignment_loss‚9.14861193106929e-08 _runtime‚28735.39547892 3loss/SRqqt2/assignment_loss‚0.07901384255989656 loss/FBt2/assignment_loss‚0AŠúëùCþ _step‚1430 1loss/FRt2/assignment_loss‚0.05385097982314638 2loss/FBt1/assignment_loss‚9.14861193106929e-08 _runtime‚28735.39547892 3loss/SRqqt2/assignment_loss‚0.07901384255989656 loss/FBt2/assignment_loss‚0 0loss/FRt1/detection_loss‚0.42467122141005015 2loss/SRqqt1/detection_loss‚0.37751189321206424 0loss/FRt1/assignment_loss‚0.7909536627347996 3loss/SRqqt1/assignment_loss‚0.05485124916473858 /loss/FRt2/detection_loss‚1.0358793271779347 2loss/SRqqt2/detection_loss‚0.42440558961700514 .loss/FBt1/detection_loss‚0.213658467440547 &loss/total_loss‚3.5159532808491125 % _timestamp‚1.7843185511785421e+09 /loss/FBt2/detection_loss‚0.0611569562228104 epoch‚43 trainer/global_step‚17349– ý˜³µ”úC‚(:‰ Û”êÒô’ƒ÷ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚263.713' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.7426 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¸ˆÿûC‚(: Û”êÒ·…Ÿù disk.sda2.in‚0 disk.sda1.out‚0 disk.sda3.out‚ 1674.51953125 proc.cpu.threads‚17 disk.sda2.out‚0$ disk.dm-0.out‚172686.60986328125 disk.drbd2165.in‚0( proc.memory.availableMB‚ 2981.1640625# proc.memory.rssMB‚ 2802.58203125$ disk./.usageGB‚574.6747093200684 disk.rbd1.out‚ 5084.6953125& disk.rbd0.in‚1.759218604441423e+13 disk.rbd0.out‚0 disk.sda.in‚ 109.4921875 disk.sda3.in‚ 109.4921875 disk.drbd2165.out‚0# disk.nvme0c0n1.in‚ 66712.0546875 disk.dm-1.in‚ 109.4921875 disk.dm-1.out‚ 1674.51953125 network.recv‚ 7.151869e+06) disk./.usagePercent‚38.56690192666765$ memory_percent‚81.80441856384277 disk.rbd1.in‚ 2.9619140625 disk.dm-0.in‚ 66712.0546875* proc.memory.percent‚17.105603218078613 network.sent‚ 3.3597576e+07' disk.nvme0n1.out‚172686.60986328125 disk.sda1.in‚0) disk.nvme0c0n1.out‚172686.60986328125 cpu‚25.58651157710622 disk.sda.out‚ 1674.51953125! disk.nvme0n1.in‚ 66712.0546875nÕ,ÊžüC‚(:“ ê”êÒ¾µôö gpu.0.gpu‚63.0 gpu.0.memory‚20, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚332.989' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚66.59779999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0«”ŠèýC‚(:Ž ê”ê҇Ζù disk.sda3.out‚ 1674.70703125 disk.drbd2165.out‚0 disk.sda2.out‚0 disk.sda.in‚ 109.4921875& disk.rbd0.in‚1.759218604441423e+13* proc.memory.percent‚17.105603218078613& disk.nvme0n1.out‚172690.8505859375 disk.rbd0.out‚0( disk.nvme0c0n1.out‚172690.8505859375 network.recv‚ 7.158322e+06$ disk./.usageGB‚574.6744270324707 disk.drbd2165.in‚0) proc.memory.availableMB‚ 2981.38671875 network.sent‚ 3.3662588e+07* disk./.usagePercent‚38.566882982110116 disk.sda.out‚ 1674.70703125 disk.dm-1.in‚ 109.4921875 disk.dm-1.out‚ 1674.70703125# disk.dm-0.out‚172690.8505859375 proc.cpu.threads‚17! disk.nvme0n1.in‚ 66712.0546875 disk.sda1.in‚0 disk.sda3.in‚ 109.4921875 disk.sda2.in‚0 disk.rbd1.in‚ 2.9619140625 disk.dm-0.in‚ 66712.0546875# memory_percent‚81.8030595779419# proc.memory.rssMB‚ 2802.58203125 disk.sda1.out‚0# disk.nvme0c0n1.in‚ 66712.0546875 disk.rbd1.out‚ 5088.1328125 cpu‚25.586527412837437Åâ$|ŸþC‚(:” ù”êÒîÓ€ø gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚273.071' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚54.614200000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0X:’{ùÿC‚(:î ù”êÒ¼¦Ðù disk.rbd1.out‚ 5267.3828125# disk.dm-0.out‚172697.9560546875$ memory_percent‚81.80301189422607) disk./.usagePercent‚38.56686890169573 disk.rbd1.in‚ 2.9619140625( disk.nvme0c0n1.out‚172697.9560546875# proc.memory.rssMB‚ 2802.58203125 disk.dm-0.in‚ 66807.703125 disk.sda1.out‚0 disk.drbd2165.in‚0) proc.memory.availableMB‚ 2981.39453125 disk.sda.in‚ 109.4921875 disk.dm-1.out‚1675 disk.nvme0n1.in‚ 66807.703125& disk.nvme0n1.out‚172697.9560546875 cpu‚25.58650753136571 network.sent‚ 3.3666315e+07$ disk./.usageGB‚574.6742172241211 disk.rbd0.out‚0 disk.sda1.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.in‚ 109.4921875 disk.drbd2165.out‚0" disk.nvme0c0n1.in‚ 66807.703125 disk.sda.out‚1675 disk.dm-1.in‚ 109.4921875 disk.sda3.out‚1675* proc.memory.percent‚17.105603218078613 network.recv‚ 7.16176e+06& disk.rbd0.in‚1.759218604441423e+13 proc.cpu.threads‚17¤ÆΔ€D‚(:‰ ˆ•êÒ˜Òàö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚292.207' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚58.4414 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Áè¢D‚(:‹ ˆ•êÒËâø disk.drbd2165.out‚0 disk.sda2.out‚0 disk.sda1.in‚0 disk.rbd1.in‚ 2.9619140625 disk.sda1.out‚0$ disk.nvme0c0n1.in‚66808.53515625 disk.sda.in‚ 109.4921875 disk.dm-1.in‚ 109.4921875 disk.dm-1.out‚ 1675.1484375" disk.dm-0.out‚172707.732421875$ memory_percent‚81.80396556854248 network.recv‚ 7.165538e+06% disk.nvme0n1.out‚172707.732421875) disk./.usagePercent‚38.56700074557586 disk.sda.out‚ 1675.1484375 disk.dm-0.in‚66808.53515625* proc.memory.percent‚17.105603218078613 proc.cpu.threads‚17 disk.sda3.out‚ 1675.1484375 disk.rbd1.out‚ 5342.1796875# proc.memory.rssMB‚ 2802.58203125 cpu‚25.586481449678466) proc.memory.availableMB‚ 2981.23828125$ disk./.usageGB‚574.6761817932129 disk.sda3.in‚ 109.4921875' disk.nvme0c0n1.out‚172707.732421875" disk.nvme0n1.in‚66808.53515625 disk.sda2.in‚0 disk.rbd0.out‚0 network.sent‚ 3.3671386e+07 disk.drbd2165.in‚0& disk.rbd0.in‚1.759218604441423e+13 ÈÝÈŸ‚D‚(:” —•êÒô¤„÷ gpu.0.gpu‚90.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚277.162' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚55.432399999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0T}½ñƒD‚(:Ž —•êÒ“¾ƒú disk.sda.out‚ 1675.546875 disk.dm-1.in‚ 109.4921875$ disk.nvme0c0n1.in‚66808.53515625" disk.nvme0n1.in‚66808.53515625 disk.dm-0.in‚66808.53515625 disk.drbd2165.in‚0' disk.nvme0n1.out‚172714.73974609375 disk.sda2.out‚0 proc.cpu.threads‚17 network.sent‚ 3.3676377e+07 disk.drbd2165.out‚0 disk.rbd1.in‚ 2.9619140625 disk.rbd1.out‚ 5402.5546875 network.recv‚ 7.168725e+06$ disk./.usageGB‚574.6763572692871 disk.sda3.out‚ 1675.546875 disk.sda2.in‚0* proc.memory.percent‚17.105603218078613) disk.nvme0c0n1.out‚172714.73974609375* disk./.usagePercent‚38.567012521922436 disk.dm-1.out‚ 1675.546875 disk.rbd0.out‚0$ disk.dm-0.out‚172714.73974609375 disk.sda1.out‚0 disk.sda3.in‚ 109.4921875 disk.sda.in‚ 109.4921875 disk.sda1.in‚0$ memory_percent‚81.80394172668457( proc.memory.availableMB‚ 2981.2421875# proc.memory.rssMB‚ 2802.58203125 cpu‚25.586451441728844& disk.rbd0.in‚1.759218604441423e+13¿eeò4„D/ _wandbruntime‚28813  _runtime‚28813‹„žÌÌ…DÆ _runtime‚28813.592010052 _step‚1431 ' lr-AdamW/pg1‚2.9629195049757518e-05 ' lr-AdamW/pg2‚2.9629195049757518e-05 trainer/global_step‚17399 % _timestamp‚1.7843186293748176e+09¿‡§VцDË _runtime‚28813.592010052 _step‚1431 ' lr-AdamW/pg1‚2.9629195049757518e-05 ' lr-AdamW/pg2‚2.9629195049757518e-05 trainer/global_step‚17399 % _timestamp‚1.7843186293748176e+09— KwWw4‡D/ _wandbruntime‚28813  _runtime‚28813‡Îœ(ˆD“P M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1371636875835904 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404724736 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚272 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚5018271853028352 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 148465023 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 $ _timestamp‚1.784318629379504e+09 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚49276675 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1371636471111168 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚5018271853028352 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚93912548 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚21112436140544 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 _runtime‚28813.596955598 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 150704235 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 148465029 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚4999743538196480 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 148465029 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16369816700928 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 299169264 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚49276947 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 299159285 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚44635878 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚4999743512768512 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚16341807393249 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36633600 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 _step‚1432 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚5001903381716992 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 150694262 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16368471311360 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 trainer/global_step‚17399 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚5018273224591360 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚5001903407890432 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 150694262 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚277 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16368471311360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚5018273224591360 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 148465023 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚5001903407890432 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚5001903381716992 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes¶ ä-&.large_pool.current‚25427968 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 150694262 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16368471311360 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 trainer/global_step‚17399 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚5018273224591360 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚5001903407890432 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 150694262 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚277 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16368471311360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚5018273224591360 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 148465023 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚5001903407890432 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404724736 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚272 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚5018271853028352 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 148465023 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 $ _timestamp‚1.784318629379504e+09 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚49276675 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1371636471111168 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚5018271853028352 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚93912548 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚21112436140544 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 _runtime‚28813.596955598 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 150704235 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 148465029 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚4999743538196480 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 148465029 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16369816700928 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 299169264 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚49276947 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 299159285 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚44635878 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚4999743512768512 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚16341807393249 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36633600 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 _step‚1432 JDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 trainer/global_step‚17399 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16369396346880 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1350601558011904 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 _runtime‚28814.984289636 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1350601926103040 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 299185962 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚257 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404717568 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1371715147013632 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚93917842 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 150702740 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 299175969 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 299175969 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16341386259173 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚5002181382447104 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚93918104 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 148473229 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚5002181408620544 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚5016364150834263 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 299185962 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 % _timestamp‚1.7843186307669194e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 150702740 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚5018552150364160 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚49279683 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚117536“¤)'ŒD£N T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚5016362780805861 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16342730859607 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 299185962 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚257 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 404717568 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1371715147013632 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚93917842 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 150702740 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 299175969 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 299175969 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16341386259173 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚5002181382447104 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚93918104 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 148473229 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚5002181408620544 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚5016364150834263 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 299185962 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 % _timestamp‚1.7843186307669194e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 150702740 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚5018552150364160 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚49279683 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 QDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 trainer/global_step‚17399 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16369396346880 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1350601558011904 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 _runtime‚28814.984289636 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1350601926103040 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 148875323 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚16388044506357 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 300005316 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚44763023 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 _step‚1436 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚256 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 299995337 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 151129987 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚49414765 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚5015803444396032 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 151120014 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1375570627018240 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚28893.238711485 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚5032218205464576 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 148875329 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚21171866938880 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 151129987 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 % _timestamp‚1.7843187090213153e+09 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚21171832402432 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚44763028 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚5013637627105280 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚49415021 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚5015803418222592 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚94178049 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1354398760079360 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1354398391988224 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚5013637601677312 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚5032218205464576 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1375570224390656 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚5030024301583347 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16386699906035 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚5032218205464576 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1375570224390656 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚5030024301583347 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16386699906035 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 148875323 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚16388044506357 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 300005316 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚44763023 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 _step‚1436 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚256 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 299995337 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 151129987 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚49414765 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚5015803444396032 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 151120014 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1375570627018240 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚28893.238711485 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚5032218205464576 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 148875329 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚21171866938880 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 151129987 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820œ úŽ4¢D/ _wandbruntime‚28894  _runtime‚28894B˜Ÿs’£DžN JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 148883529 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚5030303106827769 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚5032498502800384 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚249 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 300012021 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 151138479 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚94183287 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1354475878395904 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16388967972715 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 148883529 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 % _timestamp‚1.7843187104071293e+09 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚1437 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 148883535 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚254 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚94183033 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚5013915483455488 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 151128492 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 151128492 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚49417716 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16387623372281 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 148883535 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚5032498502800384 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚5032497131230208 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1375649312008704 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1375648909388288 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 300022014 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚5016081418952704 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 300012021 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚21173065521664 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 300022014 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚16417057674240 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚21173030992384 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚28894.624510014 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16415712277504 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚5013915508883456 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1354476246487040 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚5016081445126144 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 148883529 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 % _timestamp‚1.7843187104071293e+09 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚1437 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 148883535 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚254 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚94183033 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚5013915483455488 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 151128492 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 151128492 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚49417716 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16387623372281 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 148883535 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚5032498502800384 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚5032497131230208 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1375649312008704 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1375648909388288 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 300022014 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚5016081418952704 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 300012021 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚21173065521664 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 300022014 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚16417057674240 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚21173030992384 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚28894.624510014 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16415712277504 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚5013915508883456 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1354476246487040 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚5016081445126144 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1354475878395904 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16388967972715 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 )*ª•4¥D/ _wandbruntime‚28894  _runtime‚28894­•ýŸ¦DŽ 1loss/SRqqt1/detection_loss‚0.4053581850385054 0loss/FBt2/detection_loss‚0.13681481477260354 1loss/FRt2/assignment_loss‚0.06344455838514682 3loss/SRqqt1/assignment_loss‚0.05029585706433596 3loss/SRqqt2/assignment_loss‚0.10750565092424386 /loss/FRt1/detection_loss‚0.4164199733952083 1loss/SRqqt2/detection_loss‚0.4202869721161511 trainer/global_step‚17449 0loss/FRt1/assignment_loss‚0.7945120633788136 3loss/FBt2/assignment_loss‚4.197500668172798e-08 /loss/FBt1/detection_loss‚0.1521817701353897 _runtime‚28894.708816029 3loss/FBt1/assignment_loss‚0.0032422051640524944 /loss/FRt2/detection_loss‚1.1201566078651548 &loss/total_loss‚3.6702187002146123 epoch‚43 % _timestamp‚1.7843187104918482e+09 _step‚1438ŸåSc§D“ 3loss/FBt1/assignment_loss‚0.0032422051640524944 /loss/FRt2/detection_loss‚1.1201566078651548 &loss/total_loss‚3.6702187002146123 epoch‚43 % _timestamp‚1.7843187104918482e+09 _step‚1438 1loss/SRqqt1/detection_loss‚0.4053581850385054 0loss/FBt2/detection_loss‚0.13681481477260354 1loss/FRt2/assignment_loss‚0.06344455838514682 3loss/SRqqt1/assignment_loss‚0.05029585706433596 3loss/SRqqt2/assignment_loss‚0.10750565092424386 /loss/FRt1/detection_loss‚0.4164199733952083 1loss/SRqqt2/detection_loss‚0.4202869721161511 trainer/global_step‚17449 0loss/FRt1/assignment_loss‚0.7945120633788136 3loss/FBt2/assignment_loss‚4.197500668172798e-08 /loss/FBt1/detection_loss‚0.1521817701353897 _runtime‚28894.708816029ž gø55œ¨D‚(:‘ €–êÒ퓦÷ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚336.465' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚67.29299999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0á%8H ©D‚(:€ €–êÒÞÛŠú disk.dm-0.in‚ 67052.8828125 disk.dm-0.out‚172775.8671875* proc.memory.percent‚17.105603218078613 cpu‚25.586249922035474$ disk./.usageGB‚574.6755638122559& disk.rbd0.in‚1.759218604441423e+13# disk.nvme0c0n1.in‚ 67052.8828125% disk.nvme0c0n1.out‚172775.8671875# disk.nvme0n1.out‚172775.8671875 disk.rbd1.out‚ 6061.1171875 disk.rbd0.out‚0 disk.drbd2165.in‚0# memory_percent‚81.8077564239502 disk.sda2.in‚0 disk.sda.in‚ 109.4921875 disk.sda.out‚ 1677.2734375 disk.dm-1.in‚ 109.4921875 network.recv‚ 7.195065e+06) disk./.usagePercent‚38.56695927235532 disk.sda3.in‚ 109.4921875( proc.memory.availableMB‚ 2980.6171875# proc.memory.rssMB‚ 2802.58203125 disk.rbd1.in‚ 2.9619140625 disk.dm-1.out‚ 1677.2734375 disk.sda1.in‚0 disk.drbd2165.out‚0! disk.nvme0n1.in‚ 67052.8828125 proc.cpu.threads‚17 network.sent‚ 3.3806234e+07 disk.sda2.out‚0 disk.sda1.out‚0 disk.sda3.out‚ 1677.2734375÷õžªD‚(:“ –êÒ“í ÷ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚264.23' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚52.846000000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚05ÙÚ«D‚(: –êÒè—ªù disk.sda1.out‚0 disk.sda.out‚ 1677.57421875& disk.rbd0.in‚1.759218604441423e+13" disk.nvme0n1.in‚67053.71484375 disk.rbd1.in‚ 2.9619140625 disk.rbd1.out‚ 6231.4296875# memory_percent‚81.8077802658081 disk.sda3.in‚ 109.4921875 disk.sda3.out‚ 1677.57421875 cpu‚25.58620659646192 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-0.in‚67053.71484375# disk.dm-0.out‚172784.3681640625 disk.dm-1.out‚ 1677.57421875$ disk.nvme0c0n1.in‚67053.71484375 proc.cpu.threads‚17 network.recv‚ 7.200488e+06 disk.sda.in‚ 109.4921875( disk.nvme0c0n1.out‚172784.3681640625 network.sent‚ 3.3869532e+07( disk./.usagePercent‚38.5669651605286 disk.dm-1.in‚ 109.4921875 disk.drbd2165.in‚0& disk.nvme0n1.out‚172784.3681640625) proc.memory.availableMB‚ 2980.61328125 disk.sda1.in‚0# disk./.usageGB‚574.675651550293 disk.rbd0.out‚0 disk.drbd2165.out‚0# proc.memory.rssMB‚ 2802.58203125* proc.memory.percent‚17.105603218078613Ö°žŸ¬D‚(:” ž–êÒ…£ëö gpu.0.gpu‚34.0 gpu.0.memory‚11, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚37.0 gpu.0.powerWatts‚111.112' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.222399999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0[Yå²­D‚(: ž–êÒ¸ìîø disk.sda3.out‚ 1677.9140625 disk.drbd2165.in‚0# memory_percent‚81.8077802658081* proc.memory.percent‚17.105603218078613$ disk.dm-0.out‚172792.05126953125 disk.sda3.in‚ 109.4921875) proc.memory.availableMB‚ 2980.61328125 disk.sda.in‚ 109.4921875 disk.sda.out‚ 1677.9140625) disk./.usagePercent‚38.56696311246833 disk.rbd1.in‚ 2.9619140625 disk.rbd1.out‚ 6352.4921875" disk.nvme0n1.in‚67053.71484375 disk.sda2.out‚0 disk.sda1.in‚0 disk.drbd2165.out‚0$ disk./.usageGB‚574.6756210327148# proc.memory.rssMB‚ 2802.58203125 proc.cpu.threads‚17& disk.rbd0.in‚1.759218604441423e+13 disk.rbd0.out‚0 disk.sda1.out‚0) disk.nvme0c0n1.out‚172792.05126953125 disk.dm-1.in‚ 109.4921875 network.sent‚ 3.3874551e+07 disk.sda2.in‚0 cpu‚25.586188696062408 disk.dm-1.out‚ 1677.9140625 network.recv‚ 7.204029e+06' disk.nvme0n1.out‚172792.05126953125 disk.dm-0.in‚67053.71484375$ disk.nvme0c0n1.in‚67053.71484375…*”lž®D‚(:“ ­–êÒô–ƒø gpu.0.gpu‚89.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚286.799' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚57.35979999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ã  y¯D‚(:† ­–êÒÀßµù disk.sda1.in‚0$ disk.nvme0c0n1.in‚67215.12890625" disk.nvme0n1.in‚67215.12890625# disk.nvme0n1.out‚172800.3046875 disk.dm-1.in‚ 109.4921875 disk.rbd0.out‚0 disk.dm-0.out‚172800.3046875 disk.rbd1.out‚ 6453.5703125# memory_percent‚81.8077802658081 proc.cpu.threads‚17$ disk./.usageGB‚574.6754264831543 disk.sda.in‚ 109.4921875 network.recv‚ 7.20736e+06 disk.sda3.out‚ 1678.25390625 network.sent‚ 3.3879625e+07 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.sda2.in‚0 disk.sda2.out‚0# proc.memory.rssMB‚ 2802.58203125* proc.memory.percent‚17.105603218078613 disk.sda3.in‚ 109.4921875% disk.nvme0c0n1.out‚172800.3046875 disk.sda.out‚ 1678.25390625 disk.dm-1.out‚ 1678.25390625 cpu‚25.586125815285058 disk.sda1.out‚0 disk.rbd1.in‚ 2.9619140625& disk.rbd0.in‚1.759218604441423e+13) proc.memory.availableMB‚ 2980.61328125) disk./.usagePercent‚38.56695005608408 disk.dm-0.in‚67215.12890625^ß”°D‚(:‰ ¼–êÒ†âïö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚258.272' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚51.6544 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0WÍs±D‚(:‹ ¼–êÒ¾›Œù# disk./.usageGB‚574.675609588623 proc.cpu.threads‚17 disk.sda.out‚ 1678.3984375 disk.dm-1.out‚ 1678.3984375 disk.dm-0.in‚67231.57421875 disk.sda1.in‚0# proc.memory.rssMB‚ 2802.58203125* proc.memory.percent‚17.105603218078613 disk.sda2.in‚0$ memory_percent‚81.80394172668457( proc.memory.availableMB‚ 2981.2421875) disk./.usagePercent‚38.56696234444573& disk.nvme0n1.out‚172805.8779296875 disk.rbd1.out‚ 6497.8046875 disk.dm-1.in‚ 109.4921875 disk.sda1.out‚0 network.recv‚ 7.211137e+06 disk.rbd1.in‚ 2.9619140625 disk.rbd0.out‚0( disk.nvme0c0n1.out‚172805.8779296875& disk.rbd0.in‚1.759218604441423e+13 disk.sda3.out‚ 1678.3984375 cpu‚25.586116334752163 network.sent‚ 3.388466e+07$ disk.nvme0c0n1.in‚67231.57421875 disk.sda3.in‚ 109.4921875 disk.drbd2165.in‚0" disk.nvme0n1.in‚67231.57421875 disk.sda2.out‚0 disk.sda.in‚ 109.4921875# disk.dm-0.out‚172805.8779296875 disk.drbd2165.out‚0þ9v£4²D/ _wandbruntime‚28973  _runtime‚28973»€é©Ì³DÆ ' lr-AdamW/pg1‚2.7424722693966547e-05 ' lr-AdamW/pg2‚2.7424722693966547e-05 trainer/global_step‚17499 % _timestamp‚1.7843187892140317e+09 _runtime‚28973.434067884 _step‚1439dâðÑ´DË _step‚1439 ' lr-AdamW/pg1‚2.7424722693966547e-05 ' lr-AdamW/pg2‚2.7424722693966547e-05 trainer/global_step‚17499 % _timestamp‚1.7843187892140317e+09 _runtime‚28973.434067884Ÿ Äé]¸4µD/ _wandbruntime‚28973  _runtime‚28973º4Ýšã¶D”P VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚5027531690586112 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16432937019143 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚5029703480901632 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 300841368 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 149285623 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚21231637606912 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚5027531716014080 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 % _timestamp‚1.7843187892215407e+09 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚49552607 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1358273080463360 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16462448562176 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚5029703480901632 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚262 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 151555739 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚94442785 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 trainer/global_step‚17499 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 300831389 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚5046165929463808 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 149285629 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚49552345 _step‚1440 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 151545766 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚5029703454728192 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16461103172608 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚5043965997633545 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16462448562176 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16461103172608 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚5046164557900800 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0ë\êá(·D™P bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1358272712372224 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1379504718070272 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚94442518 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚44890173 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚267 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚5046164557900800 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1379504315442688 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚5043964627605255 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 _runtime‚28973.438909664 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚5046164557900800 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚5027531690586112 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16432937019143 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚5029703480901632 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 300841368 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 149285623 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚21231637606912 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚5027531716014080 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 % _timestamp‚1.7843187892215407e+09 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚49552607 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1358273080463360 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16462448562176 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚5029703480901632 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚262 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 151555739 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚94442785 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 trainer/global_step‚17499 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 300831389 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚5046165929463808 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 149285629 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚49552345 _step‚1440 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 151545766 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚5029703454728192 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16461103172608 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚5043965997633545 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16462448562176 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16461103172608 JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚94447785 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚44892716 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16462028208128 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1379582991600128 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚21232827349504 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7843187906242633e+09 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 300848073 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚5029981455458304 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚16463373604864 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 151554244 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 300858066 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚5027809597792256 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16433860485389 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚44892721 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚94448040 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 149293835 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚5029981481631744 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 149293829 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16462028208128 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚49555069 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 149293829 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 151554244 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚5046444855236608 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 trainer/global_step‚17499 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚250 _step‚1441 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚255 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚5044243432849677 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0θƒŠ/$ºD£N trainer/global_step‚17499 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚250 _step‚1441 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚255 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚5044243432849677 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚5046444855236608 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 _runtime‚28974.841682257 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚16463373604864 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 151564231 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚5029981481631744 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚5029981455458304 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚49555319 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚21232792820224 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 300848073 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚94447785 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚44892716 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16462028208128 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1379582991600128 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚21232827349504 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7843187906242633e+09 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 300848073 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚5029981455458304 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚16463373604864 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 151554244 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 300858066 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚5027809597792256 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16433860485389 QDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚44892721 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚94448040 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 149293835 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚5029981481631744 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 149293829 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16462028208128 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚49555069 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 149293829 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 151554244 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚5046444855236608 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0¡ KZ‰y4»D/ _wandbruntime‚28974  _runtime‚28974Èážh¼D 4loss/FBt1/assignment_loss‚1.6235656211449622e-06 0loss/FBt1/detection_loss‚0.17033403820994678 epoch‚43 2loss/FRt2/assignment_loss‚0.055982671233520874 3loss/FBt2/assignment_loss‚2.020496312633117e-09 /loss/FRt2/detection_loss‚1.0722673032981183 1loss/SRqqt2/detection_loss‚0.4075947406407858 %loss/total_loss‚3.627288979656012 _runtime‚28974.919335972 3loss/SRqqt1/assignment_loss‚0.09350919362445395 3loss/SRqqt2/assignment_loss‚0.11304629430663224 /loss/FRt1/detection_loss‚0.4170967868440168 0loss/FRt1/assignment_loss‚0.8244638310663003 1loss/SRqqt1/detection_loss‚0.4054094255364274 0loss/FBt2/detection_loss‚0.06758306930969211 trainer/global_step‚17499 % _timestamp‚1.7843187907021875e+09 _step‚1442¦¡Þ½D• 2loss/FRt2/assignment_loss‚0.055982671233520874 3loss/FBt2/assignment_loss‚2.020496312633117e-09 /loss/FRt2/detection_loss‚1.0722673032981183 1loss/SRqqt2/detection_loss‚0.4075947406407858 %loss/total_loss‚3.627288979656012 _runtime‚28974.919335972 3loss/SRqqt1/assignment_loss‚0.09350919362445395 3loss/SRqqt2/assignment_loss‚0.11304629430663224 /loss/FRt1/detection_loss‚0.4170967868440168 0loss/FRt1/assignment_loss‚0.8244638310663003 1loss/SRqqt1/detection_loss‚0.4054094255364274 0loss/FBt2/detection_loss‚0.06758306930969211 trainer/global_step‚17499 % _timestamp‚1.7843187907021875e+09 _step‚1442 4loss/FBt1/assignment_loss‚1.6235656211449622e-06 0loss/FBt1/detection_loss‚0.17033403820994678 epoch‚43¢ 0m™Ÿ¾D‚(:” Ë–êÒþž÷ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚38.0 gpu.0.powerWatts‚110.823' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.164599999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚09OÀ¿D‚(:‹ Ë–êÒû¶ø# proc.memory.rssMB‚ 2802.58203125 cpu‚25.586107504174542) proc.memory.availableMB‚ 2980.94140625) disk./.usagePercent‚38.56707114764778$ disk./.usageGB‚574.6772308349609 disk.sda1.in‚0 disk.sda3.out‚ 1679.234375 proc.cpu.threads‚17 disk.sda2.in‚0) disk.nvme0c0n1.out‚172819.82666015625' disk.nvme0n1.out‚172819.82666015625 disk.rbd0.out‚0$ disk.dm-0.out‚172819.82666015625 disk.drbd2165.out‚0 disk.sda3.in‚ 109.4921875 disk.drbd2165.in‚0! disk.nvme0n1.in‚ 67232.4296875 network.sent‚ 3.3889719e+07 network.recv‚ 7.214705e+06 disk.sda.in‚ 109.4921875 disk.sda.out‚ 1679.234375 disk.sda1.out‚0# disk.nvme0c0n1.in‚ 67232.4296875 disk.dm-1.out‚ 1679.234375 disk.dm-0.in‚ 67232.4296875$ memory_percent‚81.80577754974365* proc.memory.percent‚17.105603218078613 disk.sda2.out‚0 disk.rbd1.out‚ 6583.9140625 disk.dm-1.in‚ 109.4921875& disk.rbd0.in‚1.759218604441423e+13 disk.rbd1.in‚ 2.96191406251UÞ“ÀD‚(:ˆ Ú–êÒÑÍ›÷ gpu.0.gpu‚23.0 gpu.0.memory‚7, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚336.177' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚67.2354 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Npì” ÁD‚(:ÿ Ú–êÒö«ø disk.sda.out‚ 1682.03125# proc.memory.rssMB‚ 2802.58203125 network.recv‚ 7.219906e+06& disk.rbd0.in‚1.759218604441423e+13$ disk./.usageGB‚575.4537696838379' disk.nvme0c0n1.out‚172874.826171875 disk.sda2.in‚0 disk.rbd1.in‚ 2.9619140625 disk.dm-1.out‚ 1682.03125# memory_percent‚81.8065881729126 cpu‚25.586097608884337 disk.sda3.in‚ 109.4921875 disk.sda3.out‚ 1682.03125 disk.sda1.in‚0 proc.cpu.threads‚17 disk.drbd2165.out‚0 disk.rbd1.out‚ 6644.7734375 disk.sda.in‚ 109.4921875 disk.dm-1.in‚ 109.4921875 disk.dm-0.in‚ 67232.7890625 disk.sda1.out‚0! disk.nvme0n1.in‚ 67232.7890625 disk.sda2.out‚0" disk.dm-0.out‚172874.826171875 network.sent‚ 3.395422e+07# disk.nvme0c0n1.in‚ 67232.7890625% disk.nvme0n1.out‚172874.826171875* proc.memory.percent‚17.105603218078613( disk./.usagePercent‚38.6191853213554 disk.drbd2165.in‚0 disk.rbd0.out‚0) proc.memory.availableMB‚ 2980.80859375|¨‹žÂD‚(:“ é–êÒϦ÷ gpu.0.gpu‚99.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚295.687' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚59.13740000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0³;3 ÃD‚(:þ é–êÒò¼Ùù disk.drbd2165.out‚0" disk.dm-0.out‚172881.529296875 disk.sda3.out‚ 1684.09765625* proc.memory.percent‚17.105603218078613 disk.sda3.in‚ 109.4921875 network.recv‚ 7.22337e+06 disk.rbd1.in‚ 2.9619140625 disk.rbd0.out‚0 disk.dm-0.in‚ 67263.875 disk.sda1.in‚0 cpu‚25.586085073512436 disk.sda1.out‚0* disk./.usagePercent‚38.617366643832156$ disk./.usageGB‚575.4266700744629 disk.sda2.in‚0 network.sent‚ 3.3959376e+07( proc.memory.availableMB‚ 2980.8984375# proc.memory.rssMB‚ 2802.58203125 proc.cpu.threads‚17 disk.drbd2165.in‚0 disk.nvme0c0n1.in‚ 67263.875 disk.sda.in‚ 109.4921875 disk.dm-1.out‚ 1684.09765625$ memory_percent‚81.80603981018066 disk.nvme0n1.in‚ 67263.875 disk.sda.out‚ 1684.09765625 disk.dm-1.in‚ 109.4921875& disk.rbd0.in‚1.759218604441423e+13' disk.nvme0c0n1.out‚172881.529296875% disk.nvme0n1.out‚172881.529296875 disk.sda2.out‚0 disk.rbd1.out‚ 6726.8828125v5´žÄD‚(:“ ø–êÒÛéÖö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚269.905' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚53.98099999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‘ÅD‚(:‹ ø–êÒä€Öø' disk.nvme0n1.out‚173684.22412109375$ disk.dm-0.out‚173684.22412109375 network.sent‚ 3.3963108e+07 network.recv‚ 7.226769e+06 disk.dm-1.in‚ 109.4921875& disk.rbd0.in‚1.759218604441423e+13 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd0.out‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd1.in‚ 2.9619140625 disk.rbd1.out‚ 6871.9140625# memory_percent‚81.8075180053711 disk.sda3.out‚ 1686.53515625 disk.drbd2165.in‚0 cpu‚25.586042506960908" disk.nvme0c0n1.in‚ 67396.640625 disk.dm-0.in‚ 67396.640625$ disk./.usageGB‚574.6794853210449 disk.sda.in‚ 109.4921875 disk.sda3.in‚ 109.4921875# proc.memory.rssMB‚ 2802.58203125) disk.nvme0c0n1.out‚173684.22412109375 disk.nvme0n1.in‚ 67396.640625& proc.memory.availableMB‚ 2980.65625 proc.cpu.threads‚17* disk./.usagePercent‚38.567222448100516 disk.dm-1.out‚ 1686.53515625* proc.memory.percent‚17.105603218078613 disk.sda.out‚ 1686.53515625 disk.drbd2165.out‚04v¾“ÆD‚(:ˆ ‡—êÒŠ†âø gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚40.0 gpu.0.powerWatts‚175.085' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚35.017 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¤]gÇD‚(: ‡—êÒµ„Øù disk.drbd2165.in‚0 disk.dm-1.in‚ 109.4921875 disk.dm-1.out‚ 1687.5703125 disk.sda1.in‚0) disk.nvme0c0n1.out‚173692.32861328125 disk.rbd1.in‚ 2.9619140625* proc.memory.percent‚17.105603218078613) disk./.usagePercent‚38.56715537412654 disk.dm-0.in‚67397.41015625$ disk.dm-0.out‚173692.32861328125$ disk.nvme0c0n1.in‚67397.41015625 disk.rbd1.out‚ 6988.6015625# memory_percent‚81.8075180053711& proc.memory.availableMB‚ 2980.65625 disk.rbd0.out‚0 disk.sda2.in‚0 disk.sda.out‚ 1687.5703125 disk.sda3.in‚ 109.4921875 disk.sda2.out‚0# proc.memory.rssMB‚ 2802.58203125 cpu‚25.586013432553674$ disk./.usageGB‚574.6784858703613 disk.sda1.out‚0' disk.nvme0n1.out‚173692.32861328125 disk.sda.in‚ 109.4921875 network.recv‚ 7.230191e+06 proc.cpu.threads‚17 network.sent‚ 3.3968169e+07& disk.rbd0.in‚1.759218604441423e+13 disk.sda3.out‚ 1687.5703125 disk.drbd2165.out‚0" disk.nvme0n1.in‚67397.41015625cv“$4ÈD/ _wandbruntime‚29054  _runtime‚29054ã+Â1ÌÉDÆ _step‚1443 ' lr-AdamW/pg1‚2.6353136388025785e-05 ' lr-AdamW/pg2‚2.6353136388025785e-05 trainer/global_step‚17549 % _timestamp‚1.7843188705612512e+09 _runtime‚29054.778426776ÞÐÓ^ÑÊDË ' lr-AdamW/pg2‚2.6353136388025785e-05 trainer/global_step‚17549 % _timestamp‚1.7843188705612512e+09 _runtime‚29054.778426776 _step‚1443 ' lr-AdamW/pg1‚2.6353136388025785e-05£ —d¡¡4ËD/ _wandbruntime‚29054  _runtime‚29054 $ F(ÌD”P G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16479174132251 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 149695929 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 149695923 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚5057904953627163 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚5041425779494912 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚45017328 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚21291442310656 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚21291407774208 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚5043603517407232 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16508764492800 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚49689985 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 151981491 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 149695929 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚16480518732573 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚1444 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚5060110910337024 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16507419103232 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚5043603517407232 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16507419103232 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚5060110910337024 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚5060112281900032 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚5043603491233792 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚5043603491233792 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚5060112281900032 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚255 trainer/global_step‚17549 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚260 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 151971518 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 151971518 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7843188705624015e+09 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1362147032756224 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1383438440530432 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008ÍfŒ (ÍD™P QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 151981491 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 149695929 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚16480518732573 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚1444 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚5060110910337024 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16507419103232 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚5043603517407232 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16507419103232 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚5060110910337024 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚5060112281900032 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚5043603491233792 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚5043603491233792 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚5060112281900032 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚255 trainer/global_step‚17549 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚260 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 151971518 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 151971518 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7843188705624015e+09 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1362147032756224 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1383438440530432 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16479174132251 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 149695929 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 149695923 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚5057904953627163 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚5041425779494912 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚45017328 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚21291442310656 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚21291407774208 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚5043603517407232 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16508764492800 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚49689985 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008¤ ØH€À’ÎD‚(:‡ –—êÒŠ‹Ëù gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚334.442' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚66.8884 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Bž#ƒÏD‚(:„ –—êÒ½áÎú disk.sda3.in‚ 109.4921875 disk.dm-1.out‚1689.875 proc.cpu.threads‚17 disk.drbd2165.in‚0 network.recv‚ 7.233777e+06 disk.dm-0.in‚67397.41015625 disk.sda3.out‚1689.875) disk.nvme0c0n1.out‚173710.24267578125) disk./.usagePercent‚38.56565696202863 disk.sda2.out‚0 disk.sda.in‚ 109.4921875 disk.sda1.in‚0 disk.dm-1.in‚ 109.4921875 disk.rbd0.out‚0 disk.drbd2165.out‚0' disk.nvme0n1.out‚173710.24267578125 disk.sda.out‚1689.875 cpu‚25.58601037047251$ memory_percent‚81.80596828460693 network.sent‚ 3.3972664e+07 disk.sda2.in‚0 disk.rbd1.out‚ 6999.6015625& disk.rbd0.in‚1.759218604441423e+13$ disk.nvme0c0n1.in‚67397.41015625" disk.nvme0n1.in‚67397.41015625 disk.rbd1.in‚ 2.9619140625) proc.memory.availableMB‚ 2980.91015625# proc.memory.rssMB‚ 2802.58203125$ disk./.usageGB‚574.6561584472656$ disk.dm-0.out‚173710.24267578125 disk.sda1.out‚0* proc.memory.percent‚17.105603218078613”ïÜ4ÐD/ _wandbruntime‚29056  _runtime‚290568üŽÑDžN OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚94712828 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1362224519163904 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚94712578 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚5060391207672832 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚5043881491963904 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚5058185128899987 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚16509689535488 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 149704135 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16481442198931 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚245 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚250 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 301694118 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 151979996 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚5060391207672832 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚16509689535488 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16508344138752 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 151979996 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚5041703661273088 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚49692957 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚29056.181255772 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 149704135 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 149704129 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚49692712 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚21292599179776 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚5041703686701056 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 149704129 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16508344138752 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1383517118343680 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1362224887255040 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 trainer/global_step‚17549 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 301684125 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚5043881518137344 % _timestamp‚1.7843188719638674e+09 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 151989983 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚5043881518137344 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 301684125æÁ˜)'ÒD£N QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 151989983 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚5043881518137344 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 301684125 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚94712828 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1362224519163904 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚94712578 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚5060391207672832 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚5043881491963904 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚5058185128899987 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚16509689535488 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 149704135 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16481442198931 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚245 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚250 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 301694118 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 151979996 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚5060391207672832 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚16509689535488 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16508344138752 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 151979996 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚5041703661273088 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚49692957 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚29056.181255772 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 149704135 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 149704129 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚49692712 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚21292599179776 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚5041703686701056 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 149704129 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16508344138752 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1383517118343680 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1362224887255040 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 trainer/global_step‚17549 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 301684125 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚5043881518137344 % _timestamp‚1.7843188719638674e+09¥ à†ÝY4ÓD/ _wandbruntime‚29056  _runtime‚29056焺ÔD &loss/total_loss‚3.5771938838728614 _step‚1446 1loss/FRt2/assignment_loss‚0.04533683951130978 3loss/SRqqt2/assignment_loss‚0.08371506186043662 2loss/FBt2/assignment_loss‚9.54554692846388e-07 1loss/SRqqt2/detection_loss‚0.4366514779622786 epoch‚43 _runtime‚29056.260658169 0loss/FRt1/assignment_loss‚0.8261653417975058 4loss/SRqqt1/assignment_loss‚0.026259881345248426 1loss/SRqqt1/detection_loss‚0.3901476615759494 % _timestamp‚1.7843188720434515e+09 3loss/FBt1/assignment_loss‚0.0003091531623649711 /loss/FRt2/detection_loss‚1.2347351784888714 0loss/FBt1/detection_loss‚0.06335942161578999 0loss/FBt2/detection_loss‚0.03926583673528547 trainer/global_step‚17549 /loss/FRt1/detection_loss‚0.4312470752631283JŠ–ÕD” _step‚1446 1loss/FRt2/assignment_loss‚0.04533683951130978 3loss/SRqqt2/assignment_loss‚0.08371506186043662 2loss/FBt2/assignment_loss‚9.54554692846388e-07 1loss/SRqqt2/detection_loss‚0.4366514779622786 epoch‚43 _runtime‚29056.260658169 0loss/FRt1/assignment_loss‚0.8261653417975058 4loss/SRqqt1/assignment_loss‚0.026259881345248426 1loss/SRqqt1/detection_loss‚0.3901476615759494 % _timestamp‚1.7843188720434515e+09 3loss/FBt1/assignment_loss‚0.0003091531623649711 /loss/FRt2/detection_loss‚1.2347351784888714 0loss/FBt1/detection_loss‚0.06335942161578999 0loss/FBt2/detection_loss‚0.03926583673528547 trainer/global_step‚17549 /loss/FRt1/detection_loss‚0.4312470752631283 &loss/total_loss‚3.5771938838728614¦ :öÇ”ÖD‚(:‰ ¥—êÒ÷‘üö gpu.0.gpu‚50.0 gpu.0.memory‚16, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚161.536' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚32.3072 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚04Eü×D‚(:‰ ¥—êÒÄžÅù disk.sda.out‚ 1692.55859375# proc.memory.rssMB‚ 2802.58203125* proc.memory.percent‚17.105603218078613 proc.cpu.threads‚17 disk.drbd2165.out‚0 network.recv‚ 7.238751e+06) disk./.usagePercent‚38.56584333551356 disk.sda1.in‚0# disk.nvme0c0n1.in‚ 67398.2421875' proc.memory.availableMB‚ 2980.640625 cpu‚25.5860023619343 disk.dm-1.out‚ 1692.55859375 disk.sda2.out‚0 network.sent‚ 3.4025085e+07# disk./.usageGB‚574.658935546875 disk.rbd0.out‚0& disk.rbd0.in‚1.759218604441423e+13( disk.nvme0c0n1.out‚173720.1708984375 disk.sda.in‚ 109.4921875& disk.nvme0n1.out‚173720.1708984375$ memory_percent‚81.80761337280273# disk.dm-0.out‚173720.1708984375 disk.sda1.out‚0 disk.sda3.in‚ 109.4921875 disk.sda3.out‚ 1692.55859375! disk.nvme0n1.in‚ 67398.2421875 disk.rbd1.in‚ 2.9619140625 disk.rbd1.out‚ 7071.5859375 disk.dm-1.in‚ 109.4921875 disk.dm-0.in‚ 67398.2421875 disk.drbd2165.in‚0 disk.sda2.in‚0ïNžØD‚(:“ ´—êÒ–þ‡÷ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚282.255' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚56.45099999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0µëÎýÙD‚(:… ´—êÒêàÌù disk.sda2.out‚0 disk.rbd1.out‚ 7156.9140625# memory_percent‚81.8096399307251) proc.memory.availableMB‚ 2980.30859375 cpu‚25.585940739492425& disk.rbd0.in‚1.759218604441423e+13 disk.rbd0.out‚0$ disk.dm-0.out‚173737.00244140625) disk.nvme0c0n1.out‚173737.00244140625$ disk./.usageGB‚574.6523056030273 disk.drbd2165.in‚0 proc.cpu.threads‚17 disk.sda1.in‚0# proc.memory.rssMB‚ 2802.58203125* disk./.usagePercent‚38.565398394419056 disk.dm-1.out‚ 1693.65234375 disk.drbd2165.out‚0* proc.memory.percent‚17.105603218078613 disk.dm-1.in‚ 109.4921875 disk.sda3.out‚ 1693.65234375 disk.nvme0c0n1.in‚ 67541.875 disk.sda.in‚ 109.4921875 network.recv‚ 7.244253e+06 disk.dm-0.in‚ 67541.875 disk.sda1.out‚0 disk.nvme0n1.in‚ 67541.875' disk.nvme0n1.out‚173737.00244140625 disk.sda2.in‚0 disk.rbd1.in‚ 2.9619140625 disk.sda.out‚ 1693.65234375 network.sent‚ 3.4078694e+07 disk.sda3.in‚ 109.4921875‰o‰ëžÚD‚(:“ ×êÒØ™‘ø gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚42.0 gpu.0.powerWatts‚294.095' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚58.81900000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‡·äÛD‚(:Ž ×êÒŠÞÒù disk.drbd2165.in‚0# proc.memory.rssMB‚ 2802.58203125 disk.drbd2165.out‚0$ disk.nvme0c0n1.in‚67541.87890625 proc.cpu.threads‚17& disk.rbd0.in‚1.759218604441423e+13" disk.dm-0.out‚173744.150390625 disk.sda1.in‚0 disk.sda2.in‚0 disk.dm-1.in‚ 109.4921875 disk.rbd0.out‚0* disk./.usagePercent‚38.565215605039604' disk.nvme0c0n1.out‚173744.150390625" disk.nvme0n1.in‚67541.87890625) proc.memory.availableMB‚ 2980.27734375* proc.memory.percent‚17.105603218078613 network.recv‚ 7.247525e+06 disk.dm-1.out‚ 1696.19921875 disk.dm-0.in‚67541.87890625 disk.sda3.in‚ 109.4921875 disk.sda3.out‚ 1696.19921875 disk.rbd1.in‚ 2.9619140625$ memory_percent‚81.80983066558838 cpu‚25.58588782082028 network.sent‚ 3.4083868e+07 disk.sda.out‚ 1696.19921875% disk.nvme0n1.out‚173744.150390625 disk.sda2.out‚0 disk.rbd1.out‚ 7200.0703125$ disk./.usageGB‚574.6495819091797 disk.sda.in‚ 109.4921875 disk.sda1.out‚0v4†”ÜD‚(:‰ Ò—êÒ˜ïÑö gpu.0.gpu‚38.0 gpu.0.memory‚12, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚334.153' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚66.8306 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÙàDÝD‚(:Ž Ò—êÒýŽÉø network.recv‚ 7.251183e+06& proc.memory.availableMB‚ 2980.28125$ disk.dm-0.out‚173757.77392578125) disk.nvme0c0n1.out‚173757.77392578125! disk.nvme0n1.in‚ 67542.7109375 disk.sda2.in‚0 disk.rbd1.out‚ 7370.6328125# proc.memory.rssMB‚ 2802.58203125 cpu‚25.585974262471936 proc.cpu.threads‚17) disk./.usagePercent‚38.56523250153687 disk.dm-1.out‚ 1696.69140625 disk.sda3.in‚ 109.4921875 disk.drbd2165.in‚0 disk.rbd1.in‚ 2.9619140625$ memory_percent‚81.80980682373047 disk.dm-1.in‚ 109.4921875 disk.dm-0.in‚ 67542.7109375 disk.sda3.out‚ 1696.69140625 disk.drbd2165.out‚0$ disk./.usageGB‚574.6498336791992 disk.sda.in‚ 109.4921875& disk.rbd0.in‚1.759218604441423e+13# disk.nvme0c0n1.in‚ 67542.7109375 disk.sda2.out‚0 network.sent‚ 3.4088893e+07' disk.nvme0n1.out‚173757.77392578125* proc.memory.percent‚17.105603218078613 disk.sda.out‚ 1696.69140625 disk.rbd0.out‚0 disk.sda1.in‚0 disk.sda1.out‚0ÅH”ÞD‚(:‰ á—êÒàÊèö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚264.538' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.9076 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0·¢5ßD‚(:“ á—êÒ±œ¼ù$ disk.nvme0c0n1.in‚67543.66796875 cpu‚25.586014132654764 disk.sda.out‚ 1697.07421875$ disk.dm-0.out‚173771.33642578125' disk.nvme0n1.out‚173771.33642578125 disk.rbd1.out‚ 7487.0859375 proc.cpu.threads‚17 disk.sda.in‚ 109.4921875* proc.memory.percent‚17.105603218078613 disk.drbd2165.out‚0) disk./.usagePercent‚38.56523326955947 disk.rbd0.out‚0) disk.nvme0c0n1.out‚173771.33642578125 disk.dm-1.out‚ 1697.07421875 disk.dm-0.in‚67543.66796875 disk.sda3.in‚ 109.4921875 disk.sda2.in‚0$ memory_percent‚81.80983066558838 disk.drbd2165.in‚0# disk./.usageGB‚574.649845123291& disk.rbd0.in‚1.759218604441423e+13 disk.sda1.in‚0 disk.sda1.out‚0 network.sent‚ 3.4095333e+07 disk.sda3.out‚ 1697.07421875" disk.nvme0n1.in‚67543.66796875 disk.sda2.out‚0) proc.memory.availableMB‚ 2980.27734375 network.recv‚ 7.254484e+06 disk.rbd1.in‚ 2.9619140625 disk.dm-1.in‚ 109.4921875# proc.memory.rssMB‚ 2802.58203125¥8þ¾4àD/ _wandbruntime‚29135  _runtime‚29135ÂÔ¼ÊáDÄ & lr-AdamW/pg1‚2.530207553599977e-05 & lr-AdamW/pg2‚2.530207553599977e-05 trainer/global_step‚17599 % _timestamp‚1.7843189512169876e+09 _runtime‚29135.434128329 _step‚14478ð™ÏâDÉ & lr-AdamW/pg1‚2.530207553599977e-05 & lr-AdamW/pg2‚2.530207553599977e-05 trainer/global_step‚17599 % _timestamp‚1.7843189512169876e+09 _runtime‚29135.434128329 _step‚1447§ Q*Ì;4ãD/ _wandbruntime‚29135  _runtime‚29135™J¯&å äD“P QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚94972171 _runtime‚29135.435389901 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚5074058634336256 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚5071846649677361 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 152397270 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 trainer/global_step‚17599 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1366021353140224 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 _step‚1448 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚21351284208128 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16555080423424 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16555080423424 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 152407243 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚16526755845681 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚5057503553912832 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16553735033856 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1387373039975936 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 150106229 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 $ _timestamp‚1.784318951218039e+09 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚49827443 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚5055319868403712 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚5057503527739392 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16525411245359 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚5055319893831680 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16553735033856 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 302503493 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚5057503527739392 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 152397270 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚94971916 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚21351318744576 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚5074057262773248 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚5057503553912832 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 150106223 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚5071845279649071 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚250 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 150106229 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 152407243 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚5074057262773248 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚45144473 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1387372637348352 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚5057503527739392 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16525411245359 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚5055319893831680 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16553735033856 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 302503493 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚5057503527739392 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 152397270 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚94971916 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚21351318744576 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚5074057262773248 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚5057503553912832 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 150106223 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚5071845279649071 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚250 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 150106229 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 152407243 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚5074057262773248 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚45144473 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1387372637348352 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 152407243 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚16526755845681 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚5057503553912832 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16553735033856 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1387373039975936 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 150106229 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 $ _timestamp‚1.784318951218039e+09 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚49827443 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚5055319868403712¨ UÈü4æD/ _wandbruntime‚29136  _runtime‚29136%Ùà7#'çDN PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚243 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 302520177 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16527679312039 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚45147016 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 302530170 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 152415735 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1366098839547904 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚16556005466112 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1366099207639040 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚5055597775609856 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚49830176 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚5074337560109056 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 _runtime‚29136.833651856 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 152405748 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16554660069376 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 150114429 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚5074337560109056 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1387451723000320 trainer/global_step‚17599 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 _step‚1449 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚16556005466112 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 150114429 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚248 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 302520177 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚94977440 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚21352480832000 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 152415735 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚21352515361280 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 150114435 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚5074336188538880 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚94977192 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚5072124084893493 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚5055597750181888 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 302530170 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1387451320379904 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚5074337560109056 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1387451723000320 trainer/global_step‚17599 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 _step‚1449 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚16556005466112 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 150114429 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚248 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 302520177 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚94977440 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚21352480832000 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 152415735 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚21352515361280 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 ;4DeviceStatsM6;åôonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 150114435 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚5074336188538880 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚94977192 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚5072124084893493 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚5055597750181888 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 QDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 302530170 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1387451320379904 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16527679312039 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚45147016 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 302530170 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 152415735 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1366098839547904© ²›•q4éD/ _wandbruntime‚29136  _runtime‚29136& ™‘êD 0loss/FBt2/detection_loss‚0.10896497595435449 _runtime‚29136.914015192 3loss/SRqqt2/assignment_loss‚0.08916999729721362 4loss/FBt1/assignment_loss‚1.0361199583387912e-05 2loss/SRqqt1/detection_loss‚0.47156866472758385 1loss/SRqqt2/detection_loss‚0.4543615170478955 0loss/FBt1/detection_loss‚0.19908363182061553 trainer/global_step‚17599 _step‚1450 3loss/SRqqt1/assignment_loss‚0.03403493073233008 /loss/FRt1/detection_loss‚0.4006473076521201 /loss/FRt2/detection_loss‚1.0550526557473352 %loss/total_loss‚3.691365091785637 epoch‚43 % _timestamp‚1.7843189526967978e+09 .loss/FRt1/assignment_loss‚0.82713787903253 2loss/FRt2/assignment_loss‚0.051332775157751497 4loss/FBt2/assignment_loss‚3.9541632352525564e-07:ùÖëD• .loss/FRt1/assignment_loss‚0.82713787903253 2loss/FRt2/assignment_loss‚0.051332775157751497 4loss/FBt2/assignment_loss‚3.9541632352525564e-07 0loss/FBt2/detection_loss‚0.10896497595435449 _runtime‚29136.914015192 3loss/SRqqt2/assignment_loss‚0.08916999729721362 4loss/FBt1/assignment_loss‚1.0361199583387912e-05 2loss/SRqqt1/detection_loss‚0.47156866472758385 1loss/SRqqt2/detection_loss‚0.4543615170478955 0loss/FBt1/detection_loss‚0.19908363182061553 trainer/global_step‚17599 _step‚1450 3loss/SRqqt1/assignment_loss‚0.03403493073233008 /loss/FRt1/detection_loss‚0.4006473076521201 /loss/FRt2/detection_loss‚1.0550526557473352 %loss/total_loss‚3.691365091785637 epoch‚43 % _timestamp‚1.7843189526967978e+09ª ¦AΔìD‚(:‰ ð—êÒÅ…øö gpu.0.gpu‚58.0 gpu.0.memory‚18, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚38.0 gpu.0.powerWatts‚111.642' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.3284 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚079Í_íD‚(: ð—êÒ”ÚÑù disk.dm-1.out‚ 1699.45703125$ disk.nvme0c0n1.in‚67636.62109375$ memory_percent‚81.82408809661865) proc.memory.percent‚17.12169647216797 cpu‚25.58672385210233 network.recv‚ 7.257768e+06 disk.dm-1.in‚ 109.4921875 disk.sda1.out‚0' disk.nvme0n1.out‚173785.86474609375 disk.drbd2165.in‚0" disk.nvme0n1.in‚67636.62109375 disk.rbd1.in‚ 2.9619140625* disk./.usagePercent‚38.568473300912814$ disk./.usageGB‚574.6981239318848 disk.sda.out‚ 1699.45703125 disk.rbd0.out‚0 disk.sda3.in‚ 109.4921875 disk.sda3.out‚ 1699.45703125) disk.nvme0c0n1.out‚173785.86474609375 disk.rbd1.out‚ 7604.7890625 disk.sda.in‚ 109.4921875& disk.rbd0.in‚1.759218604441423e+13 network.sent‚ 3.4099057e+07$ disk.dm-0.out‚173785.86474609375 disk.sda2.in‚0 disk.sda1.in‚0 disk.drbd2165.out‚0 proc.memory.rssMB‚ 2805.21875 proc.cpu.threads‚17 disk.dm-0.in‚67636.62109375 disk.sda2.out‚0) proc.memory.availableMB‚ 2977.94140625Y.³²’îD‚(:‡ ÿ—êÒúµ¹ù gpu.0.gpu‚44.0 gpu.0.memory‚14, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚345.59' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚69.118 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0–9¨ïD‚(:Š ÿ—êÒëʱû disk.sda2.in‚0 disk.sda.out‚ 1700.66796875 disk.dm-1.out‚ 1700.66796875 disk.dm-0.in‚ 67637.453125 disk.nvme0n1.in‚ 67637.453125 disk.rbd1.in‚ 2.9619140625 proc.memory.rssMB‚ 2805.21875) proc.memory.percent‚17.12169647216797 disk.dm-1.in‚ 109.4921875 disk.sda3.in‚ 109.4921875 disk.drbd2165.in‚0 disk.drbd2165.out‚0" disk.nvme0c0n1.in‚ 67637.453125$ memory_percent‚81.82599544525146) proc.memory.availableMB‚ 2977.62890625 proc.cpu.threads‚17* disk./.usagePercent‚38.571344937424385$ disk.dm-0.out‚173809.01708984375' disk.nvme0n1.out‚173809.01708984375 disk.sda2.out‚0$ disk./.usageGB‚574.7409133911133 disk.sda.in‚ 109.4921875 disk.rbd0.out‚0 disk.sda1.out‚0 disk.sda3.out‚ 1700.66796875 network.recv‚ 7.262834e+06& disk.rbd0.in‚1.759218604441423e+13 disk.sda1.in‚0) disk.nvme0c0n1.out‚173809.01708984375 cpu‚25.588021035755926 network.sent‚ 3.416245e+07 disk.rbd1.out‚ 7757.3515625Ó‰)u4ðD/ _wandbruntime‚29164  _runtime‚29164º2™9>ñD³| :"Purity/0FRt2SRqqt0FBt/event_purity‚0.47506260871887207 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4537036716938019 jet/accuracy_5_of_5‚0 % _timestamp‚1.7843189806301007e+09 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.625 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4465472400188446 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9740923047065735 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9966082572937012 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.46125343441963196 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.09503225237131119 jet/accuracy_3_of_5‚1 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.5618206858634949 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 particle/accuracy_2_of_5‚1 particle/accuracy_5_of_5‚1 _runtime‚29164.847924043 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8888888955116272 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.026518747210502625 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9975752830505371 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.05000000074505806 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.10310959815979004 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 *jet/accuracy_1_of_1‚0.2814485728740692 +particle/specificity‚0.8964949250221252 ) Purity/2FRt*SRqqt2FBt/FRt_purity‚0.75 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.1714721918106079 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.12551966309547424 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6545648574829102 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6099014282226562 8"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.680503249168396 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9916297197341919 *jet/accuracy_1_of_2‚0.7285606861114502 /particle/accuracy_3_of_4‚0.9974937438964844 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7142857313156128 8"Purity/1FRt*SRqqt*FBt/event_purity‚0.103104367852211 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.2083333283662796 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9130856394767761 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9579395055770874 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.19422586262226105 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7315381765365601 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7322846055030823 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7546296119689941 (particle/accuracy‚0.8288611173629761 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.26851850748062134 - Purity/0FRt2SRqqt*FBt/FBt_purity‚0.703125 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.1336931437253952 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.8500000238418579 )jet/accuracy_1_of_3‚0.903439462184906 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6775925159454346 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6885745525360107 particle/accuracy_1_of_3‚1 jet/accuracy_2_of_5‚1 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.01041794940829277 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.8991447687149048 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.1321653425693512 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.7881214618682861 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.22197073698043823 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9928159117698669 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9705640077590942 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6326938271522522 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6099014282226562 7"Purity/1FRt*SRqqt2FBt/event_purity‚0.63900226354599 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6835616230964661 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9936507940292358 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.1457381397485733 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9941437840461731 /"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.578125 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.13519035279750824 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.26923078298568726 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6245468258857727 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.006428509950637817 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.2265007644891739 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.5882983207702637 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.09986905008554459 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.11953811347484589 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9888888597488403 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 particle/accuracy_1_of_1‚1 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.5807291269302368 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.1298871785402298 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9841936826705933 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7455267906188965 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3564845025539398 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7188208699226379 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 epoch‚43 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9813519716262817 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6235358119010925 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6120753884315491 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6928879022598267 jet/accuracy_1_of_5‚1 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.7741392850875854 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9966082572937012 8"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.817793071269989 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.7407407760620117 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6536994576454163 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 8"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.817793071269989 particle/accuracy_2_of_4‚1 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7857142686843872 _step‚1451 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5074136853218079 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7546296119689941 particle/accuracy_2_of_3‚1 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.02522539533674717 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.12386812269687653 *loss/val_total_loss‚12.934446800034726 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.6999433040618896 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.01969514787197113 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_3_of_3‚1 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7329950332641602 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9843529462814331 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.2222222238779068 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7598989605903625 7validation_average_jet_accuracy‚0.32400527596473694 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9675565361976624 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 *jet/accuracy_2_of_4‚0.6898382902145386 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9870997667312622 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3467606008052826 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.025204142555594444 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6655051112174988 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.2222222238779068 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.12956158816814423 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.12457907944917679 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5960425734519958 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9906229376792908 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.05000000074505806 +jet/accuracy_2_of_2‚0.11539675295352936 particle/accuracy_1_of_2‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 6 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.712169349193573 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.26851850748062134 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 jet/accuracy_4_of_5‚0 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.20774757862091064 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.02101125940680504 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 8"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.655370831489563 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6879763007164001 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.13342076539993286 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6356443762779236 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.42500001192092896 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.19571156799793243 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7280319333076477 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7236076593399048 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.1420973390340805 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.1300049126148224 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6332865953445435 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9840483665466309 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9852820038795471 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.16759207844734192 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6558898687362671 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7554802298545837 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 +particle/sensitivity‚0.5815834999084473 'particle/f_score‚0.5934165716171265 particle/accuracy_1_of_5‚1 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.14355532824993134 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0035971442703157663 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.11896580457687378 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.26923078298568726 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.5622007250785828 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.7407407760620117 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9913384318351746 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.02597670815885067 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.09666667133569717 /particle/accuracy_4_of_4‚0.9401100873947144 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9773070216178894 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.8461538553237915 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2569190561771393 8"Purity/*FRt*SRqqt1FBt/event_purity‚0.586164653301239 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9868460893630981 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ,jet/accuracy_4_of_4‚0.013899499550461769 particle/accuracy_2_of_2‚1 particle/accuracy_4_of_5‚1 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8401549458503723 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.5486111044883728 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6658906936645508 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.11370202153921127 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 )jet/accuracy_3_of_4‚0.043703593313694 particle/accuracy_1_of_4‚1 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7237340211868286 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.8564814329147339 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9693400263786316 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.1298871785402298 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.1201680600643158 6 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.705895721912384 9"Purity/1FRt1SRqqt*FBt/event_purity‚0.0283015426248312 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.17789433896541595 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9768246412277222 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5149999856948853 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6499772071838379 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.02101125940680504 *jet/accuracy_1_of_4‚0.9537616968154907 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 particle/accuracy_3_of_5‚1 trainer/global_step‚17599 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.14259697496891022 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 +jet/accuracy_3_of_3‚0.04058575630187988 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.21171465516090393 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.024790801107883453 - Purity/0FRt2SRqqt1FBt/FBt_purity‚0.703125 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6685931086540222 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4393920302391052 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.8500000238418579 *jet/accuracy_2_of_3‚0.4705500900745392 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5054793357849121×l—”—òD¸| 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4465472400188446 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9740923047065735 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9966082572937012 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.46125343441963196 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.09503225237131119 jet/accuracy_3_of_5‚1 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.5618206858634949 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 particle/accuracy_2_of_5‚1 particle/accuracy_5_of_5‚1 _runtime‚29164.847924043 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8888888955116272 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.026518747210502625 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9975752830505371 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.05000000074505806 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.10310959815979004 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 *jet/accuracy_1_of_1‚0.2814485728740692 +particle/specificity‚0.8964949250221252 ) Purity/2FRt*SRqqt2FBt/FRt_purity‚0.75 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.1714721918106079 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.12551966309547424 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.6545648574829102 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6099014282226562 8"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.680503249168396 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9916297197341919 *jet/accuracy_1_of_2‚0.7285606861114502 /particle/accuracy_3_of_4‚0.9974937438964844 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7142857313156128 8"Purity/1FRt*SRqqt*FBt/event_purity‚0.103104367852211 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.2083333283662796 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9130856394767761 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9579395055770874 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.19422586262226105 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7315381765365601 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7322846055030823 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7546296119689941 (particle/accuracy‚0.8288611173629761 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.26851850748062134 - Purity/0FRt2SRqqt*FBt/FBt_purity‚0.703125 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.1336931437253952 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.8500000238418579 )jet/accuracy_1_of_3‚0.903439462184906 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6775925159454346 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6885745525360107 particle/accuracy_1_of_3‚1 jet/accuracy_2_of_5‚1 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.01041794940829277 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.8991447687149048 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.1321653425693512 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.7881214618682861 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.22197073698043823 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9928159117698669 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9705640077590942 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6326938271522522 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6099014282226562 7"Purity/1FRt*SRqqt2FBt/event_purity‚0.63900226354599 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6835616230964661 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9936507940292358 7 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.1457381397485733 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9941437840461731 /"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.578125 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.13519035279750824 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.26923078298568726 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6245468258857727 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.006428509950637817 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.2265007644891739 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.5882983207702637 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.09986905008554459 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.11953811347484589 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9888888597488403 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 particle/accuracy_1_of_1‚1 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.5807291269302368 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.1298871785402298 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9841936826705933 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7455267906188965 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3564845025539398 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7188208699226379 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 epoch‚43 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9813519716262817 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6235358119010925 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6120753884315491 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6928879022598267 jet/accuracy_1_of_5‚1 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.7741392850875854 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9966082572937012 8"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.817793071269989 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.7407407760620117 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6536994576454163 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 8"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.817793071269989 particle/accuracy_2_of_4‚1 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7857142686843872 _step‚1451 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5074136853218079 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7546296119689941 particle/accuracy_2_of_3‚1 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.02522539533674717 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.12386812269687653 *loss/val_total_loss‚12.934446800034726 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 9"Purity/1FRt0SRqqt2FBt/event_purity ed¥§‚0.6999433040618896 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.01969514787197113 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_3_of_3‚1 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7329950332641602 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9843529462814331 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9988385438919067 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.2222222238779068 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7598989605903625 7validation_average_jet_accuracy‚0.32400527596473694 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9675565361976624 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 *jet/accuracy_2_of_4‚0.6898382902145386 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9870997667312622 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3467606008052826 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.025204142555594444 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6655051112174988 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.2222222238779068 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.12956158816814423 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.12457907944917679 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5960425734519958 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9906229376792908 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.05000000074505806 +jet/accuracy_2_of_2‚0.11539675295352936 particle/accuracy_1_of_2‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 6 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.712169349193573 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.26851850748062134 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 jet/accuracy_4_of_5‚0 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.20774757862091064 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.02101125940680504 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 8"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.655370831489563 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6879763007164001 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.13342076539993286 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6356443762779236 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.42500001192092896 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.19571156799793243 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7280319333076477 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.7236076593399048 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.1420973390340805 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.1300049126148224 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6332865953445435 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9840483665466309 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9852820038795471 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.16759207844734192 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6558898687362671 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7554802298545837 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 +particle/sensitivity‚0.5815834999084473 'particle/f_score‚0.5934165716171265 particle/accuracy_1_of_5‚1 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.14355532824993134 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0035971442703157663 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.11896580457687378 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.26923078298568726 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.5622007250785828 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.7407407760620117 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9913384318351746 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.02597670815885067 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.09666667133569717 /particle/accuracy_4_of_4‚0.9401100873947144 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9773070216178894 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.8461538553237915 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2569190561771393 8"Purity/*FRt*SRqqt1FBt/event_purity‚0.586164653301239 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9868460893630981 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ,jet/accuracy_4_of_4‚0.013899499550461769 particle/accuracy_2_of_2‚1 particle/accuracy_4_of_5‚1 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8401549458503723 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.5486111044883728 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6658906936645508 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.11370202153921127 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 )jet/accuracy_3_of_4‚0.043703593313694 particle/accuracy_1_of_4‚1 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7237340211868286 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.8564814329147339 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9693400263786316 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.1298871785402298 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.1201680600643158 6 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.705895721912384 9"Purity/1FRt1SRqqt*FBt/event_purity‚0.0283015426248312 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.17789433896541595 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9768246412277222 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5149999856948853 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6499772071838379 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.02101125940680504 *jet/accuracy_1_of_4‚0.9537616968154907 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 particle/accuracy_3_of_5‚1 trainer/global_step‚17599 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.14259697496891022 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 +jet/accuracy_3_of_3‚0.04058575630187988 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.21171465516090393 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.024790801107883453 - Purity/0FRt2SRqqt1FBt/FBt_purity‚0.703125 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6685931086540222 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4393920302391052 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.8500000238418579 *jet/accuracy_2_of_3‚0.4705500900745392 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5054793357849121 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.47506260871887207 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4537036716938019 jet/accuracy_5_of_5‚0 % _timestamp‚1.7843189806301007e+09 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.625« .f“óD‚(:ˆ Ž˜êÒ­Ìóö gpu.0.gpu‚26.0 gpu.0.memory‚8, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚332.173' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚66.4346 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0’ƉôD‚(:ƒ Ž˜êÒϳÌù network.recv‚ 7.265764e+06 disk.sda3.out‚ 1703.66015625% disk.nvme0n1.out‚173870.587890625 disk.rbd1.in‚ 2.9619140625) disk./.usagePercent‚38.56960613425183 disk.dm-0.in‚ 67637.453125" disk.nvme0c0n1.in‚ 67637.453125' disk.nvme0c0n1.out‚173870.587890625$ memory_percent‚81.82547092437744) proc.memory.availableMB‚ 2977.71484375$ disk./.usageGB‚574.7150039672852 disk.nvme0n1.in‚ 67637.453125 disk.sda2.out‚0 disk.sda.in‚ 109.4921875 disk.drbd2165.in‚0 disk.sda2.in‚0 disk.sda.out‚ 1703.66015625 disk.rbd0.out‚0 network.sent‚ 3.4166816e+07" disk.dm-0.out‚173870.587890625 disk.drbd2165.out‚0 disk.dm-1.in‚ 109.4921875) proc.memory.percent‚17.12169647216797 proc.cpu.threads‚17 disk.sda3.in‚ 109.4921875 disk.rbd1.out‚ 7865.1953125 proc.memory.rssMB‚ 2805.21875 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-1.out‚ 1703.66015625& disk.rbd0.in‚1.759218604441423e+13 cpu‚25.58854418723753ì?ÁNžõD‚(:“ ˜êÒß®üö gpu.0.gpu‚99.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚279.79' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚55.958000000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0"l>öD‚(:… ˜êÒôÈ£ù disk.sda.in‚ 109.4921875) proc.memory.availableMB‚ 2977.69921875 proc.cpu.threads‚17$ memory_percent‚81.82556629180908 network.sent‚ 3.4224418e+07* disk./.usagePercent‚38.572457546168195 disk.drbd2165.out‚0$ disk.nvme0c0n1.in‚67638.41015625 network.recv‚ 7.270409e+06" disk.dm-0.out‚173882.923828125 disk.drbd2165.in‚0' disk.nvme0c0n1.out‚173882.923828125% disk.nvme0n1.out‚173882.923828125! disk.rbd1.out‚7969.7724609375 proc.memory.rssMB‚ 2805.21875$ disk./.usageGB‚574.7574920654297 disk.dm-1.in‚ 109.4921875& disk.rbd0.in‚1.759218604441423e+13 disk.sda1.out‚0 disk.sda3.in‚ 109.4921875 disk.sda3.out‚ 1704.90625" disk.nvme0n1.in‚67638.41015625 disk.rbd1.in‚ 2.9658203125 disk.sda2.in‚0 disk.sda.out‚ 1704.90625 disk.dm-1.out‚ 1704.90625 disk.dm-0.in‚67638.41015625 disk.sda2.out‚0 cpu‚25.588601988145395 disk.rbd0.out‚0 disk.sda1.in‚0) proc.memory.percent‚17.12169647216797ËJ*0”÷D‚(:‰ ¬˜êÒñ¼÷ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚288.246' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚57.6492 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0D—¶øD‚(:‰ ¬˜êÒ»™½ù network.recv‚ 7.274152e+06 disk.sda1.in‚0 disk.sda3.out‚ 1705.41015625 disk.drbd2165.out‚0 proc.memory.rssMB‚ 2805.21875) disk./.usagePercent‚38.56950833937375 disk.sda3.in‚ 109.4921875 disk.rbd0.out‚0# memory_percent‚81.8260908126831) proc.memory.availableMB‚ 2977.61328125$ disk.nvme0n1.out‚173940.16015625$ disk./.usageGB‚574.7135467529297! disk.rbd1.out‚8096.6005859375 disk.dm-1.in‚ 109.4921875& disk.rbd0.in‚1.759218604441423e+13! disk.dm-0.out‚173940.16015625 disk.drbd2165.in‚0) proc.memory.percent‚17.12169647216797 proc.cpu.threads‚17$ disk.nvme0c0n1.in‚67738.95703125 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-1.out‚ 1705.41015625& disk.nvme0c0n1.out‚173940.16015625 disk.rbd1.in‚ 2.9658203125 disk.sda.in‚ 109.4921875 disk.sda.out‚ 1705.41015625 cpu‚25.588641692967705 disk.sda1.out‚0" disk.nvme0n1.in‚67738.95703125 network.sent‚ 3.4229484e+07 disk.dm-0.in‚67738.95703125 o=’ùD‚(:‡ »˜êÒð˜÷ gpu.0.gpu‚3.0 gpu.0.memory‚3, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚334.052' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚66.8104 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÌÆuúD‚(: »˜êÒ¯Üù& disk.rbd0.in‚1.759218604441423e+13( proc.memory.availableMB‚ 2977.5703125 network.sent‚ 3.42346e+07 disk.dm-1.out‚ 1708.44921875 disk.sda1.out‚0 disk.sda2.in‚0! disk.rbd1.out‚8135.2724609375 disk.sda2.out‚0# disk.dm-0.out‚173957.1865234375 disk.sda3.in‚ 109.4921875( disk.nvme0c0n1.out‚173957.1865234375& disk.nvme0n1.out‚173957.1865234375 disk.dm-0.in‚67738.95703125 disk.rbd1.in‚ 2.9658203125) proc.memory.percent‚17.12169647216797 network.recv‚ 7.277384e+06 disk.sda.in‚ 109.4921875 disk.sda3.out‚ 1708.44921875 disk.drbd2165.in‚0$ disk.nvme0c0n1.in‚67738.95703125$ memory_percent‚81.82635307312012 proc.memory.rssMB‚ 2805.21875 disk.dm-1.in‚ 109.4921875) disk./.usagePercent‚38.57095990209289 disk.rbd0.out‚0 disk.sda1.in‚0 disk.drbd2165.out‚0" disk.nvme0n1.in‚67738.95703125 cpu‚25.588724005079705 proc.cpu.threads‚17$ disk./.usageGB‚574.7351760864258 disk.sda.out‚ 1708.44921875è¡À{”ûD‚(:‰ ʘêÒòÿçö gpu.0.gpu‚88.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚277.663' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.5326 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0§Û;üD‚(:… ʘêÒ‰¶ºù proc.cpu.threads‚17# disk.dm-0.out‚173971.3193359375$ disk./.usageGB‚574.7121353149414 disk.dm-1.in‚ 109.4921875# disk.nvme0c0n1.in‚ 67739.9140625 disk.rbd1.in‚ 2.9658203125 disk.dm-1.out‚ 1709.8359375& disk.rbd0.in‚1.759218604441423e+13) proc.memory.percent‚17.12169647216797 disk.drbd2165.out‚0 disk.sda.out‚ 1709.8359375$ proc.memory.availableMB‚2977.625 disk.sda1.out‚0 disk.sda3.in‚ 109.4921875! disk.rbd1.out‚8141.4599609375 disk.dm-0.in‚ 67739.9140625 proc.memory.rssMB‚ 2805.21875$ memory_percent‚81.82601928710938& disk.nvme0n1.out‚173971.3193359375 network.recv‚ 7.28106e+06* disk./.usagePercent‚38.569413616586075 disk.sda1.in‚0 disk.sda3.out‚ 1709.8359375 disk.drbd2165.in‚0 disk.sda2.in‚0( disk.nvme0c0n1.out‚173971.3193359375 disk.sda2.out‚0 disk.sda.in‚ 109.4921875 disk.rbd0.out‚0 cpu‚25.588755900384946! disk.nvme0n1.in‚ 67739.9140625 network.sent‚ 3.4239622e+07Î Y4ýD/ _wandbruntime‚29243  _runtime‚29243j€‡üÌþDÆ ' lr-AdamW/pg2‚2.4271607645563073e-05 trainer/global_step‚17649 % _timestamp‚1.7843190596220522e+09 _runtime‚29243.839338465 _step‚1452 ' lr-AdamW/pg1‚2.4271607645563073e-05yxíÑÿDË ' lr-AdamW/pg1‚2.4271607645563073e-05 ' lr-AdamW/pg2‚2.4271607645563073e-05 trainer/global_step‚17649 % _timestamp‚1.7843190596220522e+09 _runtime‚29243.839338465 _step‚1452¬ ^ª’4€E/ _wandbruntime‚29243  _runtime‚29243¬º«É(E”P VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚5091837439256064 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 150648529 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚5075231545679872 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚5091837439256064 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚5075231571853312 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚239 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 150648535 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16607238965760 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1391362428504576 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚5073039891673088 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚5075231545679872 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16605893576192 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 150648535 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 303530533 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚95280075 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 trainer/global_step‚17649 % _timestamp‚1.7843190596232743e+09 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚49999768 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚5073039866245120 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1369944153862144 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 _runtime‚29243.840596691 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 152882004 _step‚1453 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 152882004 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚49999534 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1369943785771008 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚5089618713763314 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16605893576192 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 150648529 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1391362831132160 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0(éÝk~‚E™P XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16605893576192 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 150648529 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1391362831132160 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚5091837439256064 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 150648529 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚5075231545679872 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚5091837439256064 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚5075231571853312 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚239 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 150648535 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16607238965760 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1391362428504576 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚5073039891673088 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚5075231545679872 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16605893576192 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 150648535 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 GADeviceStatsMonitor.on_train_bay?1Æ¡tch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 303530533 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚95280075 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 trainer/global_step‚17649 % _timestamp‚1.7843190596232743e+09 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚49999768 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚5073039866245120 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1369944153862144 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 _runtime‚29243.840596691 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 152882004 _step‚1453 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 152882004 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚49999534 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1369943785771008 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚5089618713763314 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995­ Yþ4ƒE/ _wandbruntime‚29245  _runtime‚29245µ)ë#'„EN MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16579745556584 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 303557210 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚5092117736591872 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚50002491 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16606818611712 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 150656741 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚5075509572583424 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚16608164008448 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚1454 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚5089896148979446 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 _runtime‚29245.235629623 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 150656741 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 303547217 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1391441522651648 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚50002264 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚95285348 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚5092116365021696 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 303547217 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚45283084 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚21419882381824 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 303557210 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16606818611712 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 $ _timestamp‚1.784319061018156e+09 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 152900469 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1391441120031232 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 150656735 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 150656735 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚227 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚5073317748023296 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚5092116365021696 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 303547217 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚45283084 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚21419882381824 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 303557210 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16606818611712 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 $ _timestamp‚1.784319061018156e+09 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 152900469 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1391441120031232 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 150656735 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 150656735 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚227 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚5073317748023296 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 QDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚1454 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚5089896148979446 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 _runtime‚29245.235629623 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 150656741 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 303547217 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1391441522651648 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚50002264 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚95285348® )ÙÉ4†E/ _wandbruntime‚29245  _runtime‚29245¶.ú\‡E 3loss/SRqqt1/assignment_loss‚0.03722807313400849 /loss/FRt1/detection_loss‚0.4236672174867064 .loss/FRt2/detection_loss‚1.100712076301415 0loss/FBt2/detection_loss‚0.03827314930750944 _runtime‚29245.318268735 _step‚1455 0loss/FRt1/assignment_loss‚0.8495993080423839 3loss/SRqqt2/assignment_loss‚0.09705997212881855 0loss/FBt1/detection_loss‚0.11740842140240984 epoch‚44 0loss/FRt2/assignment_loss‚0.0645133906516476 3loss/FBt1/assignment_loss‚4.230005581649171e-08 2loss/SRqqt1/detection_loss‚0.38753552891021287 2loss/SRqqt2/detection_loss‚0.41806589293872776 4loss/FBt2/assignment_loss‚1.4652112524795883e-06 &loss/total_loss‚3.5340645378151483 trainer/global_step‚17649 % _timestamp‚1.7843190611011572e+09øYr@ˆE• 0loss/FRt2/assignment_loss‚0.0645133906516476 3loss/FBt1/assignment_loss‚4.230005581649171e-08 2loss/SRqqt1/detection_loss‚0.38753552891021287 2loss/SRqqt2/detection_loss‚0.41806589293872776 4loss/FBt2/assignment_loss‚1.4652112524795883e-06 &loss/total_loss‚3.5340645378151483 trainer/global_step‚17649 % _timestamp‚1.7843190611011572e+09 3loss/SRqqt1/assignment_loss‚0.03722807313400849 /loss/FRt1/detection_loss‚0.4236672174867064 .loss/FRt2/detection_loss‚1.100712076301415 0loss/FBt2/detection_loss‚0.03827314930750944 _runtime‚29245.318268735 _step‚1455 0loss/FRt1/assignment_loss‚0.8495993080423839 3loss/SRqqt2/assignment_loss‚0.09705997212881855 0loss/FBt1/detection_loss‚0.11740842140240984 epoch‚44¯ ïQʽž‰E‚(:“ Ù˜êÒŠîö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚273.684' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚54.73680000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0:-IŠE‚(:‡ Ù˜êÒƒ¤ù$ disk.nvme0c0n1.in‚67740.74609375 disk.sda2.out‚0 disk.dm-1.out‚ 1713.734375 disk.sda3.in‚ 109.4921875# disk./.usageGB‚574.741325378418( disk.nvme0c0n1.out‚173991.0205078125 disk.rbd1.in‚ 2.9658203125 disk.sda.out‚ 1713.734375 disk.sda3.out‚ 1713.734375 disk.drbd2165.in‚0# memory_percent‚81.8263053894043" disk.nvme0n1.in‚67740.74609375# disk.dm-0.out‚173991.0205078125 disk.sda1.out‚0 disk.drbd2165.out‚0 network.recv‚ 7.284377e+06! disk.rbd1.out‚8293.3037109375 disk.dm-1.in‚ 109.4921875 disk.rbd0.out‚0 disk.sda1.in‚0 cpu‚25.588821804813524& disk.rbd0.in‚1.759218604441423e+13 disk.dm-0.in‚67740.74609375 proc.cpu.threads‚17' proc.memory.availableMB‚ 2977.578125 network.sent‚ 3.4244864e+07* disk./.usagePercent‚38.571372586238084& disk.nvme0n1.out‚173991.0205078125 disk.sda.in‚ 109.4921875 proc.memory.rssMB‚ 2805.21875) proc.memory.percent‚17.12169647216797 disk.sda2.in‚0Š,!Ÿ‹E‚(:” è˜êÒ¾ˆ€÷ gpu.0.gpu‚43.0 gpu.0.memory‚13, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚37.0 gpu.0.powerWatts‚111.353' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.270599999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0£ZS ŒE‚(: è˜êÒËæ²ù disk.dm-0.in‚ 67870.453125& disk.rbd0.in‚1.759218604441423e+13 disk.rbd0.out‚0 cpu‚25.588887026258952' disk.nvme0c0n1.out‚174020.212890625 disk.sda.out‚ 1715.2421875) proc.memory.percent‚17.12169647216797 network.sent‚ 3.430805e+07 network.recv‚ 7.28939e+06" disk.dm-0.out‚174020.212890625 disk.sda3.out‚ 1715.2421875" disk.nvme0c0n1.in‚ 67870.453125 disk.rbd1.in‚ 2.9697265625 proc.cpu.threads‚17$ disk./.usageGB‚574.7153854370117 disk.sda1.in‚0 disk.sda2.out‚0 disk.dm-1.out‚ 1715.2421875) disk./.usagePercent‚38.56963173500525 disk.sda1.out‚0 disk.sda3.in‚ 109.4921875 disk.drbd2165.out‚0 disk.nvme0n1.in‚ 67870.453125 proc.memory.rssMB‚ 2805.21875( proc.memory.availableMB‚ 2977.5703125$ memory_percent‚81.82635307312012 disk.drbd2165.in‚0! disk.rbd1.out‚8423.0224609375 disk.dm-1.in‚ 109.4921875% disk.nvme0n1.out‚174020.212890625 disk.sda2.in‚0 disk.sda.in‚ 109.4921875±qßg”E‚(:‰ ÷˜êÒ˜ŸÍö gpu.0.gpu‚90.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚275.808' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚55.1616 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¶XŒŽE‚(: ÷˜êÒ½ÊËø disk.sda1.in‚0 disk.drbd2165.in‚0# disk.nvme0c0n1.in‚ 67871.4140625! disk.nvme0n1.in‚ 67871.4140625 network.recv‚ 7.292582e+06* disk./.usagePercent‚38.569194730144304 disk.dm-1.out‚ 1717.84765625$ disk.dm-0.out‚174038.25830078125 disk.sda1.out‚0' disk.nvme0n1.out‚174038.25830078125 proc.memory.rssMB‚ 2805.21875 disk.rbd0.out‚0 disk.sda3.in‚ 109.4921875 proc.cpu.threads‚17 disk.rbd1.in‚ 3.0439453125" disk.rbd1.out‚8520.44482421875 cpu‚25.588928317896276 disk.sda.in‚ 109.4921875 disk.dm-0.in‚ 67871.4140625 disk.sda3.out‚ 1717.84765625 disk.drbd2165.out‚0& disk.rbd0.in‚1.759218604441423e+13$ memory_percent‚81.82964324951172 network.sent‚ 3.4313116e+07$ disk./.usageGB‚574.7088737487793 disk.sda.out‚ 1717.84765625 disk.sda2.in‚0 disk.sda2.out‚0& proc.memory.availableMB‚ 2977.03125 disk.dm-1.in‚ 109.4921875) disk.nvme0c0n1.out‚174038.25830078125) proc.memory.percent‚17.12169647216797*çKŸE‚(:” †™êÒõÂñö gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚277.903' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚55.580600000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0MÌÿE‚(:ô †™êÒ÷íÌù network.recv‚ 7.29606e+06 disk.sda.out‚ 1719.40625 disk.sda2.out‚0 disk.dm-0.in‚ 67873.734375" disk.nvme0c0n1.in‚ 67873.734375) proc.memory.percent‚17.12169647216797 proc.cpu.threads‚17& disk.rbd0.in‚1.759218604441423e+13$ disk.nvme0c0n1.out‚ 174046.453125 cpu‚25.58897476593076 disk.sda3.out‚ 1719.40625 disk.nvme0n1.in‚ 67873.734375" disk.nvme0n1.out‚ 174046.453125 proc.memory.rssMB‚ 2805.21875 network.sent‚ 3.4319497e+07 disk.dm-1.out‚ 1719.40625 disk.dm-0.out‚ 174046.453125 disk.sda1.in‚0" disk.rbd1.out‚8646.08544921875 disk.rbd0.out‚0 disk.rbd1.in‚ 3.0439453125* disk./.usagePercent‚38.568975331687454 disk.drbd2165.in‚0 disk.drbd2165.out‚0$ memory_percent‚81.82845115661621 disk.sda2.in‚0( proc.memory.availableMB‚ 2977.2265625$ disk./.usageGB‚574.7056045532227 disk.sda.in‚ 109.4921875 disk.dm-1.in‚ 109.4921875 disk.sda1.out‚0 disk.sda3.in‚ 109.4921875ͬlî’‘E‚(:‡ •™êÒîºçö gpu.0.gpu‚99.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚338.59' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚67.718 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0îË’E‚(:Ž •™êÒ§Ãöø disk.drbd2165.out‚0 disk.sda3.in‚ 109.4921875 disk.drbd2165.in‚0' disk.nvme0n1.out‚174058.31884765625" disk.rbd1.out‚8673.28857421875 proc.memory.rssMB‚ 2805.21875 network.sent‚ 3.4323271e+07 network.recv‚ 7.299214e+06) disk./.usagePercent‚38.56897737974773$ disk./.usageGB‚574.7056350708008 disk.nvme0n1.in‚ 67873.734375 disk.sda.in‚ 109.4921875) proc.memory.percent‚17.12169647216797 disk.dm-0.in‚ 67873.734375 disk.sda3.out‚ 1722.58984375 disk.sda2.in‚0 proc.cpu.threads‚17 disk.dm-1.out‚ 1722.58984375 disk.sda1.in‚0 disk.rbd1.in‚ 3.0439453125) proc.memory.availableMB‚ 2977.31640625& disk.rbd0.in‚1.759218604441423e+13 disk.rbd0.out‚0 disk.sda1.out‚0) disk.nvme0c0n1.out‚174058.31884765625$ memory_percent‚81.82790279388428" disk.nvme0c0n1.in‚ 67873.734375 cpu‚25.589066864553264 disk.dm-1.in‚ 109.4921875$ disk.dm-0„IЙN.out‚174058.31884765625 disk.sda2.out‚0 disk.sda.out‚ 1722.58984375´ÄN)4“E/ _wandbruntime‚29323  _runtime‚29323è¤Ì”EÆ ' lr-AdamW/pg1‚2.3261798901742672e-05 ' lr-AdamW/pg2‚2.3261798901742672e-05 trainer/global_step‚17699 % _timestamp‚1.7843191396229587e+09 _runtime‚29323.840136631 _step‚1456Ô eâÑ•EË ' lr-AdamW/pg1‚2.3261798901742672e-05 ' lr-AdamW/pg2‚2.3261798901742672e-05 trainer/global_step‚17699 % _timestamp‚1.7843191396229587e+09 _runtime‚29323.840136631 _step‚1456° éõi}4–E/ _wandbruntime‚29323  _runtime‚29323P¬”{(—E“P ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚5089131582185472 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16653554896384 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16652209506816 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1373818474246144 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚5103559039785222 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 151058829 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16653554896384 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 151058835 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚255 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚5086933955153920 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16652209506816 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚5089131608358912 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1395296799105536 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 _step‚1457 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404724736 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 153307756 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚5105785163255296 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚50137256 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1395297203830272 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 $ _timestamp‚1.784319139624059e+09 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 153307756 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚5105783791692288 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚21478729584128 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 trainer/global_step‚17699 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 _runtime‚29323.841387322 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚5089131608358912 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36633600 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚21478692950528 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 153317729 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚95544952 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 151058835 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926¦ÁÛ)(˜E˜P =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 153307756 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚5105785163255296 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚50137256 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1395297203830272 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 $ _timestamp‚1.784319139624059e+09 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 153307756 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚5105783791692288 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚21478729584128 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 trainer/global_step‚17699 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 _runtime‚29323.841387322 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚5089131608358912 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚36633600 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚21478692950528 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 153317729 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚95544952 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 151058835 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚5089131582185472 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16653554896384 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16652209506816 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1373818474246144 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚5103559039785222 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 151058829 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16653554896384 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 151058835 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚255 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚5086933955153920 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16652209506816 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚5089131608358912 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1395296799105536 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 _step‚1457 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 404724736± t¯½¸”™E‚(:‰ ¤™êÒ©èŒ÷ gpu.0.gpu‚98.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚262.589' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.5178 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‘iO¨šE‚(:Š ¤™êÒ³ûŠú disk.sda2.out‚0 network.sent‚ 3.4328368e+07) disk./.usagePercent‚38.56897609971006 disk.sda3.in‚ 109.4921875 disk.sda3.out‚ 1723.171875) proc.memory.percent‚17.12169647216797 disk.dm-0.in‚67976.38671875 disk.drbd2165.in‚0 proc.cpu.threads‚17# disk.dm-0.out‚174066.2822265625 disk.sda.out‚ 1723.171875& disk.rbd0.in‚1.759218604441423e+13 disk.sda1.out‚0( disk.nvme0c0n1.out‚174066.2822265625 disk.sda2.in‚0) proc.memory.availableMB‚ 2978.62109375 proc.memory.rssMB‚ 2805.21875& disk.nvme0n1.out‚174066.2822265625$ memory_percent‚81.81993961334229 cpu‚25.589121455869655 disk.rbd1.in‚ 3.0439453125" disk.rbd1.out‚8800.06982421875 disk.dm-1.in‚ 109.4921875$ disk.nvme0c0n1.in‚67976.38671875 disk.sda.in‚ 109.4921875 disk.drbd2165.out‚0 disk.rbd0.out‚0 network.recv‚ 7.30266e+06" disk.nvme0n1.in‚67976.38671875$ disk./.usageGB‚574.7056159973145 disk.dm-1.out‚ 1723.171875 disk.sda1.in‚0šçRÑ4›E/ _wandbruntime‚29325  _runtime‚29325Ö™¤–$'œEžN KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 151067035 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16653134542336 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 304393262 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚21479878629888 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1395375471192576 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 153316234 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚21479915256320 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚5089409609089024 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚45410239 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16653134542336 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚5087211862360064 % _timestamp‚1.7843191410121539e+09 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚5106064089028096 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚5089409609089024 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36626432 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 153326221 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 153316234 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚5106062717457920 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16624638069258 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 153326221 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 304383269 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚5089409582915584 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚50139665 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚16654479939072 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 304383269 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚256 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 151067035 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚95549899 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚5103836475001354 _runtime‚29325.229546872 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚50139916 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1373895960653824 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16625982669692 _step‚1458 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚5103836475001354 _runtime‚29325.229546872 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚50139916 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1373895960653824 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16625982669692 _step‚1458 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 304393262 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚21479878629888 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1395375471192576 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 153316234 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚21479915256320 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚5089409609089024 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚45410239 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16653134542336 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚5087211862360064 % _timestamp‚1.7843191410121539e+09 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚5106064089028096 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚5089409609089024 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚36626432 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 153326221 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 153316234 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚5106062717457920 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16624638069258 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 153326221 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 304383269 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚5089409582915584 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚50139665 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚16654479939072 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 304383269 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚256 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 151067035 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚95549899 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 FDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 153743481 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16699870827008 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚277 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 _runtime‚29403.436155895 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 151469129 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16669951716120 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 305202637 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 % _timestamp‚1.7843192192188172e+09 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚95807489 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚5117499365807130 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚5119731515691520 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚45534846 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 153743481 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1399230379716096 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚5119730144128512 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 305212616 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚21537991907840 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚21537953177088 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 153743481 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16699870827008 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚277 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 _runtime‚29403.436155895 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 151469129 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16669951716120 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 305202637 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 % _timestamp‚1.7843192192188172e+09 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚95807489 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚5117499365807130 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚5119731515691520 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚45534846 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 153743481 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1399230379716096 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚5119730144128512 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 305212616 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚21537991907840 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚21537953177088 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 JDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚5101105951268864 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚5120010441464320 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16699450472960 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚5120009069894144 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 153741986 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚5120010441464320 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚5103309619421184 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚45537389 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16699450472960 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚95813031 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚5117778171051664 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _step‚1462 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚5103309619421184 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 % _timestamp‚1.7843192206090384e+09 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚95812754 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1399309041496576 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1399309448311296 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16670875182366 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 305229314 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 305219321 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 153751973 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1377770281037824 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 305229314 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚21539167273472 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 151477341 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 305219321 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 151477341 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 153751973 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16672219782800 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 153741986 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚5103309645594624 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚38723584xHh)'³E£N D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚5101105951268864 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚5120010441464320 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16699450472960 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚5120009069894144 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 153741986 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚5120010441464320 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚5103309619421184 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚45537389 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16699450472960 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚95813031 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚5117778171051664 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _step‚1462 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚5103309619421184 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 % _timestamp‚1.7843192206090384e+09 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚95812754 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1399309041496576 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1399309448311296 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16670875182366 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 305229314 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 305219321 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 153751973 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1377770281037824 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 305229314 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚21539167273472 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 151477341 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 305219321 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 151477341 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 153751973 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16672219782800 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 153741986 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚5103309645594624 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚38723584 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 406814720 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0¶ ª!ž4´E/ _wandbruntime‚29404  _runtime‚29404åÆëµE 1loss/FRt2/assignment_loss‚0.05233993212183812 3loss/SRqqt2/assignment_loss‚0.06784139553632355 3loss/FBt2/assignment_loss‚6.875384967434051e-05 2loss/SRqqt1/detection_loss‚0.39267656241477933 %loss/total_loss‚3.428379905407067 _step‚1463 0loss/FRt1/assignment_loss‚0.7863797175024673 /loss/FRt2/detection_loss‚1.1062471829021268 1loss/SRqqt2/detection_loss‚0.4306933657829444 _runtime‚29404.908481404 0loss/FBt1/detection_loss‚0.10394308554259404 1loss/FBt2/detection_loss‚0.061456641228626846 epoch‚44 trainer/global_step‚17749 $ _timestamp‚1.784319220691587e+09 3loss/SRqqt1/assignment_loss‚0.03523822157668524 3loss/FBt1/assignment_loss‚4.205735742619965e-06 0loss/FRt1/detection_loss‚0.39149084121326505%\Ó¶E• 3loss/SRqqt1/assignment_loss‚0.03523822157668524 3loss/FBt1/assignment_loss‚4.205735742619965e-06 0loss/FRt1/detection_loss‚0.39149084121326505 1loss/FRt2/assignment_loss‚0.05233993212183812 3loss/SRqqt2/assignment_loss‚0.06784139553632355 3loss/FBt2/assignment_loss‚6.875384967434051e-05 2loss/SRqqt1/detection_loss‚0.39267656241477933 %loss/total_loss‚3.428379905407067 _step‚1463 0loss/FRt1/assignment_loss‚0.7863797175024673 /loss/FRt2/detection_loss‚1.1062471829021268 1loss/SRqqt2/detection_loss‚0.4306933657829444 _runtime‚29404.908481404 0loss/FBt1/detection_loss‚0.10394308554259404 1loss/FBt2/detection_loss‚0.061456641228626846 epoch‚44 trainer/global_step‚17749 $ _timestamp‚1.784319220691587e+09· xo Ÿ·E‚(:” þ™êÒæú÷ gpu.0.gpu‚100.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚339.941' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚67.98819999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ϽCã ¸E‚(: þ™êҲߖú( disk./.usagePercent‚38.5688089267902 disk.sda3.out‚ 1726.3359375 cpu‚25.58954520902464 disk.sda1.in‚0$ memory_percent‚81.82263374328613 disk./.usageGB‚ 574.703125 disk.rbd1.in‚ 3.0439453125 proc.cpu.threads‚17 network.sent‚ 3.4455402e+07 disk.nvme0n1.in‚ 68080.703125 disk.dm-1.out‚ 1726.3359375$ disk.dm-0.out‚174112.14111328125 disk.drbd2165.in‚0" disk.nvme0c0n1.in‚ 68080.703125' disk.nvme0n1.out‚174112.14111328125" disk.rbd1.out‚9614.38232421875& disk.rbd0.in‚1.759218604441423e+13 disk.sda1.out‚0 disk.dm-1.in‚ 109.4921875 disk.rbd0.out‚0) proc.memory.percent‚17.12169647216797) disk.nvme0c0n1.out‚174112.14111328125 network.recv‚ 7.326819e+06 disk.sda2.in‚0 disk.sda.in‚ 109.4921875 disk.sda.out‚ 1726.3359375 disk.dm-0.in‚ 68080.703125 disk.drbd2165.out‚0( proc.memory.availableMB‚ 2978.1796875 disk.sda2.out‚0 disk.sda3.in‚ 109.4921875 proc.memory.rssMB‚ 2805.21875N<Ë”¹E‚(:‰ šêÒ·Õùö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚263.118' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚52.6236 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0`^ÿøºE‚(:‡ šêÒ¡ÛÄù network.sent‚ 3.451749e+07$ memory_percent‚81.82756900787354& disk.rbd0.in‚1.759218604441423e+13 disk.rbd0.out‚0 disk.drbd2165.out‚0' disk.nvme0n1.out‚174119.51904296875" disk.rbd1.out‚9683.19482421875 proc.memory.rssMB‚ 2805.21875 disk.sda1.in‚0 disk.sda3.in‚ 109.4921875 disk.sda3.out‚ 1726.703125 disk.drbd2165.in‚0 network.recv‚ 7.332047e+06 disk.dm-0.in‚ 68080.703125$ disk.dm-0.out‚174119.51904296875) disk.nvme0c0n1.out‚174119.51904296875 disk.rbd1.in‚ 3.0439453125) proc.memory.availableMB‚ 2977.37109375 disk.nvme0n1.in‚ 68080.703125 disk.sda2.out‚0 disk.dm-1.in‚ 109.4921875 proc.cpu.threads‚17$ disk./.usageGB‚574.7034301757812" disk.nvme0c0n1.in‚ 68080.703125 cpu‚25.589611121285618) disk./.usagePercent‚38.56882940739294 disk.sda1.out‚0 disk.sda2.in‚0 disk.sda.in‚ 109.4921875 disk.sda.out‚ 1726.703125 disk.dm-1.out‚ 1726.703125) proc.memory.percent‚17.12169647216797Ö¨JŸ»E‚(:” œšêÒ—Ìõö gpu.0.gpu‚67.0 gpu.0.memory‚20, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚38.0 gpu.0.powerWatts‚119.444' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.888800000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ѿܼE‚(:Š œšêÒô¥Øù disk.sda3.in‚ 109.4921875$ disk.nvme0c0n1.in‚68175.56640625 disk.sda1.out‚0' disk.nvme0c0n1.out‚174127.607421875$ memory_percent‚81.82940483093262 disk.sda.in‚ 109.4921875% disk.nvme0n1.out‚174127.607421875" disk.nØLÃvme0n1.in‚68175.56640625 disk.sda2.in‚0 network.recv‚ 7.335651e+06 disk.dm-0.in‚68175.56640625" disk.dm-0.out‚174127.607421875 disk.sda3.out‚ 1726.87890625 disk.drbd2165.in‚0 disk.rbd1.in‚ 3.0439453125" disk.rbd1.out‚9750.38232421875) proc.memory.percent‚17.12169647216797$ disk./.usageGB‚574.7034873962402 disk.rbd0.out‚0 disk.dm-1.out‚ 1726.87890625& disk.rbd0.in‚1.759218604441423e+13 disk.drbd2165.out‚0 cpu‚25.58963306781568 proc.cpu.threads‚17 disk.sda2.out‚0( proc.memory.availableMB‚ 2977.0703125 network.sent‚ 3.45225e+07 disk.sda.out‚ 1726.87890625 disk.dm-1.in‚ 109.4921875 proc.memory.rssMB‚ 2805.21875) disk./.usagePercent‚38.56883324750595 disk.sda1.in‚0é%óþŸ½E‚(:” «šêÒ»§³÷ gpu.0.gpu‚100.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚291.631' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚58.32619999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0î9Õ¾E‚(:‹ «šêÒíö ù disk.rbd0.out‚0 disk.sda1.out‚0 disk.sda3.in‚ 109.4921875 disk.sda3.out‚ 1727.14453125 disk.dm-1.in‚ 109.4921875 disk.dm-1.out‚ 1727.14453125' disk.nvme0n1.out‚174138.51318359375 disk.rbd1.in‚ 3.0439453125" disk.rbd1.out‚9927.61669921875 disk.sda.out‚ 1727.14453125# memory_percent‚81.8303108215332 proc.memory.rssMB‚ 2805.21875 cpu‚25.58971546970491' disk./.usagePercent‚38.568942050708# disk.nvme0c0n1.in‚ 68176.5234375 disk.drbd2165.out‚0 network.sent‚ 3.4527569e+07 network.recv‚ 7.338943e+06$ disk.dm-0.out‚174138.51318359375) proc.memory.percent‚17.12169647216797 proc.cpu.threads‚17& disk.rbd0.in‚1.759218604441423e+13) disk.nvme0c0n1.out‚174138.51318359375 disk.drbd2165.in‚0! disk.nvme0n1.in‚ 68176.5234375 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda.in‚ 109.4921875' proc.memory.availableMB‚ 2976.921875$ disk./.usageGB‚574.7051086425781 disk.dm-0.in‚ 68176.5234375 disk.sda1.in‚0`Tæ/Ÿ¿E‚(:” ºšêÒþ®÷ö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚290.567' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚58.113400000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0[øJFÀE‚(: ºšêÒàĽù$ disk.dm-0.out‚174141.81396484375) disk./.usagePercent‚38.56882454324979! disk.nvme0n1.in‚ 68176.5234375 disk.sda2.in‚0& proc.memory.availableMB‚ 2976.90625) proc.memory.percent‚17.12169647216797 cpu‚25.589788069321347 disk.drbd2165.in‚0 disk.drbd2165.out‚0 disk.sda1.out‚0 disk.sda.in‚ 109.4921875 disk.sda.out‚ 1727.37890625 disk.dm-1.in‚ 109.4921875& disk.rbd0.in‚1.759218604441423e+13 disk.sda3.out‚ 1727.37890625 disk.sda3.in‚ 109.4921875) disk.nvme0c0n1.out‚174141.81396484375 disk.sda2.out‚0 disk.rbd1.in‚ 3.0439453125 proc.cpu.threads‚17 disk.dm-0.in‚ 68176.5234375 network.recv‚ 7.342624e+06 disk.dm-1.out‚ 1727.37890625# disk.nvme0c0n1.in‚ 68176.5234375' disk.nvme0n1.out‚174141.81396484375 network.sent‚ 3.4532596e+07 disk.sda1.in‚0$ memory_percent‚81.83040618896484 proc.memory.rssMB‚ 2805.21875# disk.rbd1.out‚10095.21044921875 disk.rbd0.out‚0$ disk./.usageGB‚574.7033576965332åªl:4ÁE/ _wandbruntime‚29483  _runtime‚29483ÑÔ—ÊÂEÄ & lr-AdamW/pg1‚2.130441695540113e-05 & lr-AdamW/pg2‚2.130441695540113e-05 trainer/global_step‚17799 % _timestamp‚1.7843192990180118e+09 _runtime‚29483.235237505 _step‚1464Ù}¢ÈÏÃEÉ % _timestamp‚1.7843192990180118e+09 _runtime‚29483.235237505 _step‚1464 & lr-AdamW/pg1‚2.130441695540113e-05 & lr-AdamW/pg2‚2.130441695540113e-05 trainer/global_step‚17799¸ ¸›Kn4ÄE/ _wandbruntime‚29483  _runtime‚29483wxÚ(ÅE”P \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16746186757632 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚45661991 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚5116931655196672 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16744841368064 % _timestamp‚1.7843192990190964e+09 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚45661996 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 151879429 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚5116931681370112 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 151879429 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 _runtime‚29483.236446798 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 154169233 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚50407561 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚5114722132971520 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16716188829228 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 _step‚1465 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚5133676496564736 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚5116931655196672 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚21597224254976 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚5133677868127744 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚5116931681370112 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 154159260 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 306038689 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚5133677868127744LR(ÆE™P UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 151879435 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚5131439691829038 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 154169233 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16746186757632 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1403163971177984 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 306038689 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚5133677868127744 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16746186757632 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚45661991 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚5116931655196672 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16744841368064 % _timestamp‚1.7843192990190964e+09 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚45661996 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 151879429 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚5116931681370112 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 151879429 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 _runtime‚29483.236446798 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 154169233 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚50407561 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚5114722132971520 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16716188829228 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 _step‚1465 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚5133676496564736 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚5116931655196672 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚21597224254976 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚5133677868127744 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚5116931681370112 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 154159260 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚50410550 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16717112295474 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 151887635 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚5131718497073572 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚96075089 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚50410311 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1381644233330688 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 154167738 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚45664534 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚21598452426240 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 trainer/global_step‚17799 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 154167738 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚5117209682100224 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚5133955422330368 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1403243053848064 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 151887641 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚16747111800320 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 306065366 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚96074845 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚45664539Îë*('ÉE¢N B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 306055373 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16745766403584 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 151887641 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 _runtime‚29484.627568421 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚50410550 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16717112295474 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 151887635 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚5131718497073572 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚96075089 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚50410311 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1381644233330688 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 154167738 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚45664534 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚21598452426240 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 trainer/global_step‚17799 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 154167738 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚5117209682100224 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚5133955422330368 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1403243053848064 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 151887641 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚16747111800320 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 306065366 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚96074845 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚45664539 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1403242651227648 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 306065366 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚21598417896960 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 _step‚1466 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚5115000040177664º 7(Ÿ¯4ÊE/ _wandbruntime‚29484  _runtime‚29484ÚÖqËE 0loss/FRt1/assignment_loss‚0.7586744709663283 3loss/SRqqt1/assignment_loss‚0.06917155217508111 3loss/FBt1/assignment_loss‚4.461798588246373e-07 1loss/SRqqt1/detection_loss‚0.4022170997078175 /loss/FBt1/detection_loss‚0.2000149316853432 trainer/global_step‚17799 _step‚1467 2loss/SRqqt2/detection_loss‚0.41711644797126884 1loss/FRt2/assignment_loss‚0.04271748426348664 4loss/FBt2/assignment_loss‚1.2537670916556812e-05 3loss/SRqqt2/assignment_loss‚0.09504907008770912 /loss/FRt1/detection_loss‚0.4435788205864357 /loss/FRt2/detection_loss‚1.0407428824419778 0loss/FBt2/detection_loss‚0.11979869593175693 &loss/total_loss‚3.5890944396679805 epoch‚44 % _timestamp‚1.7843193004925635e+09 _runtime‚29484.709594525M2eÌE• trainer/global_step‚17799 _step‚1467 2loss/SRqqt2/detection_loss‚0.41711644797126884 1loss/FRt2/assignment_loss‚0.04271748426348664 4loss/FBt2/assignment_loss‚1.2537670916556812e-05 3loss/SRqqt2/assignment_loss‚0.09504907008770912 /loss/FRt1/detection_loss‚0.4435788205864357 /loss/FRt2/detection_loss‚1.0407428824419778 0loss/FBt2/detection_loss‚0.11979869593175693 &loss/total_loss‚3.5890944396679805 epoch‚44 % _timestamp‚1.7843193004925635e+09 _runtime‚29484.709594525 0loss/FRt1/assignment_loss‚0.7586744709663283 3loss/SRqqt1/assignment_loss‚0.06917155217508111 3loss/FBt1/assignment_loss‚4.461798588246373e-07 1loss/SRqqt1/detection_loss‚0.4022170997078175 /loss/FBt1/detection_loss‚0.2000149316853432» ªÛM’ÍE‚(:‡ ÉšêÒ͘åö gpu.0.gpu‚20.0 gpu.0.memory‚6, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚330.385' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚66.077 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0²¶ÎE‚(:Š ÉšêÒßÔÑù network.sent‚ 3.4537624e+07 disk.drbd2165.in‚0 disk.sda.out‚ 1727.859375 disk.sda2.in‚0 disk.sda2.out‚0& disk.rbd0.in‚1.759218604441423e+13" disk.nvme0n1.in‚68177.37109375 disk.rbd1.in‚ 3.0439453125 disk.sda.in‚ 109.4921875# disk.dm-0.out‚174152.5908203125( proc.memory.availableMB‚ 2977.2109375 disk.sda1.out‚0 disk.sda3.in‚ 109.4921875& disk.nvme0n1.out‚174152.5908203125 disk.dm-1.out‚ 1727.859375) disk./.usagePercent‚38.56883427153609 disk.sda3.out‚ 1727.859375$ memory_percent‚81.82854652404785 proc.memory.rssMB‚ 2805.21875 proc.cpu.threads‚17 disk.rbd0.out‚0$ disk./.usageGB‚574.7035026550293 disk.sda1.in‚0 disk.drbd2165.out‚0( disk.nvme0c0n1.out‚174152.5908203125 disk.dm-1.in‚ 109.4921875) proc.memory.percent‚17.12169647216797 network.recv‚ 7.345796e+06# disk.rbd1.out‚10255.42919921875 cpu‚25.58989464292417$ disk.nvme0c0n1.in‚68177.37109375 disk.dm-0.in‚68177.37109375üàhº”ÏE‚(:‰ ØšêÒÈüž÷ gpu.0.gpu‚98.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚265.822' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.1644 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0óÎ@,ÐE‚(:ú ØšêÒÚíÀù# disk./.usageGB‚574.703441619873 disk.sda2.out‚0 disk.sda1.in‚0$ memory_percent‚81.82816505432129 cpu‚25.589916937546352" disk.nvme0n1.in‚68185.24609375 disk.rbd1.in‚ 3.0439453125 disk.dm-1.out‚1728.5 disk.rbd0.out‚0 disk.dm-0.in‚68185.24609375 disk.dm-1.in‚ 109.4921875) proc.memory.percent‚17.12169647216797 network.recv‚ 7.35107e+06 disk.sda2.in‚0# disk.rbd1.out‚10415.77294921875# disk.dm-0.out‚174160.0224609375 proc.cpu.threads‚17 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.drbd2165.out‚0) disk./.usagePercent‚38.56883017541554$ disk.nvme0c0n1.in‚68185.24609375( disk.nvme0c0n1.out‚174160.0224609375& disk.nvme0n1.out‚174160.0224609375& disk.rbd0.in‚1.759218604441423e+13( proc.memory.availableMB‚ 2977.2734375 proc.memory.rssMB‚ 2805.21875 network.sent‚ 3.4600974e+07 disk.sda.in‚ 109.4921875 disk.sda.out‚1728.5 disk.sda3.in‚ 109.4921875 disk.sda3.out‚1728.5t]õ ŸÑE‚(:” çšêÒžžåö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚290.567' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚58.113400000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ŒœXiÒE‚(: çšêÒÛêêø* disk./.usagePercent‚38.568827615340204 disk.sda.out‚ 1728.7578125 cpu‚25.58994865275488 disk.rbd0.out‚0& disk.nvme0n1.out‚174167.6162109375 disk.sda2.in‚0) proc.memory.percent‚17.12169647216797 disk.sda.in‚ 109.4921875 disk.dm-0.in‚68255.97265625 disk.dm-1.out‚ 1728.7578125# disk.dm-0.out‚174167.6162109375 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.rbd1.in‚ 3.0439453125# disk.rbd1.out‚10470.72607421875) proc.memory.availableMB‚ 2977.16796875 proc.memory.rssMB‚ 2805.21875 network.sent‚ 3.4606192e+07$ disk./.usageGB‚574.7034034729004 disk.sda3.out‚ 1728.7578125 disk.sda2.out‚0$ memory_percent‚81.82880878448486 proc.cpu.threads‚17 network.recv‚ 7.354482e+06 disk.dm-1.in‚ 109.4921875 disk.drbd2165.out‚0 disk.sda1.in‚0$ disk.nvme0c0n1.in‚68255.97265625( disk.nvme0c0n1.out‚174167.6162109375" disk.nvme0n1.in‚68255.97265625& disk.rbd0.in‚1.759218604441423e+13 disk.sda3.in‚ 109.4921875“è ÍžÓE‚(:“ öšêÒ€Ž÷ gpu.0.gpu‚90.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚286.799' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚57.35979999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0œ|ÖÌÔE‚(:’ öšêÒŒßóù disk.sda1.out‚0 disk.drbd2165.in‚0) disk.nvme0c0n1.out‚174176.74755859375$ memory_percent‚81.83033466339111) proc.memory.availableMB‚ 2976.91796875 network.recv‚ 7.358257e+06 disk.dm-1.in‚ 109.4921875 disk.dm-0.in‚ 68256.8046875) proc.memory.percent‚17.12169647216797 proc.cpu.threads‚17 network.sent‚ 3.4611202e+07 disk.sda3.out‚ 1729.23046875! disk.nvme0n1.in‚ 68256.8046875 disk.sda2.out‚0& disk.rbd0.in‚1.759218604441423e+13 disk.rbd0.out‚0 disk.sda1.in‚0$ disk.dm-0.out‚174176.74755859375# disk.nvme0c0n1.in‚ 68256.8046875 disk.rbd1.in‚ 3.0439453125 disk.sda.in‚ 109.4921875 proc.memory.rssMB‚ 2805.21875 disk.sda3.in‚ 109.4921875 disk.drbd2165.out‚0' disk.nvme0n1.out‚174176.74755859375 disk.sda2.in‚0 disk.sda.out‚ 1729.23046875 disk.dm-1.out‚ 1729.23046875) disk./.usagePercent‚38.56881891108404$ disk./.usageGB‚574.7032737731934# disk.rbd1.out‚10626.05419921875 cpu‚25.590019491838724½afœžÕE‚(:“ …›êÒ¸Éáö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚280.367' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚56.07340000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ˆ7 yÖE‚(:‘ …›êÒ©ä¥ú disk.dm-1.out‚ 1729.5703125 disk.sda3.in‚ 109.4921875 disk.rbd1.in‚ 3.0439453125$ disk.dm-0.out‚174188.95654296875 disk.sda3.out‚ 1729.5703125 disk.sda1.in‚0$ disk.nvme0c0n1.in‚68257.76171875) disk.nvme0c0n1.out‚174188.95654296875) proc.memory.availableMB‚ 2976.91796875 disk.sda.in‚ 109.4921875 disk.rbd0.out‚0 disk.drbd2165.in‚0 disk.sda2.in‚0# disk.rbd1.out‚10790.30419921875) proc.memory.percent‚17.12169647216797 cpu‚25.59002385724294 network.recv‚ 7.360817e+06 disk.sda.out‚ 1729.5703125 disk.dm-0.in‚68257.76171875 disk.sda2.out‚0 disk.drbd2165.out‚0 proc.memory.rssMB‚ 2805.21875 proc.cpu.threads‚17 disk.dm-1.in‚ 109.4921875& disk.rbd0.in‚1.759218604441423e+13 disk.sda1.out‚0$ memory_percent‚81.83033466339111$ disk./.usageGB‚574.7031288146973" disk.nvme0n1.in‚68257.76171875' disk.nvme0n1.out‚174188.95654296875 network.sent‚ 3.461532e+07* disk./.usagePercent‚38.568809182797736RêÐÙ4×E/ _wandbruntime‚29562  _runtime‚29562V)2HÉØEà trainer/global_step‚17849 % _timestamp‚1.7843193787048109e+09 _runtime‚29562.92201699 _step‚1468 & lr-AdamW/pg1‚2.035696947186475e-05 & lr-AdamW/pg2‚2.035696947186475e-05¤§ômÎÙEÈ % _timestamp‚1.7843193787048109e+09 _runtime‚29562.92201699 _step‚1468 & lr-AdamW/pg1‚2.035696947186475e-05 & lr-AdamW/pg2‚2.035696947186475e-05 trainer/global_step‚17849¼ ÒXµj4ÚE/ _wandbruntime‚29562  _runtime‚29562`my(ÛE“P RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 152289729 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1407098455663104 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚5145378647822656 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1385441067307008 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16762425942336 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚5130831717875712 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚5130831691702272 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16792502688256 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 trainer/global_step‚17849 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 154594985 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚240 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 306884720 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 152289729 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚5147624220563968 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16792502688256 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16791157298688 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚5147622849000960 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚96332183 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 152289735 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1385441435398144 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 154594985 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚45789141 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚16763770542658 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 _step‚1469 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚50543042 _runtime‚29562.923516602 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16791157298688 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚245 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚5130831717875712 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚5147624220563968 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚21657020264960 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚5145380017850946 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚5147622849000960 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚5128616221880320 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚96332428 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚5130831691702272 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 $ _timestamp‚1.784319378705964e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 154585012 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0×´DËúÜE˜P H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 152289735 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1385441435398144 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 154594985 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚45789141 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.7S$%allocated‚16763770542658 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 _step‚1469 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚50543042 _runtime‚29562.923516602 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16791157298688 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚245 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚5130831717875712 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚5147624220563968 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚21657020264960 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚5145380017850946 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚5147622849000960 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12154 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚5128616221880320 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚96332428 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚5130831691702272 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 $ _timestamp‚1.784319378705964e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 154585012 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 152289729 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 YJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 1345389568 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1407098455663104 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚5145378647822656 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1385441067307008 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16762425942336 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚5130831717875712 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚5130831691702272 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16792502688256 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 trainer/global_step‚17849 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 154594985 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚240 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 306884720 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 152289729 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚5147624220563968 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16792502688256 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16791157298688 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚5147622849000960 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚96332183 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175½ (Àn4ÝE/ _wandbruntime‚29564  _runtime‚29564É üþ$'ÞEžN I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 152297935 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚45791684 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚17849 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚96337678 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 _step‚1470 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚16793427730944 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚5147903146336768 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚5131109718605824 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚5131109692432384 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚96337440 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚21658210426368 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚5145657453067078 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚5128894103658496 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 154603477 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1385518553714688 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚5131109692432384 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚45791689 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 306901418 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 152297941 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 306901418 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1407177132232192 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚5147903146336768 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚5131109718605824 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚21658175897088 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16764694009016ø7aÚ)'ßE£N OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 154593490 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚5128894129086464 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 YDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚1407176729611776 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1385518921805824 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 306891425 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚16793427730944 XCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚5147903146336768 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚5131109718605824 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 1719664640 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚5131109692432384 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚96337440 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚21658210426368 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚5145657453067078 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚5128894103658496 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 154603477 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1385518553714688 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚5131109692432384 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚45791689 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 306901418 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12154 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 152297941 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 306901418 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1407177132232192 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 QDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚5147903146336768 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚5131109718605824 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚21658175897088 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16764694009016 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 152297935 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚45791684 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚17849 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚96337678 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 _step‚1470 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 FæE‚(:Œ £›êÒÞ¨½ù network.sent‚ 3.468358e+07 disk.sda2.out‚0 disk.dm-0.in‚68325.51953125 proc.cpu.threads‚17 disk.sda.out‚ 1729.9296875 disk.rbd0.out‚0) disk./.usagePercent‚38.56882403123472 disk.drbd2165.out‚0 disk.sda2.in‚0 disk.rbd1.in‚ 3.0439453125 disk.dm-1.out‚ 1729.9296875& disk.rbd0.in‚1.759218604441423e+13" disk.nvme0n1.in‚68325.51953125 disk.sda3.out‚ 1729.9296875# disk.dm-0.out‚174202.2373046875# memory_percent‚81.8209171295166 disk.dm-1.in‚ 109.4921875 disk.sda.in‚ 109.4921875$ disk./.usageGB‚574.7033500671387 disk.sda1.out‚0 disk.drbd2165.in‚0$ disk.nvme0c0n1.in‚68325.51953125 cpu‚25.590119879954507 network.recv‚ 7.369235e+06 disk.sda1.in‚0( disk.nvme0c0n1.out‚174202.2373046875# disk.rbd1.out‚11023.83544921875( proc.memory.availableMB‚ 2978.4609375 proc.memory.rssMB‚ 2805.21875& disk.nvme0n1.out‚174202.2373046875 disk.sda3.in‚ 109.4921875) proc.memory.percent‚17.12169647216797÷à^¬”çE‚(:‰ ²›êÒ¡ˆŽ÷ gpu.0.gpu‚88.0 gpu.0.memory‚28, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚46.0 gpu.0.powerWatts‚348.163' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚69.6326 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0§»l«èE‚(:Ž ²›êÒÂéóù" disk.nvme0n1.in‚68325.51953125 disk.sda2.out‚0 disk.rbd1.in‚ 3.0439453125$ disk./.usageGB‚574.7035751342773 disk.dm-1.out‚ 1730.2109375 disk.sda2.in‚0) proc.memory.availableMB‚ 2978.44140625 disk.sda3.out‚ 1730.2109375$ disk.nvme0c0n1.in‚68325.51953125# disk.rbd1.out‚11179.66357421875 disk.dm-1.in‚ 109.4921875& disk.rbd0.in‚1.759218604441423e+13$ disk.dm-0.out‚174205.12646484375 disk.sda1.in‚0 disk.sda3.in‚ 109.4921875 cpu‚25.59020008877237 network.sent‚ 3.46886e+07 disk.sda.in‚ 109.4921875 disk.dm-0.in‚68325.51953125 proc.memory.rssMB‚ 2805.21875 proc.cpu.threads‚17) disk./.usagePercent‚38.56883913567924 disk.sda1.out‚0 disk.drbd2165.out‚0 disk.sda.out‚ 1730.2109375 disk.rbd0.out‚0 disk.drbd2165.in‚0) disk.nvme0c0n1.out‚174205.12646484375' disk.nvme0n1.out‚174205.12646484375$ memory_percent‚81.82103633880615) proc.memory.percent‚17.12169647216797 network.recv‚ 7.37297e+06C˜®“éE‚(:ˆ Á›êÒŸââö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚265.005' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0fý §êE‚(: Á›êÒîɉù# disk.nvme0c0n1.in‚ 68326.4765625( disk.nvme0c0n1.out‚174214.0498046875 disk.sda2.in‚0 proc.cpu.threads‚17 disk.sda3.in‚ 109.4921875& disk.nvme0n1.out‚174214.0498046875 network.sent‚ 3.469366e+07 disk.drbd2165.out‚0 proc.memory.rssMB‚ 2805.21875 disk.sda.in‚ 109.4921875$ memory_percent‚81.82294368743896 network.recv‚ 7.37625e+06 disk.dm-1.out‚ 1730.93359375 disk.drbd2165.in‚0) proc.memory.percent‚17.12169647216797 disk.rbd1.in‚ 3.0439453125# disk.rbd1.out‚11331.94482421875 disk.dm-0.in‚ 68326.4765625# disk.dm-0.out‚174214.0498046875 disk.sda1.in‚0 disk.sda3.out‚ 1730.93359375 disk.sda2.out‚0 disk.sda.out‚ 1730.93359375) proc.memory.availableMB‚ 2978.12890625) disk./.usagePercent‚38.56881609500116 disk.dm-1.in‚ 109.4921875 disk.sda1.out‚0! disk.nvme0n1.in‚ 68326.4765625 cpu‚25.590257163082292$ disk./.usageGB‚574.7032318115234& disk.rbd0.in‚1.759218604441423e+13 disk.rbd0.out‚0²QÖžëE‚(:“ ЛêÒÅÆÿö gpu.0.gpu‚88.0 gpu.0.memory‚27, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚46.0 gpu.0.powerWatts‚336.465' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚67.29299999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚09½ìE‚(:‡ ЛêÒ»•‰ù network.recv‚ 7.379871e+06" disk.dm-0.out‚174221.787109375 disk.sda3.out‚ 1731.4453125 disk.sda2.in‚0# disk.rbd1.out‚11396.52294921875$ memory_percent‚81.82449340820312 proc.cpu.threads‚17) disk./.usagePercent‚38.56881148686554$ disk./.usageGB‚574.7031631469727$ disk.nvme0c0n1.in‚68327.30859375 disk.dm-1.in‚ 109.4921875 disk.sda1.out‚0 disk.sda3.in‚ 109.4921875% disk.nvme0n1.out‚174221.787109375 disk.sda.out‚ 1731.4453125 cpu‚25.590312680013486' disk.nvme0c0n1.out‚174221.787109375 disk.rbd1.in‚ 3.0439453125 disk.rbd0.out‚0$ proc.memory.availableMB‚2977.875 disk.sda2.out‚0 network.sent‚ 3.4698779e+07 disk.dm-1.out‚ 1731.4453125 disk.dm-0.in‚68327.30859375 disk.sda1.in‚0 disk.drbd2165.in‚0) proc.memory.percent‚17.12169647216797& disk.rbd0.in‚1.759218604441423e+13 disk.drbd2165.out‚0" disk.nvme0n1.in‚68327.30859375 proc.memory.rssMB‚ 2805.21875 disk.sda.in‚ 109.49218758çYŸíE‚(:” ß›êÒ±ö÷ö gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚284.579' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚56.915800000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0À–V› îE‚(: ß›êÒÉÁ°ü network.sent‚ 3.4703808e+07) disk./.usagePercent‚38.56880943880527% disk.nvme0c0n1.out‚174227.0234375 disk.sda2.in‚0 disk.dm-1.out‚ 1731.7890625 disk.rbd0.out‚0 disk.sda3.in‚ 109.4921875 proc.memory.rssMB‚ 2805.21875$ disk./.usageGB‚574.7031326293945 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd2165.in‚0$ disk.nvme0c0n1.in‚68460.28515625" disk.nvme0n1.in‚68460.28515625 disk.sda.in‚ 109.4921875& disk.rbd0.in‚1.759218604441423e+13 disk.dm-0.in‚68460.28515625 disk.dm-0.out‚174227.0234375 disk.drbd2165.out‚0# disk.nvme0n1.out‚174227.0234375 cpu‚25.590336971060086 disk.sda2.out‚0# disk.rbd1.out‚11574.69482421875 disk.sda.out‚ 1731.7890625 disk.sda3.out‚ 1731.7890625$ memory_percent‚81.82449340820312 disk.rbd1.in‚ 3.0439453125 disk.dm-1.in‚ 109.4921875$ proc.memory.availableMB‚2977.875 proc.cpu.threads‚17 network.recv‚ 7.383322e+06) proc.memory.percent‚17.12169647216797äGKh4ïE/ _wandbruntime‚29642  _runtime‚29642æ¸äËðEÅ trainer/global_step‚17899 $ _timestamp‚1.784319457828714e+09 _runtime‚29642.046017968 _step‚1472 ' lr-AdamW/pg1‚1.9430432564839446e-05 ' lr-AdamW/pg2‚1.9430432564839446e-05‘\‰xÐñEÊ _step‚1472 ' lr-AdamW/pg1‚1.9430432564839446e-05 ' lr-AdamW/pg2‚1.9430432564839446e-05 trainer/global_step‚17899 $ _timestamp‚1.784319457828714e+09 _runtime‚29642.046017968À ŸôÚÎ4òE/ _wandbruntime‚29642  _runtime‚29642¼=Æã(óE”P QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 152700035 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 155010764 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚5161569201437184 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 155010764 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16838818618880 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚5161570573000192 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚5161570573000192 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚21716721203712 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16837473229312 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚45916291 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 152700029 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚5142510336217088 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚252 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 307710793 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚247 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1411032074358272 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _runtime‚29642.050052614 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 _step‚1473 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚5161569201437184 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1389315387691008 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚5144731728207872 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚45916296 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚5144731754381312 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 307720772 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1411032476985856 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0Ç•Ž(ôE™P _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16838818618880 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚5159318973844564 ]HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚5144731728207872 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚5159320343872854 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚21716686667264 trainer/global_step‚17899 % _timestamp‚1.7843194578325155e+09 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16837473229312 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 TDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 46141448192 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚5142510336217088 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚252 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 307710793 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 F=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚1995 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚247 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1411032074358272 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _runtime‚29642.050052614 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 _step‚1473 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚6 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚5161569201437184 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚26173440 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1389315387691008 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚5144731728207872 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 49071259648 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚45916296 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚5144731754381312 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 307720772 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1411032476985856 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 152700035 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 155010764 YJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 1344600322 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚5161569201437184 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 155010764 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16838818618880 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚5161570573000192 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚5161570573000192 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚21716721203712 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16837473229312 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚45916291 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 152700029 JDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚96600492 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚5142788217995264 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚5142788192567296 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚16839743661568 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 152708241 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚50681658 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1411111152014848 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16809586521690 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 trainer/global_step‚17899 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1389393242189824 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚45918839 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚5145009755111424 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 152708235 H?DeviceStatsMonitor.on_train_batch_end/ac€7rtive.small_pool.current‚9987 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 155029229 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚5159597779088986 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16810931122124 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 307737470 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚21717875295744 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _step‚1474 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚21717909825024 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚5161849498772992 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 155019242 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚45918834 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚96600735 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚5161848127202816 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 307737470 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚50681896 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 155019242 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1389392874098688 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 % _timestamp‚1.7843194592176633e+09 QDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0þÖ¾ž)'÷E£N VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚45918839 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚5145009755111424 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 152708235 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚545 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 155029229 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚5159597779088986 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16810931122124 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 307737470 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚21717875295744 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _step‚1474 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚21717909825024 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚5161849498772992 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 155019242 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚45918834 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚96600735 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚5161848127202816 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 307737470 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 1680116736 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚50681896 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 155019242 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1389392874098688 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 % _timestamp‚1.7843194592176633e+09 QDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16838398264832 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 307727477 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚96600492 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚5142788217995264 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚5142788192567296 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚16839743661568 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 152708241 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚50681658 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1411111152014848 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16809586521690 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 trainer/global_step‚17899 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1389393242189824 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 -%IŽ4øE/ _wandbruntime‚29643  _runtime‚29643]b;GùEû 1loss/FRt2/assignment_loss‚0.07010928985591118 3loss/SRqqt2/assignment_loss‚0.08618778502552236 2loss/SRqqt1/detection_loss‚0.35040491443630256 0loss/FBt2/detection_loss‚0.05116446840178374 % _timestamp‚1.7843194593029907e+09 _runtime‚29643.519952959 _step‚1475 loss/FBt1/assignment_loss‚0 /loss/FRt2/detection_loss‚1.0505368311736534 1loss/SRqqt2/detection_loss‚0.3887621928235114 &loss/total_loss‚3.3211930458489753 trainer/global_step‚17899 epoch‚44 0loss/FRt1/assignment_loss‚0.7923825760345674 2loss/SRqqt1/assignment_loss‚0.0578784510782145 /loss/FRt1/detection_loss‚0.3847052442453542 0loss/FBt1/detection_loss‚0.08906125333658116 3loss/FBt2/assignment_loss‚3.943757374136659e-08])S”úE€ 1loss/SRqqt2/detection_loss‚0.3887621928235114 &loss/total_loss‚3.3211930458489753 trainer/global_step‚17899 epoch‚44 0loss/FRt1/assignment_loss‚0.7923825760345674 2loss/SRqqt1/assignment_loss‚0.0578784510782145 /loss/FRt1/detection_loss‚0.3847052442453542 0loss/FBt1/detection_loss‚0.08906125333658116 3loss/FBt2/assignment_loss‚3.943757374136659e-08 1loss/FRt2/assignment_loss‚0.07010928985591118 3loss/SRqqt2/assignment_loss‚0.08618778502552236 2loss/SRqqt1/detection_loss‚0.35040491443630256 0loss/FBt2/detection_loss‚0.05116446840178374 % _timestamp‚1.7843194593029907e+09 _runtime‚29643.519952959 _step‚1475 loss/FBt1/assignment_loss‚0 /loss/FRt2/detection_loss‚1.0505368311736534à 럒$“ûE‚(:ˆ î›êÒàóçö gpu.0.gpu‚86.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚39.0 gpu.0.powerWatts‚110.125' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.025 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0wT‰üE‚(:Š î›êÒÓ±Òú disk.dm-1.in‚ 109.4921875 network.sent‚ 3.4708836e+07& disk.rbd0.in‚1.759218604441423e+13 disk.rbd0.out‚0 disk.sda1.in‚0 proc.memory.rssMB‚ 2805.21875 disk.sda.in‚ 109.4921875 disk.sda2.out‚0) proc.memory.percent‚17.12169647216797' proc.memory.availableMB‚ 2978.109375 network.recv‚ 7.386761e+06" disk.dm-0.out‚174234.990234375 disk.sda3.out‚ 1732.44921875 disk.drbd2165.out‚0! disk.nvme0n1.in‚ 68461.2421875 proc.cpu.threads‚17 disk.sda1.out‚0# disk.nvme0c0n1.in‚ 68461.2421875 disk.sda2.in‚0 disk.dm-0.in‚ 68461.2421875% disk.nvme0n1.out‚174234.990234375 disk.sda.out‚ 1732.44921875 disk.dm-1.out‚ 1732.44921875' disk.nvme0c0n1.out‚174234.990234375$ memory_percent‚81.82306289672852) disk./.usagePercent‚38.56888035289225$ disk./.usageGB‚574.7041893005371 disk.sda3.in‚ 109.4921875 disk.drbd2165.in‚0 cpu‚25.590319653559163 disk.rbd1.in‚ 3.0439453125# disk.rbd1.out‚11578.27294921875%o|”ýE‚(:‰ ý›êÒ¼™áö gpu.0.gpu‚100.0 gpu.0.memory‚33, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚242.965' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚48.593 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0\kHþE‚(:‰ ý›êÒß–Êú disk.dm-1.out‚ 1732.96484375 disk.dm-0.in‚68462.07421875 disk.drbd2165.in‚0 disk.dm-0.out‚174245.0859375 disk.sda3.in‚ 109.4921875 disk.sda3.out‚ 1732.96484375% disk.nvme0c0n1.out‚174245.0859375 cpu‚25.59029939749969 disk.dm-1.in‚ 109.4921875$ disk.nvme0c0n1.in‚68462.07421875$ memory_percent‚81.82742595672607) proc.memory.percent‚17.12169647216797" disk.nvme0n1.in‚68462.07421875# disk.nvme0n1.out‚174245.0859375 proc.memory.rssMB‚ 2805.21875$ disk./.usageGB‚574.7040863037109 disk.sda1.in‚0 disk.drbd2165.out‚0 disk.sda2.out‚0* disk./.usagePercent‚38.568873440688826 disk.sda.out‚ 1732.96484375& disk.rbd0.in‚1.759218604441423e+13) proc.memory.availableMB‚ 2977.39453125 network.sent‚ 3.4772003e+07 network.recv‚ 7.392179e+06 disk.rbd1.in‚ 3.0439453125 disk.sda.in‚ 109.4921875 disk.rbd0.out‚0 disk.sda1.out‚0 disk.sda2.in‚0 proc.cpu.threads‚17# disk.rbd1.out‚11733.67919921875¬PÃ#“ÿE‚(:ˆ ŒœêÒäö gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚265.005' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0©£T€F‚(: ŒœêÒ™Ãú$ disk.dm-0.out‚174248.49755859375 disk.sda1.in‚0 disk.sda3.out‚ 1733.203125 disk.sda2.in‚0# disk.rbd1.out‚11787.17919921875 disk.sda.in‚ 109.4921875$ memory_percent‚81.82921409606934 disk.drbd2165.out‚0$ disk.nvme0c0n1.in‚68462.07421875) disk.nvme0c0n1.out‚174248.49755859375 disk.sda.out‚ 1733.203125 disk.dm-1.in‚ 109.4921875 disk.dm-0.in‚68462.07421875) proc.memory.percent‚17.12169647216797 proc.cpu.threads‚17 network.recv‚ 7.395926e+06 disk.sda2.out‚0 proc.memory.rssMB‚ 2805.21875 cpu‚25.590287579600165 disk.sda3.in‚ 109.4921875 disk.sda1.out‚0' disk.nvme0n1.out‚174248.49755859375( proc.memory.availableMB‚ 2977.1015625" disk.nvme0n1.in‚68462.07421875 disk.rbd0.out‚0 network.sent‚ 3.477712e+07) disk./.usagePercent‚38.56886550445527 disk.drbd2165.in‚0 disk.rbd1.in‚ 3.0439453125 disk.dm-1.out‚ 1733.203125& disk.rbd0.in‚1.759218604441423e+13$ disk./.usageGB‚574.7039680480957GF ù“F‚(:ˆ ›œêÒ醄ù gpu.0.gpu‚27.0 gpu.0.memory‚9, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚333.378' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚66.6756 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0?ZÒ& ‚F‚(:• ›œêÒ™½„ú disk.sda3.out‚ 1733.43359375 disk.sda2.out‚0 cpu‚25.59031858157484' disk.nvme0n1.out‚174258.35205078125 disk.sda.in‚ 109.4921875& disk.rbd0.in‚1.759218604441423e+13$ memory_percent‚81.83085918426514 disk.sda1.out‚0 disk.drbd2165.out‚0" disk.nvme0n1.in‚68547.75390625# disk.rbd1.out‚11931.16357421875 proc.memory.rssMB‚ 2805.21875$ disk./.usageGB‚574.7039566040039$ disk.nvme0c0n1.in‚68547.75390625 disk.rbd1.in‚ 3.0439453125 disk.dm-1.out‚ 1733.43359375 disk.rbd0.out‚0) proc.memory.percent‚17.12169647216797 network.sent‚ 3.4782176e+07 disk.drbd2165.in‚0) disk.nvme0c0n1.out‚174258.35205078125) proc.memory.availableMB‚ 2976.83203125 network.recv‚ 7.399213e+06 disk.sda2.in‚0 disk.dm-1.in‚ 109.4921875 disk.dm-0.in‚68547.75390625$ disk.dm-0.out‚174258.35205078125 disk.sda3.in‚ 109.4921875 disk.sda.out‚ 1733.43359375* disk./.usagePercent‚38.568864736432666 disk.sda1.in‚0 proc.cpu.threads‚176Ý”ƒF‚(:‰ ªœêÒ¥Äîø gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚266.358' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚53.2716 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ©#Ç„F‚(: ªœêÒè•ú disk.dm-0.in‚ 68548.5859375 disk.sda3.out‚ 1736.16015625! disk.nvme0n1.in‚ 68548.5859375( disk.nvme0c0n1.out‚174279.6240234375$ disk./.usageGB‚574.7427406311035 disk.sda1.out‚0 disk.drbd2165.in‚0 disk.sda2.out‚0 disk.rbd1.in‚ 3.0439453125# disk.rbd1.out‚12002.71044921875 cpu‚25.590373686598497 network.sent‚ 3.4786666e+07) disk./.usagePercent‚38.57146756503329 disk.sda.out‚ 1736.16015625 disk.sda3.in‚ 109.4921875) proc.memory.availableMB‚ 2977.26171875 proc.cpu.threads‚17 disk.dm-1.out‚ 1736.16015625& disk.rbd0.in‚1.759218604441423e+13& disk.nvme0n1.out‚174279.6240234375 disk.sda2.in‚0 disk.rbd0.out‚0 proc.memory.rssMB‚ 2805.21875) proc.memory.percent‚17.12169647216797 network.recv‚ 7.402656e+06 disk.dm-1.in‚ 109.4921875# disk.dm-0.out‚174279.6240234375# disk.nvme0c0n1.in‚ 68548.5859375 disk.sda1.in‚0 disk.drbd2165.out‚0$ memory_percent‚81.82823657989502 disk.sda.in‚ 109.4921875ª£·4…F/ _wandbruntime‚29721  _runtime‚29721p_–ʆFÄ & lr-AdamW/pg1‚1.852486574405944e-05 & lr-AdamW/pg2‚1.852486574405944e-05 trainer/global_step‚17949 % _timestamp‚1.7843195375588806e+09 _runtime‚29721.775983481 _step‚1476÷Ü·»Ï‡FÉ trainer/global_step‚17949 % _timestamp‚1.7843195375588806e+09 _runtime‚29721.775983481 _step‚1476 & lr-AdamW/pg1‚1.852486574405944e-05 & lr-AdamW/pg2‚1.852486574405944e-05Ä *w4ˆF/ _wandbruntime‚29721  _runtime‚29721§‹ÒgA‰F”P P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1414966560197120 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 _runtime‚29721.777539338 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚16856244768874 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16883789159936 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚5156404399697920 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1414966157569536 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚5175515553873408 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚5158631790886912 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 155436516 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚96858975 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚46043446 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚96859222 G?DeviceStatsMonitor.ÿfÙ on_train_batch_start/inactive_split.all.peak‚572 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚46043441 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚247 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 _step‚1477 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16885134549504 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚21776449494528 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚5175515553873408 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚5173260669894762 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚5158631790886912 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚50815776 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚5156404425125888 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16883789159936 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚21776484030976 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440eQçù(ŠF™P ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚5175515553873408 ZEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚5173260669894762 aLDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚5158631790886912 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚50815776 A8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚1995 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 1680116736 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 aLDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚5156404425125888 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16883789159936 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚21776484030976 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚820 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚26173440 P@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 47545124352 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 _JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚1414966560197120 MDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚1175 _runtime‚29721.777539338 RCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 1371563008 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 YIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 47351595008 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9979 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 1680116736 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚ 368091136 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚16856244768874 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16883789159936 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚5156404399697920 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1414966157569536 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 S>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚5175515553873408 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚5158631790886912 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 155436516 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚96858975 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚46043446 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 1719664640 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 1679318926 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚820 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚96859222 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚46043441 VGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 1345389568 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12154 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚247 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 _step‚1477 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16885134549504 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚21776449494528Å c‹4‹F/ _wandbruntime‚29723  _runtime‚29723¼]ü"'ŒFœN M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16884714195456 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 155454981 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚5156682306904064 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚5158909791617024 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚5158909765443584 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚5175794479639040 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16884714195456 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 153118541 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚5173539475139296 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 308573522 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚46045989 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚5173538105110894 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 308563529 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1415045239377408 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 153118535 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚5175795851209216 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚96864462 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 155444994 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚46045984 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 308573522 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 155454981 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 155444994 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1393267562573824 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚235 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 153118535 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 $ _timestamp‚1.784319538945805e+09 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚50818243 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚96864227 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚5158909791617024 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 _runtime‚29723.16320724 _step‚1478 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚5158909765443584 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 308573522 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚46045989 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 PADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 1371570176 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚5173538105110894 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 308563529 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1415045239377408 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚ 368091136 TDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 47351595008 F=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚1175 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 153118535 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚820 U@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚5175795851209216 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚96864462 P@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 49071259648 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 UEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 46141448192 WHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 1345396736 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 155444994 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚46045984 YJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 1105813504 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 1719664640 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 308573522 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 155454981 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 155444994 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1393267562573824 N>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 ?6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚1995 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚235 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 I@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚1175 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 N>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 B9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚1995 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 153118535 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 UEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 46096207872 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 $ _timestamp‚1.784319538945805e+09 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚50818243 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚820 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚96864227 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚5158909791617024 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12154 YIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 47351595008 _runtime‚29723.16320724 _step‚1478 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚5158909765443584 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 RBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 46141448192 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16884714195456 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 155454981 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚51 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚5156682306904064 M=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 49071259648 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚5158909791617024 [FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚5158909765443584 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚5175794479639040 Yi]Ù˜FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16884714195456 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 153118541 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚5173539475139296Æ ÒR¨ß4ŽF/ _wandbruntime‚29723  _runtime‚29723ë`¤F‘ 1loss/FRt2/assignment_loss‚0.06608709748290895 3loss/SRqqt1/assignment_loss‚0.04240755431364992 4loss/FBt2/assignment_loss‚1.0276662279771887e-09 1loss/SRqqt2/detection_loss‚0.5152711449385858 epoch‚44 _runtime‚29723.248495996 4loss/FBt1/assignment_loss‚2.1586514764282115e-07 0loss/FBt1/detection_loss‚0.12582045739133252 0loss/FBt2/detection_loss‚0.08568731080382323 _step‚1479 0loss/FRt1/assignment_loss‚0.7624997781069055 &loss/total_loss‚3.5593034734169016 % _timestamp‚1.7843195390315387e+09 2loss/SRqqt2/assignment_loss‚0.0773709547947441 0loss/FRt1/detection_loss‚0.42188129478081093 /loss/FRt2/detection_loss‚1.1108053599702197 1loss/SRqqt1/detection_loss‚0.3514723039411071 trainer/global_step‚17949p$Ê°F– 0loss/FRt1/assignment_loss‚0.7624997781069055 &loss/total_loss‚3.5593034734169016 % _timestamp‚1.7843195390315387e+09 2loss/SRqqt2/assignment_loss‚0.0773709547947441 0loss/FRt1/detection_loss‚0.42188129478081093 /loss/FRt2/detection_loss‚1.1108053599702197 1loss/SRqqt1/detection_loss‚0.3514723039411071 trainer/global_step‚17949 1loss/FRt2/assignment_loss‚0.06608709748290895 3loss/SRqqt1/assignment_loss‚0.04240755431364992 4loss/FBt2/assignment_loss‚1.0276662279771887e-09 1loss/SRqqt2/detection_loss‚0.5152711449385858 epoch‚44 _runtime‚29723.248495996 4loss/FBt1/assignment_loss‚2.1586514764282115e-07 0loss/FBt1/detection_loss‚0.12582045739133252 0loss/FBt2/detection_loss‚0.08568731080382323 _step‚1479Ç žwø˜”‘F‚(:‰ ¹œêÒ¼æ€÷ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚336.177' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚67.2354 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0×oA’F‚(:… ¹œêÒ·¹…ú) disk.nvme0c0n1.out‚174295.98095703125# disk.rbd1.out‚12093.47607421875& disk.rbd0.in‚1.759218604441423e+13 disk.sda1.in‚0 disk.sda3.in‚ 109.4921875 disk.drbd2165.in‚0 network.sent‚ 3.4795698e+07 disk.dm-1.out‚ 1738.6875) proc.memory.percent‚17.12169647216797 proc.cpu.threads‚17 disk.dm-1.in‚ 109.4921875 disk.sda.out‚ 1738.6875! disk.nvme0n1.in‚ 68548.5859375 disk.dm-0.in‚ 68548.5859375$ disk.dm-0.out‚174295.98095703125' disk.nvme0n1.out‚174295.98095703125 disk.sda2.in‚0 disk.rbd0.out‚0 proc.memory.rssMB‚ 2805.21875 network.recv‚ 7.413244e+06) disk./.usagePercent‚38.56926590023882 disk.sda1.out‚0$ memory_percent‚81.83152675628662) proc.memory.availableMB‚ 2976.72265625$ disk./.usageGB‚574.7099342346191 disk.drbd2165.out‚0 cpu‚25.59042917026035 disk.sda3.out‚ 1738.6875# disk.nvme0c0n1.in‚ 68548.5859375 disk.sda2.out‚0 disk.rbd1.in‚ 3.0439453125 disk.sda.in‚ 109.4921875؃:HŸ“F‚(:” ÈœêÒ¨™Ïö gpu.0.gpu‚90.0 gpu.0.memory‚26, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚279.015' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚55.803000000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0™5ñ•”F‚(:’ ÈœêÒ¶ÙŸù" disk.nvme0n1.in‚68549.54296875$ memory_percent‚81.83002471923828 cpu‚25.59037662765204 proc.cpu.threads‚17 disk.rbd1.in‚ 3.0439453125 disk.dm-0.in‚68549.54296875 disk.sda3.in‚ 109.4921875) disk.nvme0c0n1.out‚174309.58251953125 disk.drbd2165.in‚0 disk.sda2.out‚0 disk.sda3.out‚ 1741.05859375' disk.nvme0n1.out‚174309.58251953125# disk.rbd1.out‚12163.53857421875 disk.dm-1.out‚ 1741.05859375$ disk.dm-0.out‚174309.58251953125& proc.memory.availableMB‚ 2976.96875 network.recv‚ 7.418597e+06$ disk./.usageGB‚574.7068061828613 disk.dm-1.in‚ 109.4921875 disk.drbd2165.out‚0* disk./.usagePercent‚38.569055974060745 disk.sda.in‚ 109.4921875 disk.sda.out‚ 1741.05859375& disk.rbd0.in‚1.759218604441423e+13 disk.rbd0.out‚0 disk.sda1.in‚0 proc.memory.rssMB‚ 2805.21875 disk.sda2.in‚0 disk.sda1.out‚0$ disk.nvme0c0n1.in‚68549.54296875) proc.memory.percent‚17.12169647216797 network.sent‚ 3.4859583e+07a”»úž•F‚(:“ לêÒáÁ÷ gpu.0.gpu‚99.0 gpu.0.memory‚29, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚296.216' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚59.24320000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Û^nv–F‚(:“ לêÒ§Á÷ø disk.sda3.in‚ 109.4921875' disk.nvme0n1.out‚174321.16455078125 disk.sda1.in‚0 proc.memory.rssMB‚ 2805.21875$ disk.nvme0c0n1.in‚68632.43359375 disk.sda.out‚ 1742.58984375$ disk./.usageGB‚574.7069664001465 disk.drbd2165.in‚0 proc.cpu.threads‚17) disk.nvme0c0n1.out‚174321.16455078125" disk.nvme0n1.in‚68632.43359375 disk.sda.in‚ 109.4921875 disk.rbd0.out‚0 cpu‚25.590373635046312) disk./.usagePercent‚38.56906672637718 disk.sda3.out‚ 1742.58984375 disk.drbd2165.out‚0 disk.rbd1.in‚ 3.0439453125# disk.rbd1.out‚12245.19482421875& disk.rbd0.in‚1.759218604441423e+13$ disk.dm-0.out‚174321.16455078125 disk.sda2.out‚0 disk.sda1.out‚0 disk.dm-1.in‚ 109.4921875 disk.dm-1.out‚ 1742.58984375 disk.dm-0.in‚68632.43359375$ memory_percent‚81.83202743530273 network.sent‚ 3.4864609e+07 network.recv‚ 7.421817e+06 disk.sda2.in‚0' proc.memory.availableMB‚ 2976.640625) proc.memory.percent‚17.12169647216797½×6”—F‚(:‰ æœêÒÉÉÏö gpu.0.gpu‚95.0 gpu.0.memory‚31, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚44.0 gpu.0.powerWatts‚290.516' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚58.1032 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0£Pd4˜F‚(:† æœêÒˆŒú disk.sda1.out‚0' disk.nvme0c0n1.out‚174329.513671875) disk./.usagePercent‚38.56889801741212 disk.rbd1.in‚ 3.0439453125 disk.sda.in‚ 109.4921875 disk.drbd2165.out‚0 cpu‚25.59033445047593 network.sent‚ 3.4869685e+07 disk.rbd0.out‚0" disk.dm-0.out‚174329.513671875 disk.sda3.in‚ 109.4921875' proc.memory.availableMB‚ 2976.890625 proc.memory.rssMB‚ 2805.21875 disk.sda3.out‚ 1743.84765625" disk.nvme0c0n1.in‚ 68633.265625 disk.dm-1.out‚ 1743.84765625 network.recv‚ 7.425469e+06# disk.rbd1.out‚12437.39794921875& disk.rbd0.in‚1.759218604441423e+13 disk.sda.out‚ 1743.84765625) proc.memory.percent‚17.12169647216797 disk.nvme0n1.in‚ 68633.265625 disk.dm-0.in‚ 68633.265625 disk.drbd2165.in‚0 disk.dm-1.in‚ 109.4921875$ memory_percent‚81.83050155639648 proc.cpu.threads‚17$ disk./.usageGB‚574.7044525146484% disk.nvme0n1.out‚174329.513671875 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda1.in‚0Ã7}”™F‚(:‰ õœêÒôŽ’÷ gpu.0.gpu‚99.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚43.0 gpu.0.powerWatts‚281.671' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚56.3342 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Q† šF‚(:Š õœêÒ‘¾¶ù# disk.rbd1.out‚12591.38232421875& disk.rbd0.in‚1.759218604441423e+13 disk.sda1.out‚0 network.recv‚ 7.428809e+06( disk./.usagePercent‚38.5689023695402# disk./.usageGB‚574.704517364502) proc.memory.availableMB‚ 2976.78515625 disk.sda.in‚ 109.4921875 disk.drbd2165.out‚0 disk.sda2.in‚0$ memory_percent‚81.83114528656006! disk.dm-0.out‚174339.78515625 network.sent‚ 3.4874711e+07 disk.sda1.in‚0 disk.drbd2165.in‚0" disk.nvme0n1.in‚68634.22265625 disk.sda2.out‚0 proc.memory.rssMB‚ 2805.21875 disk.sda.out‚ 1744.15234375 disk.rbd0.out‚0 disk.dm-0.in‚68634.22265625 disk.sda3.in‚ 109.4921875$ disk.nvme0n1.out‚174339.78515625 proc.cpu.threads‚17 disk.rbd1.in‚ 3.0439453125 disk.dm-1.out‚ 1744.15234375 disk.sda3.out‚ 1744.15234375) proc.memory.percent‚17.12169647216797 disk.dm-1.in‚ 109.4921875$ disk.nvme0c0n1.in‚68634.22265625& disk.nvme0c0n1.out‚174339.78515625 cpu‚25.590315352571192’5‹õ4›F/ _wandbruntime‚29801  _runtime‚29801'ñƒÌœFÆ ' lr-AdamW/pg1‚1.7640327172389387e-05 ' lr-AdamW/pg2‚1.7640327172389387e-05 trainer/global_step‚17999 % _timestamp‚1.7843196170737634e+09 _runtime‚29801.294047712 _step‚1480Sl¶ÑFË % _timestamp‚1.7843196170737634e+09 _runtime‚29801.294047712 _step‚1480 ' lr-AdamW/pg1‚1.7640327172389387e-05 ' lr-AdamW/pg2‚1.7640327172389387e-05 trainer/global_step‚17999È Ï¬¡4žF/ _wandbruntime‚29801  _runtime‚29801 =gI(ŸF”P VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 % _timestamp‚1.7843196170811965e+09 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚5189461906309632 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1418900263717376 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 _runtime‚29801.298478115 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16930105090560 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 155862268 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 155862268 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1397064028459008 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚21836235258368 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚5189463277872640 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1397064396550144 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16930105090560 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚5170298488606720 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚5189461906309632 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚5172531801219072 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 309382897 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚5189463277872640 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚246 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚46170591 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚16902481881982 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 _step‚1481 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚46170596 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0Wú <( F™P VADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚5189461906309632 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12097 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚25427968 ZEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚5172531801219072 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 309382897 WBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚5189463277872640 O@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 1371563008 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚572 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚246 O?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 49071259648 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9973 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚46170591 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚545 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚16902481881982 RBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 49071259648 _step‚1481 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚46170596 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 402627584 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚51 [LDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 1105813504 % _timestamp‚1.7843196170811965e+09 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 1719664640 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚34536448 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚5 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 295881216 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚5189461906309632 [FDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚1418900263717376 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚1175 WGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 46141448192 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1627 A8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚1995 _runtime‚29801.298478115 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16930105090560 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 46096207872 TEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 1341808128 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 155862268 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12097 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 P@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47499471119 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 155862268 bMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚1397064028459008 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚21836235258368 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚5189463277872640 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1627 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9973 VFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 47351595008 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 1719664640 KBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚1175 [KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 47351595008 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9979 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚6 fQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚1397064396550144 RCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 1370028290 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚1995 M=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 47545124352 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16930105090560 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ]HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚5170298488606720 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚820 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0É |†Î4¡F/ _wandbruntime‚29802  _runtime‚29802Ü ‹¢FN QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1397141514866688 $ _timestamp‚1.784319618466357e+09 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚50955175 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚46173139 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 153528841 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 155870746 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 QBDeviceStatsMonitor.on_train_batch_end/active_bwsÚ˜ytes.small_pool.peak‚ 1680116736 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16902060747906 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚5170576395812864 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚97128081 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚5172809801949184 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 309409574 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚21837465985536 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚5187478431132802 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚5170576370384896 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 309399581 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚16932375522816 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1418979348943360 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚21837431456256 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚5187479801161204 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚16932375522816 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 153528835 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 309409574 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 309399581 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚5189740832075264 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚5172809828122624 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚5172809828122624 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1397141882957824 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 _step‚1482ƒAùY('£F¢N BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 47545124352 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 309399581 RCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 1341808128 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚34529280 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚6 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚16932375522816 ]HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚1418979348943360 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚21837431456256 ?6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚1995 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚5187479801161204 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚16932375522816 TEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 1345396736 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 1679318926 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚572 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 295881216 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚25427968 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 1719664640 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 153528835 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 1344600434 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 402620416 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚1995 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 309409574 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 309399581 T?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚5189740832075264 _JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚5172809828122624 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚5 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚5172809828122624 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚6 dODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚1397141882957824 WGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 47351595008 _step‚1482 QDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 47499471119 RBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 49071259648 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚820 K;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 47545124352 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1627 `KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚1397141514866688 $ _timestamp‚1.784319618466357e+09 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9993 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚50955175 PADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 1370028402 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚46173139 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 153528841 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 155870746 M>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 1371570176 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 1680116736 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚26173440 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9987 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12097 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16902060747906 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚1175 _JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚5170576395812864 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚97128081 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚26173440 XCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚5172809801949184 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 309409574 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9987 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚21837465985536 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 T?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚5187478431132802 [FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚5170576370384896Ê !·éÕ4¤F/ _wandbruntime‚29802  _runtime‚29802±ä¤–¥F‘ &loss/total_loss‚3.4138163700068445 epoch‚44 0loss/FRt2/assignment_loss‚0.0500327600996093 0loss/FRt1/detection_loss‚0.41884367144853557 /loss/FBt1/detection_loss‚0.2750870627206964 0loss/FBt2/detection_loss‚0.02759598300189371 % _timestamp‚1.7843196185527077e+09 4loss/FBt1/assignment_loss‚1.0410184040462615e-05 1loss/SRqqt1/detection_loss‚0.3652021201636113 2loss/SRqqt2/detection_loss‚0.38583662137693386 4loss/SRqqt1/assignment_loss‚0.009951700014031447 3loss/SRqqt2/assignment_loss‚0.05994454830102341 trainer/global_step‚17999 _runtime‚29802.769638722 _step‚1483 0loss/FRt1/assignment_loss‚0.7957232593934556 3loss/FBt2/assignment_loss‚3.304648835684379e-05 /loss/FRt2/detection_loss‚1.0255551868146566 Ñ&¦F– /loss/FRt2/detection_loss‚1.0255551868146566 &loss/total_loss‚3.4138163700068445 epoch‚44 0loss/FRt2/assignment_loss‚0.0500327600996093 0loss/FRt1/detection_loss‚0.41884367144853557 /loss/FBt1/detection_loss‚0.2750870627206964 0loss/FBt2/detection_loss‚0.02759598300189371 % _timestamp‚1.7843196185527077e+09 4loss/FBt1/assignment_loss‚1.0410184040462615e-05 1loss/SRqqt1/detection_loss‚0.3652021201636113 2loss/SRqqt2/detection_loss‚0.38583662137693386 4loss/SRqqt1/assignment_loss‚0.009951700014031447 3loss/SRqqt2/assignment_loss‚0.05994454830102341 trainer/global_step‚17999 _runtime‚29802.769638722 _step‚1483 0loss/FRt1/assignment_loss‚0.7957232593934556 3loss/FBt2/assignment_loss‚3.304648835684379e-05Ë óF쟧F‚(:” „êҶ϶÷ gpu.0.gpu‚100.0 gpu.0.memory‚32, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚45.0 gpu.0.powerWatts‚332.989' gpu.0.enforcedPowerLimitWatts‚500.0( gpu.0.powerPercent‚66.59779999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0=_?f¨F‚(:‘ „êÒÕ”Ãú disk.dm-1.out‚ 1744.6484375) disk.nvme0c0n1.out‚174343.18017578125 proc.cpu.threads‚17 disk.sda3.in‚ 109.4921875 disk.sda2.in‚0 network.recv‚ 7.432481e+06) disk./.usagePercent‚38.56890518562307 disk.drbd2165.out‚0 proc.memory.rssMB‚ 2805.21875 network.sent‚ 3.4879721e+07 disk.sda.in‚ 109.4921875 disk.sda.out‚ 1744.6484375& disk.rbd0.in‚1.759218604441423e+13 disk.dm-0.in‚68634.22265625 disk.sda1.out‚0 disk.drbd2165.in‚0 cpu‚25.59039369563127# disk.rbd1.out‚12668.80419921875 disk.dm-1.in‚ 109.4921875 disk.sda3.out‚ 1744.6484375$ memory_percent‚81.82699680328369 disk.rbd0.out‚0$ disk.nvme0c0n1.in‚68634.22265625' disk.nvme0n1.out‚174343.18017578125 disk.sda2.out‚0 disk.sda1.in‚0) proc.memory.availableMB‚ 2977.46484375$ disk.dm-0.out‚174343.18017578125$ disk./.usageGB‚574.7045593261719 disk.rbd1.in‚ 3.0439453125" disk.nvme0n1.in‚68634.22265625) proc.memory.percent‚17.12169647216797lJбŸ©F‚(:” “êÒ×Á¢÷ gpu.0.gpu‚97.0 gpu.0.memory‚30, gpu.0.memoryAllocated‚57.761306762695305* gpu.0.memoryAllocatedBytes‚ 49616584704 gpu.0.temp‚38.0 gpu.0.powerWatts‚111.112' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.222399999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0–ú t ªF‚(: “êÒ¥óù disk.dm-1.in‚ 109.4921875 disk.dm-1.out‚1747.25 disk.sda1.in‚0 disk.sda3.out‚1747.25$ disk.nvme0c0n1.in‚68741.16796875) proc.memory.percent‚17.12169647216797) proc.memory.availableMB‚ 2977.44921875 network.sent‚ 3.4943084e+07* disk./.usagePercent‚38.569510899449085 disk.sda.out‚1747.25& disk.rbd0.in‚1.759218604441423e+13 disk.rbd0.out‚0 disk.drbd2165.out‚0$ disk./.usageGB‚574.7135848999023 disk.sda2.out‚0# disk.rbd1.out‚12806.17919921875 disk.sda3.in‚ 109.4921875 network.recv‚ 7.437727e+06 disk.sda1.out‚0& disk.nvme0n1.out‚174372.2763671875 proc.memory.rssMB‚ 2805.21875 disk.rbd1.in‚ 3.0439453125 disk.dm-0.in‚68741.16796875$ memory_percent‚81.82709217071533 disk.drbd2165.in‚0( disk.nvme0c0n1.out‚174372.2763671875 cpu‚25.591364418186835 proc.cpu.threads‚17 disk.sda.in‚ 109.4921875# disk.dm-0.out‚174372.2763671875 disk.sda2.in‚0" disk.nvme0n1.in‚68741.16796875z14«F/ _wandbruntime‚29830  _runtime‚29830¼2ìÄò'¬FÅ| 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.7859167456626892 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9759107232093811 0"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚ 0.5703125 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4397655725479126 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 :"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.26923078298568726 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.025902308523654938 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.13738176226615906 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.12873823940753937 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.024593906477093697 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7565905451774597 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.021106984466314316 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 jet/accuracy_5_of_5‚0 6 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.500277578830719 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.34528660774230957 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.7407407760620117 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9803212285041809 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 'particle/f_score‚0.5923117995262146 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9857770204544067 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.11878987401723862 ) Purity/2FRt*SRqqt2FBt/FRt_purity‚0.75 jet/accuracy_1_of_5‚1 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.21687394380569458 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.5487425923347473 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.6972978115081787 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.11331728845834732 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.5863795280456543 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.42500001192092896 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 % _timestamp‚1.7843196466130412e+09 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2543368935585022 8"Purity/*FRt0SRqqt2FBt/event_purity‚0.896346926689148 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.7407407760620117 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9825320243835449 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.027394099161028862 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.5842082500457764 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6523069739341736 particle/accuracy_2_of_4‚1 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7114134430885315 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.19225947558879852 *jet/accuracy_1_of_3‚0.9034304618835449 jet/accuracy_2_of_5‚1 epoch‚44 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.02740025892853737 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.1430366039276123 _step‚1484 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.21281029284000397 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9964991211891174 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6863253116607666 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 *jet/accuracy_1_of_4‚0.9560292363166809 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9746695160865784 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 particle/accuracy_1_of_2‚1 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.35506778955459595 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6784787774085999 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9930816888809204 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6848925948143005 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7318423986434937 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6237251162528992 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 particle/accuracy_1_of_5‚1 _runtime‚29830.83098323 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.12529762089252472 7"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.62238609790802 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.2222222238779068 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7361111044883728 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.1692270189523697 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.6730736494064331 8"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.685478687286377 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.12908266484737396 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9099110960960388 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.43981480598449707 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7176871299743652 8"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.817220151424408 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6535739302635193 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7335518598556519 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5024999976158142 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.019947098568081856 particle/accuracy_1_of_4‚1 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.8657407164573669 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.6972978115081787 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7294651865959167 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.48634982109069824 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.625 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.2083333283662796 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9770596027374268 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9951076507568359 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6643782258033752 particle/accuracy_1_of_3‚1 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.5506650805473328 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9673559069633484 - Purity/0FRt2SRqqt1FBt/FBt_purity‚0.703125 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.08790212869644165 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 8"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.658606231212616 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9970238208770752 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.2222222238779068 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.26851850748062134 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.09635727852582932 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.010547448880970478 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 *jet/accuracy_2_of_4‚0.6787904500961304 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6305987238883972 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.12527677416801453 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 particle/accuracy_3_of_5‚1 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9574466943740845 - Purity/0FRt2SRqqt*FBt/FBt_purity‚0.703125 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7361111044883728 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.17486660182476044 *particle/sensitivity‚0.580460786819458 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.26851850748062134 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9951076507568359 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.09955176711082458 ,jet/accuracy_3_of_3‚0.039067432284355164 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9673894047737122 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 (particle/accuracy‚0.8284079432487488 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5953171253204346 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6539391279220581 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6318874955177307 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 particle/accuracy_2_of_3‚1 +particle/specificity‚0.8962242603302002 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9913487434387207 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9888888597488403 ,jet/accuracy_4_of_4‚0.013899499550461769 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.22388628125190735 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.1296084225177765 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.003723046276718378 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 7validation_average_jet_accuracy‚0.32242751121520996 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7857142686843872 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.13826721906661987 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8888888955116272 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 & Purity/2